Human activity recognition system and method

Abstract

The present approach enables neural networks to understand human behavioral patterns without the need for expert data labeling or laboratory studies. First and second neural networks are trained to learn these patterns without labeling. Once trained, these neural networks are deployed with trained classifiers to determine or classify human activity based on sensor inputs.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Application No. 63 / 399,002, filed August 18, 2022, the contents of which are incorporated herein by reference in their entirety.

[0002] The technical field of the invention relates to training neural networks to understand human behavior patterns, or to using trained neural networks in various applications. [Background technology]

[0003] Human Activity Recognition (HAR) is a branch of machine learning (ML) research that aims to classify or recognize human activities from available sensor data, such as video signals or accelerometer signals from wearable sensors. HAR could be useful, for example, in the medical field, for monitoring and assessing health conditions outside of acute care settings.

[0004] Deep neural network (DNN) architectures are used in many machine learning fields, however, they have not played as dominant a role in HAR as they have in other fields.

[0005] Deep neural networks require large amounts of data to be successful. Ideally, this data is hand-labeled by experts. In the field of image recognition in particular, state-of-the-art models are created by combining camera and human annotations.

[0006] Compared to the amount of publicly available image, text, and audio data, the amount of publicly available wearable sensor data is nearly zero. Furthermore, labeling wearable sensor data, unlike labeling image and audio data, is not easily accomplished by non-experts. For example, data obtained from wearable accelerometers cannot be directly recognized by humans, which is different from labeling objects in images or types of audio. For this reason, HAR research is typically conducted in a laboratory environment, where researchers have subjects perform specific activities and then label the data as predetermined activities.

[0007] However, in such laboratory studies, subjects often perform unnatural movements in real life, making human activity labeling extremely difficult. Human activities inherently have a hierarchical structure, and complex activities can always be decomposed into simpler actions, which can then be further classified. For experimental convenience, HAR studies often label only activities at a specific hierarchical level, while activity labels at other levels are often ignored due to the difficulty of obtaining them. As a result, almost all HAR studies have different definitions of the activity classes they recognize, resulting in incomplete overlap with other studies. This situation makes it difficult to improve activity recognition using deep neural networks by leveraging large amounts of data. Summary of the Invention

[0008] Therefore, there is a need for a method that enables activity recognition using deep neural networks without relying on simple human labeling through artificial activity studies in the clinic, and for computationally powerful network architectures that can extract relevant action features from available data at all levels of the action hierarchy to achieve better recognition. [Brief explanation of the drawings]

[0009] For a more complete understanding of the detailed description of the present invention, please see the following detailed description and accompanying drawings.

[0010] [Figure 1A] 1 illustrates a system configured in accordance with embodiments of the present invention.

[0011] [Figure 1B] 1 illustrates a system configured in accordance with embodiments of the present invention.

[0012] [Figure 2] 1 is a flowchart of an approach according to embodiments of the present invention.

[0013] [Figure 3] 1 illustrates a system configured in accordance with embodiments of the present invention.

[0014] [Figure 4] 1 illustrates a system configured in accordance with embodiments of the present invention.

[0015] [Figure 5] 1 illustrates a system configured in accordance with embodiments of the present invention.

[0016] [Figure 6A] 1 illustrates a system configured in accordance with embodiments of the present invention.

[0017] [Figure 6B] 1 illustrates a system configured in accordance with embodiments of the present invention.

[0018] [Figure 7] 1 illustrates a system configured in accordance with embodiments of the present invention.

[0019] [Figure 8] 1 illustrates an example approach according to embodiments of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0020] An advantage of the approach provided herein is that it enables deep neural networks to understand human behavioral patterns without the need for expert data labeling or laboratory studies. Generally, the approach provided herein employs a neural network that receives signals from wearable sensors (e.g., accelerometers, gyroscopes, etc.) over a fixed time window (e.g., one minute) and generates a detailed, hierarchical description of each second of the input signal. While this description is not directly understandable to humans, it can be easily combined with a small amount of manually labeled data to generate activity classifications at any level. Once trained, the neural network can be used in a variety of applications and perform a variety of tasks, as described elsewhere herein.

[0021] Another aspect of the approach provided herein is to use the large amount of data obtained by simultaneously wearing two or more wearable sensors to train a neural network to understand human activity. Compared to collecting labeled laboratory data, it is relatively easy to have subjects wear multiple wearable devices and collect data during daily activities. For example, by having a group of subjects wear smartwatches on their wrists and monitoring patches on their chests, both devices can simultaneously collect accelerometer data.

[0022] When data is collected simultaneously from multiple body parts of a subject, the signals must originate from the same activity. This eliminates the need for labeled data to train deep neural networks, since the paired data are effectively "labeled" themselves. Simultaneous data collected in this way is guaranteed to belong to the same activity class. On the other hand, if paired data are artificially mixed, mismatched paired data cannot belong to exactly the same activity class.

[0023] In many embodiments of the present invention, a first neural network is iteratively trained to obtain a first trained neural network, and a second neural network is iteratively trained to obtain a second trained neural network. In this training process, the first neural network is input with a first set of wearable sensor data, which causes the first neural network to generate a first feature vector. The first wearable sensor data describes a first physiological feature of a human, and the label included in the first set of wearable sensor data is ignored.

[0024] Further, a second set of wearable sensor data is input to a second neural network, which generates a second feature vector. The second wearable sensor data describes a second physiological characteristic of the human. At this time, labels included in the second set of wearable sensor data are also ignored. At least a portion of the first set of wearable sensor data and the second set of wearable sensor data are from the same activity occurring at the same time, time range, or time frame by the same human.

[0025] The training process also includes a step of predicting in a comparison neural network whether a first feature vector from the first neural network and a second feature vector from the second neural network match or mismatch, where the first feature vector and the second feature vector are determined to match if they were obtained from the same person at approximately the same time.

[0026] The training process further includes backpropagating the error generated by the cost function through the first and second neural networks. The cost function penalizes incorrect match / mismatch determinations of feature vectors. This backpropagation independently updates the parameters of the first and second neural networks. This update optimizes the feature generation of the first and second neural networks and also optimizes the prediction accuracy of the comparison neural network.

[0027] The first trained neural network is then deployed (deployed) by monitoring at least one current subject to obtain current wearable sensor data, applying the current wearable sensor data to the first trained neural network to obtain a current feature vector.

[0028] A classifier (classification model) is trained. In the trained classifier, the current feature vector is mapped to a classification result that represents one or more activity classes. Based on this classification, one or more actions are performed.

[0029] As one example, the action may be quantifying the health impact in a clinical trial between a control group receiving a first intervention or no intervention and a test group receiving a second intervention. As another example, the action may be determining a potential change in a health status in a monitored human. As yet another example, the action may be alerting a clinician to investigate the health status of the monitored human.

[0030] As yet another example, the action may include selectively controlling activation or deactivation of a device. As another example, the action may be controlling the operation of or setting parameters of a medical device associated with treating or monitoring the monitored human. As yet another example, the action may be controlling the operation of or setting parameters of a user electronic device associated with treating or monitoring the monitored human.

[0031] In another example, the mapping is performed using known labeled vectors obtained from other monitored humans.

[0032] In one aspect of the present invention, the sensor is a wrist-based sensor or a chest-based sensor. Currently known technology allows for sensors to be worn continuously throughout activities of daily living, thereby recording waveform data characterizing the wearer's movements. Additionally, data characterizing environmental conditions, cardiopulmonary function, temperature, and the like are often recorded. Such waveform data includes accelerometer waveforms, gyroscope waveforms, electrocardiogram (ECG) waveforms, photoplethysmography (PPG) waveforms, impedance waveforms, electromyography (EMG) waveforms, and electroencephalography (EEG) waveforms. Such sensors may be provided as wristwatch-type devices or adhesive devices such as bandages. In other aspects, the user electronic device may be a smartphone, personal computer, laptop, or tablet. Other examples of sensors and user electronic devices are also contemplated.

[0033] As another example, the trained classifier may include a Random Forest or a third neural network. Other examples are also possible.

[0034] As yet another example, the first neural network and the second neural network are trained at a central location, although training can also occur elsewhere.

[0035] In yet another aspect, the comparison neural network consists of an ensemble (aggregate model) of multiple comparison neural networks with fixed input window sizes but different time scales.

[0036] In some embodiments of the present invention, a system for determining and implementing appropriate lifestyle-improving actions includes a first neural network, a second neural network, and a comparison neural network, wherein the comparison neural network is connected to the first neural network and the second neural network.

[0037] The first neural network is configured to receive a first set of wearable sensor data and, in response, generate a first feature vector, the first wearable sensor data describing a first physiological characteristic of the human, while ignoring labels included in the first set of wearable sensor data.

[0038] The second neural network is configured to receive a second set of wearable sensor data and, in response, generate a second feature vector, the second wearable sensor data describing a second physiological characteristic of the human, while also ignoring any labels included in the second set of wearable sensor data.

[0039] At least a portion of the first set of wearable sensor data and the second set of wearable sensor data are obtained from the same activity occurring at the same time by the same person.

[0040] The comparison neural network is configured to predict whether a first feature vector from the first neural network and a second feature vector from the second neural network are a match or a mismatch, and if the first feature vector and the second feature vector are obtained from the same person at approximately the same time, they are determined to match.

[0041] The error is backpropagated to the first and second neural networks. The error is generated by a cost function that penalizes failure to correctly determine whether a vector is a match or mismatch. This backpropagation independently updates the parameters of the first and second neural networks, producing the first trained neural network and the second trained neural network.

[0042] The system further comprises a trained classifier and a wearable sensor, the wearable sensor being worn by the current subject.

[0043] A first trained neural network is deployed, and current wearable sensor data is acquired from a current subject via a wearable sensor. The data is applied to the first trained neural network to generate a current feature vector. In a trained classifier, the current feature vector is mapped to a classification representing one or more activity classes.

[0044] Based on this classification, one or more actions are performed. As one example, the action may be quantifying the health impact in a clinical trial between a control group receiving a first intervention or no intervention and a test group receiving a second intervention. As another example, the action may be determining a potential health change in the monitored human. As yet another example, the action may be alerting a clinician to investigate the monitored human's health condition.

[0045] As yet another example, the action may include selectively controlling activation or deactivation of a device. As another example, the action may be controlling the operation of or setting parameters of a medical device associated with treating or monitoring the monitored human. As yet another example, the action may be controlling the operation of or setting parameters of a user electronic device associated with treating or monitoring the monitored human.

[0046] In another aspect, this mapping is performed by utilizing known labeled vectors obtained from other monitored humans.

[0047] By way of example, the wearable sensor is a wrist sensor or a chest sensor, although other examples of sensors are also contemplated.

[0048] As another example, the first trained neural network may be deployed at a central location. As yet another example, the trained neural network may be deployed at a remote location.

[0049] As yet another example, training may occur at a central location. Alternatively, training may occur at a remote location.

[0050] In one aspect, the trained classifier includes a Random Forest or a third neural network, although other examples are possible.

[0051] In another embodiment of the present invention, a system for training a neural network includes a first neural network; a second neural network; and a comparison neural network connected to the first neural network and the second neural network; and control circuitry connected to the first neural network, the second neural network, and the comparison neural network.

[0052] The control circuitry is configured to acquire a first set of samples of wearable sensor data from a plurality of humans wearing a first type of wearable sensors. The control circuitry is also configured to acquire a second set of samples of wearable sensor data from a plurality of humans wearing a second type of wearable sensors, at least some of the samples being acquired from the same humans and within substantially the same time window as the samples in the first set of samples. The control circuitry is further configured to train the first neural network to generate a first trained neural network. Training is performed by repeatedly performing the following process: inputting samples from a first sample set into a first neural network to generate features; inputting samples from a second sample set into a second neural network to generate features; inputting the features generated by the first neural network and the features generated by the second neural network into a comparison neural network to predict whether the samples are a match or a mismatch; backpropagating the errors generated by a cost function that penalizes failure to correctly determine whether the input samples are a match or a mismatch to the first neural network and the second neural network, and independently modifying the neural parameters of the first neural network and the second neural network to optimize feature generation by both neural networks and improve the prediction success rate of the comparison neural network.

[0053] In another aspect, the system further includes a trained classifier, wherein a first trained neural network is deployed and wearable sensor data is acquired from a monitored human, and the acquired wearable sensor data is applied to the first neural network, where the first neural network generates a feature set responsive to a time window of the wearable sensor data.

[0054] The trained classifier maps the generated features to one or more activity classes within the time window. Based on this classification, one or more actions are performed. In one example, the action is quantifying the health impact in a clinical trial between a control group receiving a first intervention or no intervention and a test group receiving a second intervention. In another example, the action is determining a potential health change in a monitored human. In yet another example, the action is alerting a clinician to investigate the monitored human's health condition.

[0055] As yet another example, the action may include selectively controlling activation or deactivation of a device. As another example, the action may be controlling the operation of or setting parameters of a medical device associated with treating or monitoring the monitored human. As yet another example, the action may be controlling the operation of or setting parameters of a user electronic device associated with treating or monitoring the monitored human.

[0056] In another embodiment of the present invention, an approach to training a neural network includes obtaining a first sample set of wearable sensor data from a plurality of humans wearing a first type of wearable sensor, obtaining a second sample set of wearable sensor data from a plurality of humans wearing a second type of wearable sensor, and verifying that at least some of the samples are from the same humans and within substantially the same time window as the samples in the first sample set.

[0057] The first neural network is trained by repeatedly performing the following process to generate a first trained neural network: inputting samples from a first sample set into the first neural network to generate features; inputting samples from a second sample set into the second neural network to generate features; inputting the features generated by the first neural network and the features generated by the second neural network into a comparison neural network to predict whether the samples are a match or a mismatch; backpropagating errors generated by a cost function that penalizes failure to correctly determine whether the input samples are a match or a mismatch to the first neural network and the second neural network, and independently updating the neural parameters of the first neural network and the second neural network to improve feature generation and the prediction success rate of the comparison neural network.

[0058] A first trained neural network is deployed and wearable sensor data is acquired from a monitored human.

[0059] The acquired wearable sensor data is transmitted to a first trained neural network, which generates a feature set responsive to a time window of the acquired wearable sensor data, and a trained classifier maps the generated features to one or more activity classes within the time window.

[0060] Based on this classification, one or more actions are taken.

[0061] As yet another example, the action may include selectively controlling activation or deactivation of a device. As another example, the action may be controlling the operation of or setting parameters of a medical device associated with treating or monitoring the monitored human. As yet another example, the action may be controlling the operation of or setting parameters of a user electronic device associated with treating or monitoring the monitored human.

[0062] In another embodiment of the invention, wearable sensor data is acquired from a monitored human. A first neural network generates a feature set responsive to a time window of the wearable sensor data. A trained classifier maps the generated features to one or more activity classes within the time window. One or more actions are performed based on the classification. As one example, the action is quantifying health effects in a clinical trial between a control group receiving a first intervention or no intervention and a test group receiving a second intervention. As another example, the action is determining a potential health change in the monitored human. As yet another example, the action is alerting a clinician to investigate the monitored human's health condition.

[0063] As yet another example, the action may include selectively controlling activation or deactivation of a device. As another example, the action may be controlling the operation of or setting parameters of a medical device associated with treating or monitoring the monitored human. As yet another example, the action may be controlling the operation of or setting parameters of a user electronic device associated with treating or monitoring the monitored human.

[0064] The first neural network is constructed by the following steps: obtaining a first sample set of wearable sensor data obtained from a plurality of humans wearing a first type of wearable sensor; obtaining a second sample set of wearable sensor data obtained from a plurality of humans wearing a second type of wearable sensor, where at least some of the samples are obtained from the same humans and within substantially the same time window as the samples in the first sample set; iteratively training the first neural network; inputting the samples in the first sample set into the first neural network and generating features; inputting the samples in the second sample set into the second neural network and generating features; The features generated by the first neural network and the features generated by the second neural network are input into a comparison neural network, which predicts whether those samples are a match or a mismatch; the errors generated by a cost function that penalizes failure to correctly determine whether an input sample is a match or a mismatch are backpropagated to the first neural network and the second neural network, and the neural parameters of the first neural network and the second neural network are independently updated, thereby improving feature generation by the first neural network and the second neural network and increasing the prediction success rate of the comparison neural network.

[0065] 1A, an example of a system 100 for training a neural network is shown. The system includes a first neural network 102, a second neural network 104, a comparison neural network 106 connected to the first neural network 102 and the second neural network 104, and a control circuit 108 connected to the first neural network 102, the second neural network 104, and the comparison neural network 106. As shown in FIG. 1A, these elements are used in the training phase according to the approach described herein.

[0066] The first neural network 102, the second neural network 104, and the comparison neural network 106 may be any type of neural network or deep neural network, such as a convolutional neural network (CNN). Other examples of neural networks are also contemplated. The first neural network 102 is configured to receive a first set of wearable sensor data and, in response, generate a first feature vector 107. The first wearable sensor data describes first physiological characteristics of one or more human training subjects 112. The first neural network 102 ignores labels included in the first set of wearable sensor data. A "label" refers to an identifier for identifying the source, content, or other characteristics of the data. The human training subject 112 wears wearable sensors 114 and 115. The wearable sensors 114 and 115 may be any type of sensors, such as chest sensors or wrist sensors. These sensors can acquire accelerometer and gyroscope waveform data that characterize movement at the location of the wearer. Additionally, waveform data such as electrocardiogram (ECG), impedance, and photoplethysmography (PPG) signals can be acquired. Data such as skin and ambient temperature can also be recorded, and vital signs such as heart rate and respiration rate can be calculated within the sensor firmware. For example, wristwatch sensors and adhesive torso patch sensors are known devices that collect this data. The neural networks 102, 104, and 106 may be constructed and stored in a database or other electronic memory device. The database 116 is any type of electronic memory device that stores information electronically.

[0067] The second neural network 104 is configured to receive a second set of wearable sensor data. In response, the second neural network 104 generates a second feature vector 109. The second wearable sensor data, as described above, describes a second physiological characteristic of the human 112. The second neural network 104 ignores labels included in the second set of wearable sensor data. A "label" refers to an identifier for identifying the origin, content, and / or other characteristics of the data. A preferred embodiment for human activity recognition uses a wristwatch-type sensor with at least a continuous three-axis accelerometer (sampling rate 5 Hz or higher) and an adhesive torso patch sensor with at least a continuous three-axis accelerometer (sampling rate 5 Hz or higher). The first feature vector 107 and the second feature vector 109 represent input signal characteristics, such as running, standing, exercising, walking at a specific speed, climbing stairs, riding a bicycle, driving a car, playing a specific game, or engaging in a specific activity.

[0068] In one aspect, the first feature vector 107 and the second feature vector 109 are collections of node output values ​​(e.g., real numbers or integers). These values ​​are generated by the activation function of the neural network 102 or 104. This activation function represents the activation level of the nodes in the output layer of the neural network 102 or 104. Activation functions include sigmoid, arctangent, hyperbolic tangent (tanh), rectified linear unit (ReLU), leaky ReLU, and exponential linear unit (ELU). These activation functions are known in the art. In any neural network, the number of nodes (i.e., the length of the vector) can be on the order of tens to hundreds of nodes. Each node receives inputs from other nodes in higher or upstream layers of the neural network. These inputs represent the activation values ​​of those nodes. The received activation values ​​are multiplied by learned weights and summed at each node. In this way, the input activation value from any upstream node is amplified or attenuated by the weights in its connections and contributes to the sum of the total input activation values ​​to the node. Additionally, each node adds a bias value to this sum. The learned weights and biases are updated as the neural network is trained using backpropagation. Adjusting these weights and biases improves the performance of neural networks 102 and 104 based on the task for which the error is being measured.

[0069] Control circuitry 108 applies training and input data to neural networks 102, 104, and 106 and controls these neural networks to generate outputs (e.g., vectors as described herein). As used herein, "control circuitry" generally refers to a microcontroller, computer, or processor-equipped device designed to control the operation of other components or devices. This control circuitry includes memory, a transceiver for communicating with other components, and other associated accessory devices. These architectural options are well known in the art and need not be described in detail herein. Control circuitry 108 is configured to perform the sequence of steps, actions, and / or functions described herein (e.g., using a corresponding program stored in memory).

[0070] At least a portion of the first set of wearable sensor data and the second set of wearable sensor data were obtained at the same time from the same activity of the same human 112. As used herein, "time" refers to a period of time, a time window, a time frame, or a particular point in time (e.g., a particular time of day).

[0071] The comparison neural network 106 is configured to predict whether a first feature vector 107 generated by the first neural network 102 and a second feature vector 109 generated by the second neural network 104 match or mismatch. The first feature vector 107 and the second feature vector 109 are considered to match if they are acquired from the same person at substantially the same time. This determination is made based on the timestamp of the collected data and information about the subject from which the wearable sensor data was acquired. This eliminates the need for post-hoc activity classification labeling by a human expert. Sets of matched and mismatched data are easily generated by programmatic processing by automatically analyzing data from a library of collected wearable sensor data for the subject. The comparison neural network 106 makes predictions by comparing the vectors output by the neural networks 102 and 104. The accuracy of the prediction is evaluated using a cross-entropy loss function. This value is calculated using an error function and used to train the neural networks 102 and 104.

[0072] Generally, training of neural networks 102 and 104 involves two distinct phases. The first phase is a forward pass through the neural network 102 or 104. In this phase, the network parameters are fixed, inputs are fed to the network 102 or 104, and an estimate of the target output value is made. It will be understood that the physical properties of the neural networks and classifiers described herein change during training.

[0073] The second phase, known as the backward pass, is performed through neural network 102 or 104. During this phase, control circuit 108 calculates an error measure regarding whether vectors 107 and 109 were correctly identified as a match or mismatch. For example, at each layer of network 102 or 104, a cost function calculates the partial derivative of each network parameter with respect to the error. The neural network parameters of networks 102 and 104 are then updated by subtracting a small coefficient from the partial derivative of each parameter. This cost function penalizes failure to correctly identify vectors as a match or mismatch. This backpropagation process effectively updates the neural network parameters independently, generating a first trained neural network 102 and a second trained neural network 104.

[0074] Training is performed on batches of available training data containing matched or mismatched data windows. This process is repeated until the neural network 102 or 104 is able to accurately generate estimates of the target output. Methods for determining when training should end are known in the art and are applicable to the present invention. This is typically accomplished when the comparison neural network 106 achieves a desired level of performance or reaches a maximum plateau in performance.

[0075] Once this training step is complete, the trained neural network 102 or 104 can be used to train the classifier 120. In other words, a deep learning model that generates features is trained in the manner described above, and then the trained networks 102 and 104 are fixed and used to generate features for new labeled input data. The features and labels are used to train the classifier 120.

[0076] As a specific example, consider training a classifier specifically for recognizing walking behavior. After neural networks 102 and 104 are trained to analyze one-second data windows, a small amount of labeled walking data is used to build a classifier using features from networks 102 and 104. One-second periods of accelerometer data are input to network 102 or 104, and given a known value for whether the data was acquired during walking or other activities, a corresponding feature vector is generated. This feature vector is then used in conjunction with a traditional classifier, such as a random forest classifier, to classify each one-second period of accelerometer data as "walking" or "non-walking."

[0077] The classifier 120 used in the present invention can be any of a variety of types of classifiers, such as random forests, neural networks, logistic regression, support vector machines (SVMs), etc. After training, the classifier 120 is used to estimate an activity label from the applied features (e.g., vectors). This activity label indicates the type of activity detected and can be used to perform various actions, as described herein.

[0078] 1B, an example of the supervision phase using trained neural networks and trained classifiers (resulting from the approach of FIG. 1A) is shown. For example, the process of FIG. 1A (or a similar process) is performed to obtain trained neural networks 102 and 104 and trained classifier 120. When either of these trained networks 102 and 104 is deployed and input data is applied, a corresponding feature vector is generated.

[0079] 1B, a first trained neural network 102 is deployed. In one aspect, the system 100 further includes a trained classifier 120 and a wearable sensor 122. The wearable sensor 122 is worn by a current subject 124. By way of example, the wearable sensor 122 may be a chest or wrist sensor to accommodate the deployed network. Other sensor examples are also contemplated. A second trained neural network 104 may also be deployed, and the following description is also applicable thereto.

[0080] In one aspect, the trained neural network 102 and the trained classifier 120 may be deployed at a central location 103. The central location 103 may be a company, a corporate headquarters, a home office, a data center, or other physical location. As another example, the trained neural network 102 may be deployed at a remote location, such as a hospital, a clinic, or a research facility. Other examples are possible. Data from the sensors may be uploaded to the central location 103 using a mobile digital communications network, a local area network, or other means. Uploads may occur periodically or continuously.

[0081] A first trained neural network 102 is deployed, and current wearable sensor data is acquired from a current subject 124 via wearable sensors 122 and applied to the first trained neural network 102 to obtain a current feature vector 128. In a trained classifier 120, the current feature vector 128 is mapped to a classification representing one or more activity classes. In one aspect, this mapping is performed using known labeled vectors obtained from other monitored humans. In another aspect, this mapping includes comparing the current feature vector 128 to known labeled vectors obtained from other monitored humans and using a nearest neighbor calculation to determine the activity class.

[0082] Based on this classification, one or more actions are performed. As one example, the action may be quantifying a health effect between at least a first intervention or a control group without intervention and a test group receiving a second intervention. As another example, the action may include determining a potential health change in the monitored human 124. As yet another example, the action may include alerting a clinician to investigate the health status of the monitored human 124.

[0083] As yet another example, an action may include selectively controlling the activation or deactivation of a device. As another example, an action may be controlling the operation of or setting parameters of medical equipment associated with treating or monitoring the monitored human 124. As yet another example, an action may be controlling the operation of or setting parameters of user electronic devices associated with treating or monitoring the monitored human 124. For example, the frequency, speed, and / or extent of treatment or monitoring may be changed. Settings on these devices (e.g., settings related to video or display quality, font size, color, or other on-screen characteristics) may also be controlled and changed. For example, the manner in which alerts are displayed (e.g., based on the type of alert) may also be controlled.

[0084] Referring to FIG. 2, an approach for training and utilizing a trained neural network is described.

[0085] In step 202, a first neural network is iteratively trained to obtain a first trained neural network. In step 204, a second neural network is iteratively trained to obtain a second trained neural network. The training includes receiving a first set of wearable sensor data at the first neural network. The first neural network responsively generates a first feature vector. The first wearable sensor data describes a first physiological characteristic of the human. Labels included in the first set of wearable sensor data are ignored. In one aspect, the sensor is a wrist sensor or a chest sensor.

[0086] The training further includes receiving a second set of wearable sensor data at a second neural network. The second neural network responsively generates a second feature vector. The second wearable sensor data describes a second physiological characteristic of the human. Labels included in the second set of wearable sensor data are ignored. At least a portion of the first set of wearable sensor data and the second set of wearable sensor data are obtained from the same activity occurring at the same time by the same human.

[0087] In step 206, the comparison neural network predicts whether the first feature vector from the first neural network and the second feature vector from the second neural network are a match or a mismatch. The first feature vector and the second feature vector are determined to match if they were obtained from the same person at substantially the same time or time frame. In one aspect, the comparison neural network includes an ensemble of multiple independent comparison neural networks with fixed input window sizes and different time scales.

[0088] In step 208, an error is backpropagated to the first and second neural networks. The error is generated by a cost function that penalizes failure to correctly determine whether a vector is a match or mismatch. Backpropagation and application of the error to the neural networks is effective to independently update the parameters of the first and second neural networks, optimizing feature generation by the first and second neural networks and optimizing the predictive accuracy of the comparison neural network.

[0089] As an example, the first neural network and the second neural network are trained at a central location using steps 202, 204, and 206. These elements may also be deployed at different locations.

[0090] In step 210, the first trained neural network is deployed, for example, at a central location where it can be conveniently used to receive data.

[0091] In step 212, at least one current subject is monitored to obtain current wearable sensor data, and the current wearable sensor data is applied to a first trained neural network to obtain a current feature vector.

[0092] In step 214, a trained classifier maps the current feature vector to classifications representing one or more activity classes. The classifier itself may also be trained. In one example, this mapping utilizes known labeled vectors obtained from other supervised humans and uses a k-nearest neighbor method to output the class of the nearest neighbor or centroid of the current feature vector. In one aspect, the trained classifier includes a random forest or a third neural network. Other examples are also possible.

[0093] Based on this classification, one or more actions are taken in step 216. The actions can take many forms.

[0094] As one example, the action may be quantifying a health effect between at least a first intervention or a control group without intervention and a test group receiving a second intervention. As another example, the action may be determining a potential change in the health status of the monitored human. As yet another example, the action may be alerting a clinician to investigate the health status of the monitored human.

[0095] As yet another example, the action may include selectively controlling activation or deactivation of a device. As another example, the action may be controlling the operation of, or setting parameters of, a medical device associated with treating or monitoring the monitored human. As yet another example, the action may be controlling the operation of, or setting parameters of, a user electronic device associated with treating or monitoring the monitored human. As an example, the user electronic device may be a smartphone, personal computer, laptop, or tablet. Other examples of sensors and user electronic devices are also possible.

[0096] 3, an example approach for training neural networks is described. In this example, a torso neural network 302 models human activity measured by sensors worn on the human torso (e.g., chest sensors such as chest patches), and a wrist neural network 304 models human activity measured by wrist sensors. A comparison neural network 306 is also provided. These are examples of human-worn sensors and their locations; other types of sensors and other locations on the human are also contemplated.

[0097] In this example, matched accelerometer data 310 and mismatched paired accelerometer data 320 are used to train neural networks 302 and 304. The torso accelerometer data is the same in both matched data 310 and mismatched data 320, but the wrist accelerometer data is modified to a selected segment of the wrist accelerometer signal at a different time and period (time frame) than the torso accelerometer segment was collected.

[0098] In step 322, the accelerometer data from the matched and mismatched time frame pairs is provided to either the torso neural network 302 or the wrist neural network 304, depending on the origin of the data. By way of example, the torso neural network 302 and the wrist neural network 304 are multi-layer convolutional neural networks, each generating a single vector representation of the input accelerometer segments. More specifically, the torso neural network 302 generates vector 332 (a single vector representation of the chest or torso input accelerometer segments), and the wrist neural network 304 generates vector 334 (a single vector representation of the wrist input accelerometer segments). For the mismatched data 320, the torso neural network 302 generates vector 336 (a single vector representation of the torso input accelerometer segments), and the wrist neural network 304 generates vector 338 (a single vector representation of the wrist input accelerometer segments selected from a time point different from the time the torso data was collected).

[0099] While matched accelerometer segment vectors 332 and 334 are generally similar (e.g., identical and / or with minor deviations in amplitude, pattern, shape, size, or other characteristics), mismatched segment vectors 336 and 338 are less similar (e.g., different and / or with larger deviations in amplitude, pattern, shape, size, or other characteristics), comparison network 306 receives as input vector representations of these activities and attempts to correctly determine whether they represent matched or mismatched data. An error signal is generated from the comparison network, indicating whether the data was correctly determined as a match or mismatch. The error signal is then backpropagated to networks 302 and 304, forcing torso neural network 302 and wrist neural network 304 to learn more detailed representations of the accelerometer data so that comparison neural network 306 can correctly classify them.

[0100] As an example, matched data 310 is taken from a time frame in which the subject is walking, while mismatched data 320 is taken from a different time frame of walking. To distinguish between these two activities, it is not enough for the torso neural network 302 and wrist neural network 304 to learn that the data is walking. The torso neural network 302 and wrist neural network 304 must further subdivide and classify the data, for example, using characteristics such as walking speed and walking style to further classify walking. In this way, the networks 302 and 304 can learn rich representations of the underlying activities without knowing the exact activities or the nature of the activities.

[0101] This approach collects data from different body parts simultaneously and can show that the paired data are from the same activity by the same person. Artificial data is introduced to distinguish between mismatched data that are not from the same activity. This approach avoids the need to use labels in the deep neural network training process or to label data, because data collected simultaneously from the same person must belong to the same activity class, regardless of the activity class.

[0102] After the torso neural network 302 and wrist neural network 304 have been trained in this manner, they can be used in a variety of applications. For example, the learned features can be combined with a small amount of manually labeled activity data and a conventional classifier trained to convert the features into a set of desired activity classifications.

[0103] The torso neural network 302 and wrist neural network 304 can be fine-tuned with a small amount of manually labeled activity data to accommodate a particular set of activity classifications.

[0104] As described elsewhere herein, a trained torso neural network or wrist neural network is deployed in the monitoring phase, data obtained from the monitored human is applied to the trained neural network, and the resulting vectors from the trained neural network are applied to a trained classifier. To train the classifier, the vectors generated by the torso neural network 302 and wrist neural network 304 can also be applied on the fly (real time) for immediate classification (a "one-shot" classifier).

[0105] For example, a clinician in physical therapy may have a subject wear a wearable device while performing a particular therapy. The characteristics of that therapy can be used to train a classifier on how the subject performs that activity. The clinician can then quantify how well the subject adheres to the physical therapy program and train a classifier to associate vectors with "good" or "poor" adherence.

[0106] As another example, a clinician during physical therapy can demonstrate correct and incorrect ways to perform a movement and train a one-shot classifier or distance measure to recognize them. Later, when the subject performs the movement independently at home, the sensor signals can be applied to the trained neural network, and the output vector of the trained neural network can be applied to the classifier to classify the movement as being performed correctly or incorrectly. The device can then issue an alert if the subject is performing the movement incorrectly.

[0107] As yet another example, sensor data can be applied to a trained neural network to generate a vector representing feature attributes that indicate the effects of a medication (assuming these features capture those attributes). For example, detecting gait pattern may indicate the specific effects of a medication taken by a subject.

[0108] Referring to Figure 4, an example of an apparatus for training a neural network is shown. In this example, an accelerometer input is used as an input to the neural network. In addition, additional signal inputs, such as a gyroscope, may also be used. In this example, two neural networks are trained, although it is possible to use more than two neural networks.

[0109] As described herein, these approaches enable the comparison of two time windows of accelerometer data acquired from different devices. In one aspect, a multi-window time scale vector is created. By generating multiple vectors for different time scales, each vector can capture features related to that time scale. An example might include features at the one-second level down to the one-minute level. These features are for general activity recognition, and the optimal time scale for all activity recognition tasks is not known in advance. For example, a feature vector created from a four-second window may be suitable for detecting running or walking, but may be insufficient for determining more advanced activities such as "playing basketball." In another example, if the desired feature window size is known prior to training, multiple time scales need not be used, and only the desired feature window size can be used.

[0110] As another example, a convolutional network using dilated convolution can keep the inputs and outputs the same at each network layer, but the receptive field of the network (the range of time the network "sees" at a particular depth) doubles with each layer. In such a network, the outputs from each layer can be paired as well.

[0111] Turning to Figure 4, the torso neural network 402 includes multiple blocks that perform convolution and max-pooling operations. The convolution and max-pooling operations roughly halve the temporal resolution of the data received at each block. In this way, the features produced at each layer become more general and high-level, and less specific. The blocks that perform these functions include blocks 410, 411, 412, 413, 414, 415, 416, 417, 418, and 419.

[0112] The linear deep convolutional layers 420, 421, 422, 423, 424, 425, and 426 reduce the features (dimensionality) of the data. In one aspect, the linear deep convolution calculates a weighted average along the feature dimensions, generating an output feature vector of, for example, 60×10. The linear deep convolutional layers 420, 421, 422, 423, 424, 425, and 426 function as linear layers because they are trained in the main path of the neural network and reduce the dimensionality of the information.

[0113] The wrist neural network 404 also includes multiple blocks that perform convolution and max-pooling operations. The blocks that perform these functions include blocks 430, 431, 432, 433, 434, 435, 436, 437, 438, and 439. The convolution and max-pooling operations roughly halve the temporal resolution at each block. In this way, the features generated at each layer become more general and high-level, and less specific. Linear deep convolution layers 440, 441, 442, 443, 444, 445, and 446 reduce the dimensionality of the data. The linear deep convolutions calculate a weighted average along the feature dimensions, producing an output feature vector of, for example, 60 x 10. As mentioned above, the linear deep convolutional layers 440, 441, 442, 443, 444, 445, and 446 function as linear layers because they are trained in the main path of the neural network and perform dimensionality reduction of information.

[0114] Data 450 passes through torso neural network 402 in the direction indicated by arrow 451. Data 452 passes through wrist neural network 404 in the direction indicated by arrow 453.

[0115] In general, data 450 passes through network 402, and data 452 passes through network 404. This data is reduced in size by each block of the respective neural network. As mentioned above, some blocks are fed into linear deep convolutional layers.

[0116] The linear depth wise convolutional layers of the torso neural network and the wrist neural network are combined by concatenators 453, 454, 455, 456, 457, 458, and 459. The outputs of concatenators 453, 454, 455, 456, 457, 458, and 459 flow to comparison neural network 460. Comparison neural network 460 includes seven neural networks: 461, 462, 463, 464, 465, 466, and 467. Once the torso and wrist features are combined, the resulting features at seven different resolutions (from every second to every minute) are input to seven separate feedforward comparison neural networks 461, 462, 463, 464, 465, 466, and 467 that make up comparison neural network 460.

[0117] The seven comparison neural networks include feedforward layers 470, 471, 472, 473, 474, 476, 477, 478, 479, 480, 481, 428, and 483, sigmoid layers 485, 486, 487, 488, 489, 490, and 491, and cross-entropy (x-entropy) layers 492, 493, 494, 495, 496, 497, and 498. The outputs of each layer are summed in step 499 to calculate a single loss. For example, data from combiner 453 passes through feedforward layer 470, feedforward layer 477, sigmoid layer 485, and x-entropy layer 492 to calculate the loss for network 461. This loss is summed with the losses of networks 462, 463, 464, 465, 466, and 467 to calculate the overall loss 499.

[0118] Sigmoid activation blocks 485, 486, 487, 488, 489, 490, and 491 output a probability of "match" or "mismatch" for each timestamp in the feature vector (e.g., a 60x1 probability for every 1-second feature input to network 461, a 2x1 probability for every 30-second feature input to network 466, etc.).

[0119] Networks 402 and 404 are trained to minimize the total cross-entropy loss across all temporal resolutions in step 499. This loss is back-propagated to the torso neural network 402 and the wrist neural network 404 to train these networks. Illustratively, the weights within these networks are adjusted.

[0120] As an example of the operation of the system of FIG. 4 , data sample 450 from the torso sensor includes 60 seconds of waveform data from a triaxial accelerometer. Similarly, data sample 452 from the wrist sensor includes 60 seconds of waveform data from a triaxial accelerometer. The triaxial accelerometer measures vibration or movement in three orthogonal directions (x, y, and z) relative to the plane of the circuit board of the sensor device (wrist or chest patch) on which the accelerometer is mounted. Human walking motion is expressed in a complex manner relative to the orthogonal directions of the accelerometer at the wrist or chest position, or at any other location where the sensor is worn, but typical walking motions such as heel strike, swing, and sway are inherently encoded. On the other hand, non-walking motions are expressed in a complex manner and are necessarily encoded in the orthogonal directions of the accelerometer unit.

[0121] The accelerometer data in each time window is resampled to 24Hz (24 samples per second), so the input data has 1440 samples in the time dimension and three channels of accelerometer (e.g., 1440 × 3 input). In the initial stage, four convolutional and max-pooling layers are applied to the resampled accelerometer signal data to build features in the channel dimension, decreasing the time resolution.

[0122] The wrist data 403 is applied to block 430 (a convolutional layer producing a 1440x24 output, followed by a maxpooling layer (maxpool) giving a 480x24 output), then to block 431 (a convolutional layer producing a 480x48 output, followed by a maxpooling layer giving a 240x48 output), then to block 432 (a convolutional layer producing a 240x96 output, followed by a maxpooling layer giving a 120x96 output), block 433 (a convolutional layer producing a 120x96 output, followed by a maxpooling layer giving a 60x96 output), and block 434 (a convolutional layer producing a 60x96 output, followed by a maxpooling layer giving a The following blocks are applied in order: block 435 (the convolutional layer produces a 30x96 output and the max pooling layer produces a 15x96 output), block 436 (the convolutional layer produces a 15x96 output and the max pooling layer produces an 8x96 output), block 437 (the convolutional layer produces an 8x96 output and the max pooling layer produces a 4x96 output), block 438 (the convolutional layer produces a 4x96 output and the max pooling layer produces a 2x96 output), and finally block 439 (the convolutional layer produces a 2x96 output and the max pooling layer produces a 1x96 output).

[0123] In layer 433 (layer 4), the wrist input data (1440x3) is transformed to a 60x96 dimension. This means that a 96-element vector is generated for each second of input data during the 60 seconds. At this point, a first linear deep convolution filter 440 is applied to create features with a 1-second resolution. The linear deep convolution calculates a weighted average along the feature dimensions, producing a 60x10 output feature vector. This layer is linear because it is trained within the main pass of the neural network, reducing the dimensionality of the existing information. Similar operations are performed in other layers 441, 442, 443, 444, 445, and 446, which extract data from convolutional / max-pooling layers 434, 435, 436, 437, 438, and 439. The same process is applied to the torso data 401, which is processed through layers 410-419 and linear deep convolutional layers 420-426.

[0124] Convolutional and max-pooling layers roughly halve the temporal resolution at each layer. In this way, the features generated at each layer become higher-level features and more abstract. However, by using linear deep convolutions at each step, the entire hierarchy of features can be represented. These features are used by the comparison network to determine matching / mismatching activity. At the highest level, there are 10 features for every 60 seconds (60 x 10), and at the lowest level, there are 30 features for every minute of data (1 x 30).

[0125] The torso and wrist features are concatenated by couplers 453, 454, 455, 456, 457, 458, and 459. Once the features are combined, the seven different resolution features (ranging from 1 second to 1 minute) are input to seven separate feedforward neural networks 461, 462, 463, 464, 465, 466, and 467. As previously mentioned, feedforward networks 461, 462, 463, 464, 465, 466, and 467 each contain two feedforward (hidden) layers, a sigmoid layer (with a sigmoid activation function), and a cross-entropy layer, and output a probability of "match" or "mismatch" for each timestamp in the feature vector (e.g., a 60x1 probability for 1 second resolution features input to network 461, and a 2x1 probability for 30 second resolution features input to network 466). The cross-entropy loss is averaged over each feedforward network, and the average cross-entropy loss over all feedforward networks is summed by 499.

[0126] The networks 402 and 404 are trained to minimize the total cross-entropy loss across all time resolutions. This value 499 is backpropagated to the torso neural network 402 and the wrist neural network 404 to train both networks. This backpropagation of error improves the ability of both networks to generate features suitable for determining whether the raw sensor data represents a matched or mismatched activity. Training is performed on the entire batch of training data, including matched and mismatched data windows. Training continues until the comparison network 460's prediction error reaches a target performance or reaches a maximum plateau. Methods for determining when training is complete are known in the art and are applicable to the present invention.

[0127] Another aspect is that after the wrist neural network 404 and the torso neural network 402 are trained, a small amount of labeled data (e.g., gait data) is used to build a classifier using features from the torso and wrist convolutional networks 402 and 404. This classifier is trained by upsampling all temporal resolution features to 1 second. For example, layer 420 (60x10 feature values) already has 60 sets of 10 values, so upsampling is not necessary. In layer 421 (30x10 feature values), 30 sets of 10 values ​​are upsampled, with each adjacent value repeated once. In layer 424 (4x25 feature values), 4 sets of 25 values ​​are repeated 14 times, creating 4 groups of 15 sets of 25 values. This process is performed for all temporal resolutions, resulting in a 140-dimensional vector for each second. The 140 feature values ​​consist of feature values ​​of 10, 10, 15, 20, 25, 30, and 30. These feature vectors are used to classify the accelerometer data every second into "walking" or "non-walking" using a conventional random forest classifier.

[0128] Referring to Figure 5, another approach to training and using neural networks is shown. A torso feature neural network 502, a wrist feature neural network 504, and a comparison neural network 506 are utilized. The neural networks 502, 504, and 506 are similar to those described elsewhere herein. This approach includes a first training mode, a second training mode, and a supervision mode (executed after the first and second training modes).

[0129] In a first training mode, unlabeled accelerometer signals 503 from sensors worn on a human's torso (e.g., chest) are applied to a torso feature neural network 502. Similarly, unlabeled accelerometer signals 505 from sensors worn on a human's wrist are applied to a wrist feature neural network 504 ("match" for the same person and same time frame, "mismatch" for different time frames or different people). The torso feature neural network 502 generates features 507, and the wrist feature neural network 504 generates feature vectors 509. These feature vectors represent the characteristics of the human's activity within the respective time windows.

[0130] The feature vectors 507 and 509 are input to a comparison neural network 506. The comparison neural network 506 uses a cost function to determine whether the vectors are a match or a mismatch. The degree of match or mismatch is represented as an error (e.g., a numerical value). This value is applied to the neural networks 502 and 504, which adjust the parameters by subtracting a small multiple of the partial derivative of each parameter. The cost function penalizes incorrect determination of vector match / mismatch (step 511). This backpropagation process updates the parameters of the neural networks 502 and 504 independently, resulting in a trained torso feature neural network 502 and a trained wrist feature neural network 504. Both networks improve their ability to generate features suitable for determining whether the activity measured by the sensors is a match or a mismatch.

[0131] In a first training mode, neural networks 502 and 504 are trained to generate feature vectors 507 and 509. Once the first training mode is complete, in a second training mode, the trained torso feature neural network 502 and / or the trained wrist feature neural network 504 are used to generate feature vectors. In this example second training mode, the trained wrist feature network generates feature vectors 522, which are used as input for classifier 520, which is trained in a supervised procedure. The feature vectors 522 are used, along with corresponding activity labels 524, to train a conventional classifier. This classifier 520 can be implemented in various forms, such as random forests, neural networks, logistic regression, or support vector machines. The classifier 520 is trained to estimate activity labels 528.

[0132] The second mode step is effective for training the classifier 520 (generating a completely independent classifier). During this time, the networks 502 and 504 are frozen. However, instead of training the classifier 520, it is also possible to fine-tune the trained networks 502 and 504 and directly estimate the activity classification. One classifier can be used for wrist features and another for torso features. Furthermore, a classifier such as the classifier 520 can be trained using feature data from multiple sensors, in which case feature data from both the networks 502 and 504 can be used as input. In combination with the activity label 524, this approach is intended for activity recognition in a supervised mode (described below). For example, it can be applied when a subject is wearing a chest patch sensor and a wrist sensor simultaneously.

[0133] In the monitoring mode, new accelerometer signals are acquired from different subjects, a feature vector is generated, and then an underlying activity class is estimated. More specifically, a trained wrist feature neural network 504 and a trained classifier 520 are used to classify human activities. New accelerometer signals 530 acquired from the monitored subject are applied to the trained wrist feature neural network 504 to generate features 532. The generated features 532 are applied to the trained classifier 520 to output a classification 531 for the human activity in the signal 530. New accelerometer signals can also be acquired from a different subject's torso sensor and applied to the trained torso network to obtain feature vectors, which can then be applied to the trained classifier 520 to estimate the activity class.

[0134] Referring to Figure 6A, an example of the generation of feature vectors by a trained network is shown, which shows accelerometer data 602 being applied to a neural network 604 (e.g., a wrist or torso neural network). Through this network, features 606 are generated and represented as vectors.

[0135] The features 606 are output at multiple time scales to generate vectors 620, 622, 624, 626, 628, 630, and 632, which capture both high-level and low-level information about the underlying activity. In this implementation, the feature vectors 620, 622, 624, 626, 628, 630, and 632 represented at each time scale have different shapes and must be combined into vector 634. Vector 634 is applied to a random forest classifier 638 or other trained classifier to output an activity classification 640.

[0136] Feature vector 620 represents features every 1 second and has a depth of 10. Feature vector 622 represents features every 2 seconds and has a depth of 10. Feature vector 624 represents features every 4 seconds and has a depth of 15. Feature vector 626 represents features every 8 seconds and has a depth of 20. Feature vector 628 represents features every 15 seconds and has a depth of 25. Feature vector 630 represents features every 30 seconds and has a depth of 30. Feature vector 632 represents features every 60 seconds and has a depth of 30. These depths add up to (10 + 10 + 15 + 20 + 25 + 30 + 30) a depth of 140, and each 1-second time window has vector 634.

[0137] In one example, the features in feature vector 620 are repeated such that each feature occurs every second. For features per second, this means there is no repetition, but features per minute are repeated 60 times by feature vector 632. As a result, vector 634 is generated, which provides a feature vector of length 140 per second for 60 seconds of accelerometer input.

[0138] Referring to FIG. 6B, an overview of the monitoring mode process is shown. This process can be seen in the following screenshot of the monitoring mode steps: Accelerometer data 602 (e.g., 60 seconds of acceleration input) is applied to a neural network 604. Seven different resolution features (vectors 620, 622, 624, 626, 628, 630, and 632) are generated and output as feature vectors. These features are combined to generate output vector 634. Output vector 634 is applied to a random forest classifier 638 to generate an activity classification decision. This means that output vector 634 represents information across all time scales at a particular point in time, generating a final activity classification 640. In this example, the activity classification is binary, with one value (e.g., 1) indicating one activity (e.g., walking) and another value (e.g., 0) indicating another activity (e.g., stationary). However, this is just an example and is not limited to binary classification; any number of classification levels are possible.

[0139] An example of matched and mismatched data is shown in Figure 7. The figure shows data acquired simultaneously from a wrist accelerometer and a trunk accelerometer, and illustrates how pairs of data can be selected for the same time frame (match) or different time frames (mismatch).

[0140] The matched data 702 includes torso sensor data 704 and wrist sensor data 706. The torso sensor data 704 and wrist sensor data 706 were acquired at the same time or within the same time frame and represent the same activity of the same person.

[0141] Meanwhile, the mismatched data 708 includes torso sensor data 710 and wrist sensor data 712. The torso sensor data 710 and wrist sensor data 712 were not acquired at the same time or in the same time frame, and do not necessarily represent the same activity of the same person. The matched and mismatched data pairs are used to train a feature generation network, with "match" and "mismatch" serving as labels in this training. This training does not require prior knowledge of the activity classification (class), or even whether the activities in the mismatched input data are similar. The feature generation network is trained to optimize the comparison network for accurate match prediction. During this process, useful features are automatically extracted to help the classifier improve the accuracy of activity classification.

[0142] A specific application of these approaches is shown with reference to Figure 8. The example in Figure 8 quantifies the health effects of a control group receiving either a first intervention or no intervention, and a test group receiving a second intervention, in a clinical trial.

[0143] A pharmaceutical company is attempting to prove the efficacy of a treatment to improve the quality of life of patients with heart failure. Two groups are involved: a treated group 802 and an untreated group 804. Subjects in both groups are monitored using wrist-worn activity monitors or sensors 806. Accelerometer data 808 from the wrist device 806 is streamed to a cloud 810 and processed using the approach described herein. The processing results in a detailed measurement of the subject's activity. This cloud 810 contains a platform including control circuitry, memory, a trained neural network, and a trained classifier, which together function as a model 812. The accelerometer data 808 is applied to the trained neural network to generate a feature vector. This feature vector is then applied to the trained classifier to perform activity classification. The final results are displayed as a graph 814. The approach of the present invention allows for much more detailed activity classification than previous methods, providing greater insight into the subject's activity.

[0144] At the end of the study, the activity characteristics of each group will be analyzed, and the advanced detection capabilities of the present approach will enable clearer indications of differences in gait speed, gait stability, and increased number of sit-to-stand transitions, demonstrating the effectiveness of new treatments.

[0145] Other use cases for these approaches are possible, and some examples are provided below. These examples utilize the cloud-based platform described above. An additional example is a method for determining possible health changes in a monitored subject. For example, a patient is discharged from a hospital, enrolled in a cloud-based platform, and provided with a wrist-worn activity monitor or sensor. Accelerometer data from the wrist device is streamed to the cloud-based platform, which automatically analyzes the signals using the inventive approach to generate an activity classification. The discharging clinician monitors the activity classification generated by these approaches daily, hoping that the patient will gradually increase their activity level as they recover.

[0146] As another example, clinicians can be alerted to investigate the health status of monitored subjects. Building on the post-discharge management scenario described above, clinicians can set up an alert for recovering patients if they do not walk at least 20 minutes per day for one week after discharge. When the clinician receives this alert, they can contact the patient and suggest changes to their medical care.

[0147] Furthermore, these approaches can be used to selectively control the activation or deactivation of devices. For example, they can operate or set parameters on medical equipment related to the treatment or monitoring of the monitored subject. A specific example is a patient with Parkinson's disease who is enrolled in a cloud-based platform and monitored with a wrist-worn activity monitor. Accelerometer data is streamed to the cloud-based platform and analyzed according to current approaches, and activity classification is used to detect signs of tremor or a worsening gait pattern. If the system detects a worsening gait pattern, the patient's medication can be automatically adjusted. It is also possible to adjust settings or change parameters on medical monitoring devices. These devices are controlled via electronic control signals generated by control circuits or other processing units.

[0148] It may also be possible to control the operation or parameter settings of a user electronic device related to the treatment or monitoring of the monitored subject. For example, a patient undergoing physical therapy may be enrolled in the platform and monitored with a torso-worn activity monitor. Accelerometer data is streamed to the platform and analyzed by the model to determine whether the patient is performing the physical therapy movements correctly. Based on the analysis, additional instructions or encouragement may be sent to the patient's mobile phone to ensure the patient is performing the movements correctly. In one aspect, graphics or icons on the electronic device may be moved, controlled, or modified to provide instructions or encouragement in a specific pattern or manner.

[0149] Preferred embodiments of the present disclosure are described herein, including the best mode contemplated by the inventors, however, the embodiments set forth herein are illustrative only and should not be construed as limiting the scope of the appended claims.

Claims

1. A method for determining and implementing appropriate lifestyle improvement actions for human beings, This involves iteratively training a first neural network to obtain a first trained neural network, and iteratively training a second neural network to obtain a second trained neural network. The first neural network receives a first set of wearable sensor data, and the first neural network responsively generates a first feature vector, wherein the first wearable sensor data describes a first physiological characteristic of a human being, and any labels included in the first set of wearable sensor data are ignored. The second neural network receives a second set of wearable sensor data, and the second neural network responsively generates a second feature vector, wherein the second wearable sensor data describes a second physiological characteristic of the human, and any labels included in the second set of wearable sensor data are ignored. At least some of the wearable sensor data from the first set and the wearable sensor data from the second set are acquired from the same person, from the same activity, at the same time. In a comparative neural network, the prediction is whether the first feature vector from the first neural network and the second feature vector from the second neural network match or mismatch, and it is determined that they match if the first and second feature vectors were obtained from the same person and at substantially the same time. The acquisition is performed by backpropagating the errors generated by a cost function to the first neural network and the second neural network, wherein the cost function penalizes the case where it fails to correctly determine whether the feature vectors are a match or a mismatch, and the backpropagation independently updates the parameters of the first neural network and the second neural network, optimizes the feature generation by the first neural network and the second neural network, and optimizes the prediction success rate of the comparison neural network; Deploying the first trained neural network; Monitoring the subject to acquire current wearable sensor data, and applying the current wearable sensor data to the first trained neural network to obtain current feature vectors; In a trained classifier, mapping the current feature vector to a classification representing one or more activity classes; and Based on the above classification, the following groups: In clinical trials, quantifying health effects between a control group receiving the first intervention and a test group receiving the second intervention; To determine the potential for health changes in subjects under surveillance; To alert clinicians to investigate the health status of monitored subjects; Selectively controlling the activation or deactivation of a device; Controlling the operation of medical devices related to the treatment or monitoring of monitored subjects, or setting their parameters; and Controlling the operation of user electronic devices related to the treatment or monitoring of monitored subjects, or setting their parameters. A method that involves performing one or more actions selected from a set of options.

2. The method according to claim 1, wherein the mapping utilizes known and labeled feature vectors obtained from other monitored subjects.

3. The method according to claim 1, wherein the sensor is a wrist sensor or a chest sensor.

4. The method according to claim 1, wherein the user electronic device is a smartphone, a personal computer, a laptop computer, or a tablet.

5. A method according to claim 1, wherein the trained classifier includes a random forest or a third neural network.

6. The method according to claim 1, wherein the first neural network and the second neural network are trained at a central location.

7. A method according to claim 1, wherein the comparison network comprises a plurality of separate comparison networks of different time scales having a fixed input window size.

8. A system for determining and implementing appropriate lifestyle improvement actions for human beings, The first neural network and, The second neural network, The first neural network and the second neural network are coupled to a comparison neural network, The first neural network is configured to receive a first set of wearable sensor data, the first neural network responsively generates a first feature vector, the first wearable sensor data describes a first physiological feature of a human, and any labels included in the first set of wearable sensor data are ignored. The second neural network is configured to receive a second set of wearable sensor data, the second neural network responsively generates a second feature vector, the second wearable sensor data describes a second physiological feature of the human, and any labels included in the second set of wearable sensor data are ignored. At least some of the wearable sensor data from the first set and the wearable sensor data from the second set are acquired from the same person, from the same activity, at the same time. The comparison neural network is configured to predict whether the first feature vector from the first neural network and the second feature vector from the second neural network match or mismatch, and it is determined that the first and second feature vectors match if they were obtained from the same person and at substantially the same time. The error generated by the cost function is backpropagated to the first neural network and the second neural network, the cost function penalizes the network if it fails to correctly determine whether the feature vector is a match or a mismatch, the backpropagation independently updates the parameters of the first neural network and the second neural network, and generates the first trained neural network and the second trained neural network. The system further includes a trained classifier and a wearable sensor, the wearable sensor being worn by the current subject, The first trained neural network is deployed, and the current wearable sensor data acquired from the current subject by the wearable sensor is applied to the first trained neural network to obtain the current feature vector. In the trained classifier, the current feature vector is mapped to a classification representing one or more activity classes, and based on the classification, the following groups are formed: In clinical trials, quantifying the health effects between a control group receiving the first intervention and a test group receiving the second intervention; To determine the possibility of health changes in the aforementioned current subjects; To alert clinicians to investigate the current health status of the aforementioned subjects; Selectively controlling the activation or deactivation of a device; Controlling the operation of medical devices related to the treatment or monitoring of the current subject, or setting their parameters; and To control the operation of a user electronic device related to the treatment or monitoring of the current subject, or to set its parameters. A system in which one or more actions selected from are performed.

9. The system according to claim 8, wherein the mapping utilizes known and labeled feature vectors obtained from other subjects.

10. The system according to claim 8, wherein the wearable sensor is a wrist sensor or a chest sensor.

11. The system according to claim 8, wherein the first trained neural network is deployed at a central location.

12. The system according to claim 8, wherein the training is performed at a central location.

13. The system according to claim 8, wherein the trained classifier includes a random forest or a third neural network.

14. A system for training neural networks, The first neural network and, The second neural network, A comparison neural network coupled to the first neural network and the second neural network, A control circuit coupled to the first neural network, the second neural network, and the comparison neural network, A sample set of first wearable sensor data obtained from multiple people using a first type of wearable sensor is acquired. A second set of wearable sensor data samples is obtained from multiple people using a second type of wearable sensor, wherein at least some of the samples relate to the same people as the samples in the first set of samples, and were obtained within approximately the same time window. The following process: Inputting the samples from the first sample group into the first neural network to generate features; Inputting the samples from the second sample group into the second neural network to generate features; The features from the first neural network and the features from the second neural network are input to the comparison neural network, and the comparison neural network responsively predicts whether the input samples are a match or a mismatch; and The error is backpropagated to the first neural network and the second neural network, where the error is generated by a cost function that penalizes the failure to correctly determine whether the input sample is a match or a mismatch; the backpropagation independently modifies the neural parameters of the first neural network and the second neural network, optimizing the feature generation by the first neural network and the second neural network, and optimizing the prediction success rate of the comparison neural network; A control circuit is configured to train the first neural network and generate a first trained neural network by repeatedly performing the following: A system that includes this.

15. The system according to claim 14, further comprising a trained classifier, The first trained neural network is deployed, and wearable sensor data is acquired from the monitored subject. The acquired wearable sensor data is applied to the first neural network, and the first neural network generates a feature set corresponding to the time window of the wearable sensor data. The trained classifier maps the generated features to one or more activity classes in the time window. Based on the above classification, the following groups: In clinical trials, quantifying the health effects between a control group receiving the first intervention and a test group receiving the second intervention; To determine the possibility of health changes in the subjects under surveillance; To alert clinicians to investigate the health status of the subjects under surveillance; Selectively controlling the activation or deactivation of a device; Controlling the operation of medical devices related to the treatment or monitoring of the subject being monitored, or setting their parameters; and A system in which one or more actions are performed, selected from controlling the operation of a user electronic device related to the treatment or monitoring of the subject being monitored, or setting its parameters.

16. The system according to claim 15, wherein the sensor is a wrist sensor or a chest sensor.

17. The system according to claim 15, wherein the trained classifier includes a random forest or a third neural network.

18. The system according to claim 15, wherein the training is performed at a central location.

19. A method for training a neural network, To obtain a sample set of first wearable sensor data acquired from multiple people using a first type of wearable sensor, The method involves obtaining a second set of wearable sensor data samples acquired from multiple individuals using a second type of wearable sensor, wherein at least some of the samples relate to the same individuals as the samples in the first set of samples, and were acquired within approximately the same time window. The following process: Inputting the samples from the first sample group into the first neural network to generate features; Inputting the samples from the second sample group into a second neural network to generate features; Inputting the features from the first neural network and the features from the second neural network into a comparison neural network to predict whether the input samples match or mismatch; and The error is backpropagated to the first neural network and the second neural network, the error is generated by a cost function that penalizes the network if it fails to correctly determine whether the input sample is a match or a mismatch, the backpropagation independently updates the neural parameters of the first neural network and the second neural network, improving the feature generation by the first neural network and the second neural network, and improving the prediction success rate of the comparison neural network. The first neural network is trained and the first trained neural network is generated by repeatedly performing the above, Methods that include...

20. The method according to claim 19, Deploying the first trained neural network; Acquiring wearable sensor data from subjects being monitored; Applying the acquired wearable sensor data to the first trained neural network, and generating a feature set corresponding to the time window of the acquired wearable sensor data by the first trained neural network; and A trained classifier maps the features to one or more activity classes; and based on the classification, the following groups: In clinical trials, quantifying the health effects between a control group receiving the first intervention and a test group receiving the second intervention; To determine the potential for health changes in subjects under surveillance; To alert clinicians to investigate the health status of monitored subjects; Selectively controlling the activation or deactivation of a device; Controlling the operation of medical devices related to the treatment or monitoring of monitored subjects, or setting their parameters; and A method further comprising performing one or more actions selected from controlling the operation of a user electronic device related to the treatment or monitoring of a monitored subject, or setting its parameters.

21. A method according to claim 20, wherein the first wearable sensor and the second wearable sensor are a wrist sensor or a chest sensor.

22. A method according to claim 20, wherein the trained classifier includes a random forest or a third neural network.

23. The method according to claim 20, wherein the training is performed at a central location.

24. It is a method, Acquiring wearable sensor data from subjects being monitored; A first neural network generates a feature set corresponding to the time window of the wearable sensor data; The trained classifier maps the generated feature set to one or more activity classes, and based on the activity classes, groups the following: In clinical trials, quantifying the health effects between a control group receiving the first intervention and a test group receiving the second intervention; To determine the potential for health changes in subjects under surveillance; To alert clinicians to investigate the health status of monitored subjects; Selectively controlling the activation or deactivation of a device; Controlling the operation of medical devices related to the treatment or monitoring of monitored subjects, or setting their parameters; and Controlling the operation of user electronic devices related to the treatment or monitoring of monitored subjects, or setting their parameters; This includes performing one or more actions selected from, The first neural network described above follows these steps: Obtaining a sample set of first wearable sensor data acquired from multiple people using a first type of wearable sensor; A second set of wearable sensor data samples is obtained from multiple people using a second type of wearable sensor, wherein at least some of the samples relate to the same people as the samples in the first set of samples, and were obtained within approximately the same time window; The following process: Inputting the samples from the first sample group into the first neural network to generate features; Inputting the samples from the second sample group into a second neural network to generate features; The features from the first neural network and the features from the second neural network are input into a comparison neural network to predict whether the input samples match or mismatch; The first neural network is trained by backpropagating errors to the first and second neural networks, where the errors are generated by a cost function that penalizes the network if it fails to correctly determine whether the input samples are a match or a mismatch, and the backpropagation is performed iteratively to independently update the neural parameters of the first and second neural networks and improve the prediction success rate of the comparison neural network. A method generated by

25. A method according to claim 24, wherein the wearable sensor data is obtained from a wrist sensor or a chest sensor.

26. A method according to claim 24, wherein the trained classifier includes a random forest or a third neural network.

27. The method according to claim 24, wherein the training is performed at a central location.

Images