Infant monitoring method and system
By attaching motion sensors to infant clothing or absorbent items, using predictive models to analyze inertial measurement results, the problem that existing systems cannot accurately distinguish infant activities is solved, and safe and accurate infant surveillance and food detection are achieved.
Patent Information
- Application Number
- CN201910371922.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2019-05-06
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2039-05-06
AI Technical Summary
Existing infant surveillance systems cannot accurately distinguish infant activities, such as deep sleep and light sleep, and traditional sensors require skin contact and are easily kicked off by the baby, and camera-based systems cannot accurately measure infant movement.
Using motion sensors, such as an accelerometer or gyroscope, collects the infant's inertia measurements through wireless communication, and uses predictive models, combining training data and loss functions, iteratively adjusts the model to determine the infant's activities such as sleeping, awakening, swaying or eating. , The sensor is attached to baby clothing or absorbent articles to avoid skin contact.
It realizes the precise classification of infant activities, distinguishes between deep sleep and shallow sleep, and the sensor is safe and reliable, does not affect the normal activities of the infant, and provides detailed activity reports and eating information.
Smart Images

Figure CN110432908B_ABST
Abstract
Description
Technical Field
[0001] The present application generally relates to electronically detecting activities, but more specifically to sensing and activity classification of infants. In a specific example, an activity classification system senses that an infant wearing a sensor is performing a specific activity such as sitting up or sleeping. Background Art
[0002] The overall health and well-being of an infant is of primary concern to parents and other caregivers. Additionally, adjusting a newborn to sleep and feeding cycles can be difficult and often stressful for new parents. Therefore, it is useful to identify and record the infant's activities, such as whether the infant is sleeping, sitting up, or feeding.
[0003] Solutions exist for tracking adults' activities. For example, fitness and sleep trackers can track exercise such as steps, sleep cycles, etc. However, these solutions are not suitable for the unique requirements of infants. For example, adult fitness trackers are not suitable for infants to wear.
[0004] Existing baby monitoring solutions also have flaws. For example, solutions that simply send audio from the baby's room to the parent cannot measure activity or sleep. Some products measure the baby's blood oxygenation, alerting the parent if the baby's blood oxygen level falls below a certain threshold. But these solutions require attaching bulky sensors to the baby, which requires skin contact and can be kicked off. Camera-based baby monitoring solutions also exist, but such camera-based systems simply indicate the presence of the baby. Some camera-based solutions use artificial intelligence, but still cannot distinguish between large and small movements. As a result, these solutions lack the precision to determine the baby's activity, such as whether the baby is in light or deep sleep. Summary of the invention
[0005] Various examples of using motion sensors to detect infant activity are described. In the examples, an activity classification system includes a sensor configured to measure an infant's activity and a monitor. An accelerometer or gyroscope detects a set of inertial measurements in three dimensions over a period of time. The monitor receives the set of inertial measurements. The monitor calculates a variance of the inertial measurements or a root mean square of the inertial measurements. The monitor provides the inertial measurements and the variance or root mean square of the inertial measurements to a prediction model. The monitor receives an activity determined from a set of predefined activities based on the measurements from the prediction model. The monitor provides the determined activity to an output device.
[0006] In one aspect, the statistical data includes (i) a variance of the inertial measurements, or (ii) a root mean square of the inertial measurements.
[0007] In one aspect, the predefined activities include one or more of (i) sleeping, (ii) awake, (iii) lying face down, (iv) lying face up, (v) settled, or (vi) stirring.
[0008] In one aspect, the predefined activities include (i) awake, (ii) sleeping, (iii) rocking, or (iv) peaceful.
[0009] In one aspect, the monitor further adjusts the determined activity. For example, when the determined activity is calm, the monitor changes the determined activity to a different state. When the determined activity is rocking, the monitor changes the determined activity to sleeping. When the determined activity is (i) sleeping, (ii) below a time threshold, and (iii) occurring during the day, the monitor changes the determined activity to awake. When the determined activity is (i) awake, (ii) below a time threshold, and (iii) occurring during the night, the monitor changes the determined activity to sleeping.
[0010] In one aspect, the predefined activities include (i) whether the baby is breastfeeding, (ii) whether the baby is bottle-feeding, or (iii) the orientation of the baby during feeding. The monitor also receives orientation or angular velocity measurements from the gyroscope. The monitor provides the orientation or angular velocity measurements to the prediction model. The determined activities are based at least in part on the angular velocity measurements.
[0011] In one aspect, a monitor receives training data comprising time series data indicating inertial measurements measured during a time period. The training data is annotated with training labels indicating corresponding ones of the activities for each time period. The monitor further iteratively adjusts the prediction model based on a loss function based on a comparison of the predictions to the corresponding training labels. Iteratively adjusting the prediction model includes determining, by the prediction model, whether data corresponding to the time period corresponds to a prediction of one of the activities, and modifying the prediction model so that a subsequent value of the loss function is reduced in a subsequent iteration.
[0012] In one aspect, the monitor also receives additional training data including time series data from the monitor device, the time series data indicating inertial measurement results of the measurement results in the additional time period. The additional training data can be annotated with training labels indicating corresponding activities in the activities. The monitor also updates the prediction model based on the additional training data and the loss function. The updating includes determining, by the prediction model, whether the data corresponding to the time period corresponds to a prediction of one of the activities. The updating also includes reducing the value of the loss function based on a comparison of the prediction with the training labels.
[0013] In one aspect, the monitor also receives training data, the training data comprising time series data indicating inertial measurements measured over time periods. The training data is annotated with training labels indicating, for each time period, whether the infant is awake or asleep. The monitor also trains a prediction model based on the training data and a loss function. The training can include: determining, by the prediction model, a prediction of (i) awake, (ii) asleep, (iii) rocking, or (iv) calm, and minimizing the loss function based on a comparison of the predictions with the training labels.
[0014] In one aspect, the monitor further stores in a memory time series data for a set of time periods. The monitor determines a set of activities based on the predictive model. Each activity corresponds to a different one of the time periods. In response to determining that a predetermined time threshold has passed, the monitor generates a log report and displays the log report on a user interface, the log report including the determined activities for each time period.
[0015] In another aspect, the monitor receives a software update including a new prediction model via the wireless transceiver. The monitor further updates the prediction model using the new prediction model.
[0016] In one aspect, an infant monitoring device is configured to determine an activity of an infant. The infant monitoring device includes an inertial measurement sensor and a processor. The processor is configured to perform operations. The operations include receiving three-dimensional inertial measurements for a time period from the inertial measurement sensor. The operations include calculating statistics derived from the inertial measurements. The operations include providing the inertial measurements and the statistics to a prediction model. The operations include receiving a breathing rate of the infant from the prediction model and based on the inertial measurements. The operations include providing the breathing rate to an output device.
[0017] In one aspect, the operations further include receiving, from the predictive model, a volume of liquid consumed by the infant during a feed.
[0018] In one aspect, the operations further include receiving from the predictive model an estimate of (i) the time until the next meal or (ii) the volume of liquid expected to be consumed at the next meal.
[0019] In one aspect, an inertial measurement sensor is attachable to and removable from an absorbent article.
[0020] In one aspect, the inertial measurement sensor is configured to be attached to the absorbent article rather than to the infant's body.
[0021] In one aspect, an infant monitoring system includes a monitoring device. The monitor device includes a first wireless transceiver and a first processor. The first processor is configurable to run instructions that cause the first processor to perform operations. The operations include receiving acceleration measurements from the first wireless transceiver of the sensing device. The operations also include providing the acceleration measurements to a prediction model. The operations also include receiving a determined activity from predefined activities from the prediction model and based on the acceleration measurements. The operations also include providing the determined activity to an output device. The sensing device includes: an accelerometer configured to sense acceleration in at least one degree of freedom; a second wireless transceiver; and a second processor. The second processor is configured to run instructions that cause the second processor to receive acceleration measurements from the accelerometer and provide the acceleration measurements to the first wireless transceiver.
[0022] In one aspect, the output device is external to the monitor device. Additionally, providing the determined activity to the output device includes sending the determined activity to the output device via a data network connection.
[0023] In one aspect, providing the acceleration measurements to the prediction model includes sending the acceleration measurements to a server device over a network. Receiving the determined activity also includes receiving the determined activity from the server device over the network.
[0024] In one aspect, the first processor may also be configured to execute instructions that store time series data for time periods in a memory, determine activities based on a predictive model, each activity corresponding to a different one of the time periods; and in response to determining that a predetermined time threshold has passed, generate a log report including the activities and the corresponding time periods.
[0025] In one aspect, the first processor may be further configured to execute instructions that cause the first processor to send the acceleration measurements over a data network to a server device configured to record the measurements from the infant monitoring system.
[0026] In one aspect, the predefined activities include (i) awake, (ii) sleeping, (iii) rocking, or (iv) calm. In one aspect, the first processor also adjusts the determined activity. For example, when the determined activity is calm, the first processor changes the determined activity to a different state. When the determined activity is rocking, the first processor changes the determined activity to sleeping. When the determined activity is (i) sleeping, (ii) below a time threshold, and (iii) occurring during the day, the first processor changes the determined activity to awake. When the determined activity is (i) awake, (ii) below a time threshold, and (iii) occurring during the night, the first processor changes the determined activity to sleeping.
[0027] In another aspect, the infant monitoring system includes an additional sensor. The first processor may be configured to execute instructions that cause the first processor to perform operations. The operations include receiving additional measurements of (i) a change in conductivity, (ii) a change in an optical property, (iii) a change in capacitance, (iv) a change in color, or (v) a change in a detected volatile organic compound from the additional sensor. The operations also include sending the additional measurements to the monitoring device via the first wireless transceiver. The sensing device may be configured to execute instructions that cause the second processor to receive the additional measurements from the first wireless transceiver via the second wireless transceiver; determine an indication of the presence of (i) urine or (ii) a bowel movement from the additional measurements; and provide the indication to an output device.
[0028] In one aspect, a wearable device includes a processor, an optical sensor, and an inertial sensor. The optical sensor is configured to obtain a color measurement result and send the color measurement result to the processor. The inertial sensor is configured to detect the inertial measurement result and provide the inertial measurement result to the processor. The processor is configured to perform operations. The operations include receiving the inertial measurement result from the inertial measurement sensor. The operations also include providing the inertial measurement result to a prediction model. The operations also include receiving an activity determined from predefined activities based on the inertial measurement result from the prediction model. The operations also include determining a wetness indication in the absorbent article from the color measurement result. The operations also include providing the determined activity and wetness indication to an output device.
[0029] In one aspect, a wearable device includes a battery operable to power a processor and an inertial measurement sensor.
[0030] Mentioning these illustrative embodiments is not intended to limit or define the scope of the present disclosure, but to provide examples to help understand the present disclosure. Illustrative examples are discussed in the detailed description, which provides further description. By examining this specification, the advantages provided by various examples can be further understood. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] The features, embodiments, and advantages of the present disclosure will be better understood when the following detailed description is read with reference to the accompanying drawings.
[0032] Figure 1 Depicted is a block diagram of an activity classification system in accordance with certain aspects of the present disclosure.
[0033] Figure 2 Depicted are examples of sensor output from an accelerometer in accordance with certain aspects of the present disclosure.
[0034] Figure 3 is a flow chart of an example method for determining activity from a motion sensor in accordance with certain aspects of the present disclosure.
[0035] Figure 4 is a flow chart of an example method for determining activity from a motion sensor by using a predictive model, in accordance with certain aspects of the present disclosure.
[0036] Figure 5 Depicted is a graph showing activity determined by a predictive model based on motion detected from an accelerometer, in accordance with certain aspects of the present disclosure.
[0037] Figure 6 Depicted are examples of sensor output from an accelerometer annotated with predicted states in accordance with certain aspects of the present disclosure.
[0038] Figure 7 Depicted are examples of sensor output from an accelerometer indicating sleep and wakefulness, in accordance with certain aspects of the present disclosure.
[0039] Figure 8 Depicted are examples of sensor output indicative of a detected infant's breathing rate, in accordance with certain aspects of the present disclosure.
[0040] Fig. 9 is a flow chart of an exemplary method for training a predictive model to determine activity from motion according to certain aspects of the present disclosure.
[0041] Fig.10 is a state diagram of an exemplary state machine for determining an activity level in accordance with aspects of the present disclosure.
[0042] Fig.11 is a flow chart of an exemplary method performed by a smoothing filter according to certain aspects of the present disclosure.
[0043] Figure 12-Figure 15 is a diagram illustrating exemplary operations performed by a smoothing filter according to aspects of the present disclosure.
[0044] Fig.16 and Fig.17 Depicted are exemplary user interfaces for viewing an infant's sleep patterns in accordance with certain aspects of the present disclosure.
[0045] Figure 18-Figure 20 Depicted are exemplary user interfaces for viewing an infant's feeding patterns in accordance with certain aspects of the present disclosure.
[0046] Fig.21 Depicted is a block diagram of a processor configured to determine activity of a wearer of a sensor, in accordance with certain aspects of the present disclosure. DETAILED DESCRIPTION
[0047] Aspects described herein provide solutions for activity classification systems, particularly activity classification systems designed to detect an infant's movements and determine whether the infant is performing a specific activity such as sleeping, feeding, nursing, or sitting up.
[0048] In an example, an activity classification system includes a motion sensor and an external monitor. The sensor is attached to baby clothing or an absorbent article such as a diaper. The sensor, which can include an accelerometer or a gyroscope, provides measurements to the monitor via a wireless communication channel. The monitor is an electronic device that can receive measurements from the sensor and run a monitor application and a prediction model (such as a machine learning model, a state flow model, or an algorithm) to determine the activities performed by the baby. In this example, the prediction model is trained to determine the activities that the baby is performing based on the baby's movements, such as sleeping or sitting up. The activity classification system can then indicate the predicted baby's activities to the operator of the monitor, such as the baby is in deep sleep.
[0049] Various aspects of the present disclosure provide certain advantages over existing solutions. For example, using precise motion collected from an accelerometer or gyroscope, an activity classification system can distinguish between activities that an infant is performing. For example, an activity classification system can be configured (such as trained) to distinguish between deep sleep and light sleep, whether an infant is lying face down or face up, or whether an infant is breastfeeding. In addition, certain aspects described herein can use predictive models to further refine the system's ability to determine activity. For example, a classification system can distinguish between light sleep and deep sleep or measure an infant's breathing rate.
[0050] Figure 1 A block diagram of an activity classification system according to certain aspects of the present disclosure is depicted. The activity classification system 100 includes components that measure the movement of an infant, and one or more processors that analyze the measurements to determine the infant's activity. The movement can be determined by inertial measurements, including acceleration or angular velocity measurements. The one or more processors use an activity function or predictive model, such as a classifier, to determine the infant's activity.
[0051] The activity classification system 100 includes one or more sensors 101 , a monitor 110 wirelessly connected to the sensor 101 via a wireless network 130 , an activity classification server 180 , or a mobile device 199 .
[0052] In an example, the monitor 110 (which can also be referred to as a hub) can be small enough to be placed near the baby, such as on a table or dresser. The monitor 110 communicates wirelessly with the sensor 101, and optionally with other sensors, such as volatile organic compound sensors, urine moisture sensors, noise sensors, light sensors, conductivity sensors, optical sensors, capacitive sensors, color change sensors, etc. The monitor 110 can aggregate data from the sensor 101 or other sensors.
[0053] The monitor 110 can optionally include a user interface 121, where alarms, status information, or other information can be displayed on the user interface 121. The user interface 121 can include buttons, dials, knobs, touch screens, etc. The monitor 110 can communicate with the mobile device 199 via the data network 150 and cause the mobile device 199 to issue warnings, alarms, or provide other information.
[0054] In one aspect, the user interface 121 can also receive feedback from the user that can help the monitor application 113 better determine when the infant is sleeping. For example, the monitor application 113 can ask the user whether the infant is sleeping, and the user interface 121 can receive an indication from the user as to whether the infant is actually sleeping. Such information can be incorporated into the prediction model as training data and can help the prediction model become more accurate over time.
[0055] In an example, a predictive model is trained to determine an activity an infant is performing, such as sleeping or sitting up, based on the infant's movements.The predictive model can be trained through data collected for an individual or a group of individuals.
[0056] For example, the prediction model can receive training data representing a particular individual. The training data can include the individual's movements determined by the sensor 101 and the associated ground truth (the correct activity is specified by the training label). Therefore, when new data is received from the sensor 101, the trained prediction model predicts the activity. In another example, the prediction model can be trained based on training data collected from a group of individuals, which group of individuals does not need to include the individuals whose activities will be predicted later. Therefore, the prediction model can learn from the recorded behavior of a particular individual or group of individuals and use the learned behavior to predict the activity of the individual or other individuals.
[0057] To measure motion, sensor 101 is placed on or affixed to a person, such as an infant. Sensor 101 includes one or more sensor devices, such as accelerometer 102 or gyroscope 103. Sensor 101 also includes wireless transceiver 104 and processor 105. Sensor 101 collects measurements of motion and sends the measurements to an external device, such as monitor 110. Based on the measurements, monitor 110 can determine the activity the infant is performing and can take action, such as notifying an operator of what activity the infant is performing.
[0058] Ensure that sensor 101 can be implemented in different ways. For example, sensor 101 can be attached to a diaper or clothing. Alternatively, sensor 101 can be pinned or adhered to clothing or absorbent articles or diapers, or fixed to the baby in any other way. Sensor 101 can be attached, removed and reattached, thereby making the sensor reusable. Sensor 101 does not need to contact the baby's skin. Sensor 101 can be inserted into a bag and removed as needed. Sensor 101 can be placed anywhere on the baby's body. But as further discussed, placing sensor 101 near the baby's waist allows certain advantages, such as the ability to determine breathing rate.
[0059] To determine motion, processor 105 receives measurements from one or more sensors, such as accelerometer 102 and gyroscope 103. Processor 105 converts the measurements into a form suitable for transmission over wireless network 130, such as by encoding the measurements into network packets. Processor 105 provides the measurements to wireless transceiver 104, which transmits the measurements to an external device, such as monitor 110.
[0060] The accelerometer 102 measures the acceleration of the infant in one or more dimensions. For example, the accelerometer 102 can be a three-dimensional accelerometer that measures acceleration in the x, y, and z directions. In this case, the accelerometer 102 provides a triplet of values corresponding to the x, y, and z directions.
[0061] Gyroscope 103 measures angular velocity. Gyroscope 103 is capable of outputting a signal proportional to the angular velocity of the baby. The angular velocity changes in the direction of the torque applied to the gyroscope. Therefore, when the baby wearing sensor 101 rolls over, gyroscope 103 detects an increase in angular velocity. When the baby stops rolling over, the angular velocity returns to zero.
[0062] The directional component of the angular velocity can be used in various ways. For example, the direction of the velocity can help indicate which side the baby is placed on, such as left or right, down or up. The processor 105 can sample the gyroscope 103 at a specific moment and periodically obtain the angular velocity.
[0063] The output from the accelerometer 102 or gyroscope 103 can be used by an activity function or predictive model to provide an indication of the baby's activity, breathing rate, or orientation such as which side the baby is breastfeeding or bottle feeding.
[0064] The processor 105 periodically samples the accelerometer 102 and / or the gyroscope 103 to obtain measurements as needed. The processor 105 provides the sampled measurements to the wireless transceiver 104 for transmission to an external device. The processor 105 can sample the accelerometer 102 or the gyroscope 103 at different rates. For example, the processor 105 can use an adaptive sampling rate. For example, when the detected motion level based on the measurements from the accelerometer 102 and the gyroscope 103 is below a threshold, the processor 105 can sample less frequently.
[0065] The reduction in sampling rate can help reduce power consumption by disabling devices such as the wireless transceiver 104 or entering and remaining in a low power state. The processor 105 can also perform additional processing or analysis on the measurements before providing the measurements to the wireless transceiver 104. For example, when the measurements indicate an activity threshold level, the processor 105 can only provide the measurements to the external device via the wireless transceiver 104. In this way, the processor 105 can save power by not causing the wireless transceiver 104 to send data. Conversely, when the measurements indicate activity above a second threshold, the processor 105 can sample the accelerometer 102 and the gyroscope 103 more frequently. In this way, the processor 105 can obtain more detailed measurements during bursts of activity.
[0066] Processor 105 sends the measurements to an external device, such as monitor 110, using wireless transceiver 104. In some examples, processor 105 can send the measurements to an external device, such as monitor 110, as they are received from accelerometer 102 and gyroscope 103, or in some examples, processor 105 can buffer the measurements and send one set of measurements at a time to monitor 110.
[0067] The wireless transceiver 104 is configured to transmit and receive radio communications to enable communication between the sensor 101 and one or more external devices, such as the monitor 110. The wireless transceiver 104 can provide wireless communications using any suitable wireless protocol, such as Bluetooth or WiFi, to send data or other information, such as acceleration measurements or angular velocity measurements from the accelerometer 102, to the monitor 110. In some examples, the wireless transceiver 104 can also send other information, such as a status message indicating whether the sensor 101 is operational, in standby mode, or disabled. In one aspect, the wireless transceiver 104 can receive radio transmissions from external devices. For example, the wireless transceiver can receive information and commands from the monitor 110. For example, the monitor 110 can send a message to the wireless transceiver 104 to turn the sensor 101 on, enter a low power state, or turn it off.
[0068] In this example, monitor 110 includes output device 111, processor 112, monitor application 113, wireless transceiver 114, and prediction model 120. Monitor 110 operates with sensor 101 and optionally with activity classification server 180 to determine the activity that the infant is performing. To do this, monitor 110 receives sensor measurements, such as acceleration and angular velocity, from sensor 101 and uses monitor application 113 in conjunction with prediction model 120 to determine the activity based on the received sensor measurements. In this example, prediction model 120 is a state machine or algorithm, but in different examples may be any suitable type of prediction model, such as a machine learning model or a classification model.
[0069] Wireless transceiver 114 can transmit and receive messages via a wireless protocol such as Bluetooth, WiFi, or any other wireless protocol. To enable wireless communication in this example, wireless transceiver 114 can be paired with a sensor device such as sensor 101. Wireless transceiver 114 receives sensor data obtained from accelerometer 102 or gyroscope 103 from wireless transceiver 104.
[0070] Monitor 110 uses the measurements to determine the activity the infant is performing. For example, monitor 110 can apply the measurements to a predictive model 120, which can determine whether the infant wearing sensor 101 is feeding on the left hand side, feeding on the right hand side, sleeping, awake and lying on his back playing, being held, sitting, or performing some other activity.
[0071] Based on the determined activity, the monitor 110 can take action. For example, the monitor 110 can display a message or sound an alarm to the operator, or the monitor 110 can use an output device 111 to alert the operator of the monitor 110 of the activity, such as whether the infant wearing the sensor 101 is awake or asleep. The output device 111 can be a display such as an LCD or LED display, a touch screen display, a light, a flash, a speaker, or other output device.
[0072] In one aspect, the prediction model 120 can determine the breathing rate of the baby. For example, if the sensor 101 is placed on the waist or abdomen of the baby, the baby's breathing will cause the sensor to move back and forth in one direction, so that the sensor outputs one or more sensor signals indicating movement. The monitor application 113 can determine the breathing rate of the baby based on the measurement results of the activity. The monitor application 113 can also use the activity level to distinguish between small falls and large falls (such as, falls that require intervention). Similarly, the monitor application 113 can determine when the baby's heart beats and thus determine the baby's heart rate. Heart rate and breathing rate can be used to help determine when the baby is in different sleep stages, such as light sleep, deep sleep, about to wake up, etc. The difference between the heart rate or resting heart rate and the active heart rate can also indicate the health of the baby, such as whether the baby has a fever.
[0073] In another aspect, the prediction model 120 can determine future activity from past activity. For example, the prediction model 120 can be trained to recognize trends, such as a time series of sensor values that indicate a period of deep sleep followed by increased levels of movement. Such a trend in a time series of acceleration data can indicate that the baby is waking up. Figure 2 Some examples are shown.
[0074] In one aspect, monitor 110 can connect to activity classification server 180 via data network 150 using wireless transceiver 114. Data network 150 can be a local network, the Internet, or both. For example, wireless transceiver 114 can wirelessly connect to activity classification server 180 or mobile device 199.
[0075] The activity classification server 180 can be configured to perform additional analysis, such as population level analysis of the movements of multiple infants, or training of one or more prediction models and providing the prediction models to the monitor 110 .
[0076] Activity classification server 180 includes prediction model 190, processor 192, and network adapter 194. The network connection from monitor 110 to activity classification server 180 provides several advantages. For example, via data network 150, monitor 110 can receive software updates from a vendor or send status information to a vendor. Software updates can include updates to prediction model 120.
[0077] Prediction model 190 can be a machine learning model, such as a decision tree classifier or a regression model. Prediction model 190 can be trained to perform similar functions as prediction model 120. For example, prediction model 120 can be trained to determine whether the wearer of the sensor is eating on the left hand side, eating on the right hand side, sleeping, awake and lying on his back, being held, or sitting. In particular, the motion detected by gyroscope 103 is useful for predicting eating.
[0078] Updates to the prediction model are possible. For example, the prediction model 190 can also update the prediction model 120. For example, as further explained herein, the prediction model 190 can be trained using data from multiple monitors 110a-n. In this way, the prediction model 190 can become more accurate because it has been trained on a larger set of data than is available from just one monitor. Updating the prediction model 120 using the training of the prediction model 190 provides each monitor 110a-n with the ability to benefit from population-based training.
[0079] The network adapter 194 is connected to a data network, such as the data network 150. In this manner, the network adapter 194 is able to communicate with the monitor 110 via the wireless transceiver 114. The network adapter 194 is thus able to receive updates, such as new training data collected from the monitor 110. The network adapter 194 is also able to provide updated software or trained predictive models to the monitor 110.
[0080] Processor 192 is a processor that receives data from network adapter 194 via data network 150. The processor can be, for example, Fig. 20 192 can be a processor of processor 2102 described in . Processor 192 can perform operations similar to processor 112. For example, processor 192 can run an application such as monitor application 113. Such an application can train a predictive model such as predictive model 190, receive measurement result data via data network 150, and use predictive model 190 to determine the activity of the infant.
[0081] The mobile device 199 can be a smart phone, tablet computer, laptop computer or other device. The mobile device 199 can include a graphical user interface that outputs data such as how long the baby has slept, predicted breastfeeding time, etc. Figure 10-Figure 14The mobile device 199 can transmit an alert to the operator, such as an indication that the infant is awake, asleep, crying, etc. The mobile device 199 can communicate with the monitor 110 or the activity classification server 180 via the data network 150. The mobile device 199 can include some or all of the processing initiated by the execution processor 105, 112 or 192.
[0082] Sensor 101, monitor 110 and activity classification server 180 operate in conjunction with each other to determine the activity of the infant. Processing of the measurement data and determination of the infant's activity can be performed on sensor 101, e.g., via processor 105, on monitor 110, e.g., via processor 112, or on activity classification server 180, e.g., via processor 192.
[0083] For example, processor 105 can receive measurements such as acceleration and angular velocity, and sensor 101 can provide unmodified measurement data to monitor 110 across wireless network 130. In turn, processor 112 receives measurements from sensor 101 via wireless transceiver 114. Processor 112, e.g., via monitor application 113, uses the activity function to determine the infant's activity.
[0084] Alternatively, processor 105 can derive an activity function from the acceleration and angular velocity measurements. Processor 105 provides the activity function to wireless transceiver 104. Processor 112 receives the actimetry function data and makes predictions about the infant's activity.
[0085] In another aspect, the monitor 110 can also provide the measurements from the sensor 101 via the data network 150 to an activity classification server 180 that can determine the infant's activity. The activity classification server 180 can provide additional processing resources to obtain more accurate predictions of the infant's activity, or the activity classification server 180 can update its prediction model 190 based on the received measurements and / or additional training data annotated based on ground truth (e.g., indications of actual activity such as sleeping or being awake).
[0086] In one aspect, the activity classification server 180 is connected to multiple activity monitors 110 and aggregates the measurement data or determines the activity of multiple infants. The activity classification server 180 can analyze the data to obtain trends. Using multiple data sets can more easily discern trends that are difficult to identify. For example, combined with demographic information such as the age of the infant being measured, the activity classification server 180 can determine the average amount of sleep received by an infant of a particular age.
[0087] The activity classification server 180 can also aggregate training data from multiple infants and improve the training of the prediction model 190. The prediction model 190 can be deployed into the prediction model 120 of each monitor 110.
[0088] Figure 2 Depicted are examples of sensor output from an accelerometer in accordance with certain aspects of the present disclosure. Figure 2 Included is a graph 200 that includes waveforms 202, 203, and 204. Waveform 202 represents a measurement of activity in the x-dimension of sensor 101, waveform 203 represents a measurement of activity in the y-dimension of sensor 101, and waveform 204 represents a measurement of activity in the z-dimension of sensor 101. Graph 200 includes a vertical axis indicating acceleration (g), and a horizontal axis indicating time (hours).
[0089] As can be seen in graph 200, there are bursts of activity, such as burst 210, shown as higher values, that are associated with motion detected by sensor 101. A burst of activity may occur when an infant moves in a manner that causes one or more sensors to output a sensor signal indicative of, for example, acceleration. For example, when an infant rolls over, an activity burst is detected because one or more accelerometer outputs an acceleration greater than a threshold away from zero acceleration or in the direction of the roll (in the direction of the roll). Figure 2 angular velocity greater than a threshold away from zero angular velocity (not shown). Similarly, when the infant is resting, the infant's breathing can appear as bursts of activity at the breathing rate. Conversely, during periods without bursts of activity, the infant either does not move or moves very little. As further described, the monitor application 113 can use an activity function or prediction model 120 to distinguish movement.
[0090] Figure 3 300 is a flow chart of an exemplary method for determining activity from a motion sensor according to certain aspects of the present disclosure. For example, the method 300 is shown with respect to the monitor application 113. However, the method 300 can be performed by software running on a processor such as the processor 192 of a remote device such as the activity classification server 180.
[0091] At box 301, the monitor application 113 receives a time series of data including inertial measurements for each time period in a set of time periods from a motion sensor. The inertial measurements can include acceleration or angular velocity. For example, the accelerometer 102 provides a triplet of values corresponding to the x, y, and z directions to the processor 105, and the processor 105 provides the triplet to the wireless transceiver 104. The processor 105 periodically samples the accelerometer 102 to create a time series of data. The processor 105 annotates each triplet with a timestamp, creating a pair including the sensor measurement and the timestamp. The processor 105 can also periodically sample the gyroscope 103. In conjunction with the measurement data from the accelerometer 102, the processor 105 can provide a set of data to the monitor 110. The set of data can include gyroscope measurements (e.g., angular velocity), accelerometer measurements (e.g., a triplet of xyz values), and a timestamp.
[0092] In one aspect, the activity classification system 100 analyzes the measurement data in real time. For example, the processor 105 can cause the wireless transceiver 104 to send a triplet of each sample and a timestamp pair to the monitor 110 in real time. In this way, the monitor application 113 can update the activity measurement function or prediction model 120 in real time.
[0093] Alternatively, the activity classification system 100 can analyze blocks of samples at a time. For example, the processor 105 can buffer the pairs until a threshold number of pairs have been received and then send a batch consisting of all the collected pairs to the monitor 110. The processor 105 can also buffer the pairs until a threshold amount of time has passed and then provide the pairs to the monitor 110. In this way, the processor 105 can analyze motion over different time windows.
[0094] Other types of motion sensors can be used, such as vibration sensors, ultrasonic sensors, or passive infrared sensors. Such sensors can have single or multiple dimensions. The processor 105 can sample such sensors and create a collection of data containing sensor values and time stamps to send to the monitor 110.
[0095] At box 302, the monitor application 113 calculates an activity function from a subset of the time series data based on statistics derived from the inertial measurements. The statistics may include data such as (i) the statistical variance of the inertial measurements or (ii) the root mean square of the inertial measurements. The monitor application 113 uses the activity measurement function to determine the activity level. Different activity measurements can be derived. For example, the monitor application 113 can calculate the statistical variance, standard deviation, or root mean square (RMS) of the signal. The monitor application 113 can use another customized metric based on accelerometer or gyroscope data. For example, a customized metric that quantifies the activity level A can be calculated for a given number n of samples using the following function, where Sx, Sy, and Sz are the sum of the squared differences of the respective means in the x, y, and z dimensions:
[0096]
[0097] At box 303, the monitor application 113 determines the activity indicated by the subset of the time series data based on the metric from the activity function being greater than a first threshold and less than a second threshold. The monitor application 113 can determine activity such as sleep or wakefulness based on activity levels having a range of values. For example, if the activity function measures an activity level below a first threshold but above zero, the monitor application 113 determines that the infant is in light sleep. If the activity function measures a level of movement below a second, lower threshold, the monitor application determines that the infant is in deep sleep. As described above with respect to Fig.10 As further discussed, the monitor application 113 can use a state machine to determine the activity state.
[0098] At block 304, the monitor application 113 provides the activity to a user interface. For example, the monitor application 113 can output an activity such as "deep sleep" or "light sleep," for example, by providing a silent alarm to the output device 111 to indicate to the caregiver that the baby is sleeping. The monitor application 113 can also log the activity and timestamp in a memory and provide the logged information to the activity classification server 180.
[0099] The monitor application 113 can also output information derived from the infant's activity. For example, the output device 111 can provide an indication to the operator such as "the baby has been asleep for two hours." The monitor application 113 can also, for example, cause a sound such as an alarm from the output device 111 when the infant wakes up after sleeping for a predetermined amount of time. The monitor application 113 can also send information to a remote device, such as an activity classification server 180. In this way, the activity classification system 100 can be useful for caregivers of multiple infants, such as in a hospital or daycare.
[0100] As discussed, in one aspect, in addition to or in lieu of an algorithm or state machine, the activity classification system 100 can determine the infant's activity using a predictive model such as the predictive model 120 or the predictive model 190. The monitor application 113 provides the accelerometer measurements, the gyroscope measurements, or the output of the activity measurement function to the predictive model 120, or both.
[0101] The prediction models discussed herein can be machine learning models, such as decision tree classifiers or regression models. Other models are also possible. The prediction model 120 is trained to determine whether the wearer of the sensor is eating on the left hand side, eating on the right hand side, sleeping, awake and lying on his back playing, being held or sitting. Other detectable activities can include sitting, playing, crawling, walking, etc. The monitor application 113 provides data for one or more time periods to the prediction model 120. In this way, the prediction model 120 can determine activity based on current or past activity levels.
[0102] The prediction model 120 can be updated or upgraded, for example, via the data network 150. As discussed further herein, the prediction model 120 can also be trained locally on the monitor 110. For example, an operator, such as a parent, can instruct the monitor 110 that the wearer of the sensor was sleeping or awake during a particular time period. The monitor application 113 can then update the prediction model 120 based on the additional information provided. In this way, the prediction model 120 can be personalized for an individual or infant. Potential benefits include higher accuracy relative to using data from other individuals.
[0103] Figure 4 4 is a flow chart of an exemplary method for determining activity from a motion sensor by using a prediction model according to certain aspects of the present disclosure. For example, the method 400 is shown with respect to the monitor application 113 and the prediction model 120. However, the activity classification server 180 and the prediction model 190 can also perform the method 400.
[0104] At block 401 of method 400, monitor application 113 receives measurements of a period of time from an inertial measurement sensor. At block 401, monitor application 113 receives inertial measurements generally as described with respect to block 301. The measurements can be obtained in one or more dimensions.
[0105] At block 402 of method 400, monitor application 113 calculates an activity function or statistics from the inertial measurements. Examples of activity functions include (i) the statistical variance of the inertial measurements and (ii) the root mean square of the inertial measurements. At block 402, the monitor application uses an activity measurement function as generally described with respect to block 302.
[0106] At block 403 of method 400, the monitor application 113 provides the inertial measurements and / or statistical data to the prediction model. More specifically, the monitor application 113 provides the output of the sensor measurements and / or activity function to the prediction model 120. Fig. 9 The predictive model is trained by the process described. When trained, the predictive model is capable of identifying activities in the activity list based on the training and inertial measurement results or statistical data.
[0107] The prediction model 120 is trained to determine activities from measurements indicative of motion. The prediction model 120 determines which class an activity belongs to from a set of predefined classes based on its training. An exemplary list of activity classes includes eating on the left side, eating on the right side, sleeping, awake but lying up playing, being held, and sitting.
[0108] Other training classes are possible. For example, the prediction model 120 can be trained to distinguish between deep sleep and light sleep, and to distinguish activities such as crawling, rolling, sitting up, eating, or breastfeeding from one another. For example, the monitor 110 can include a prediction model that is trained to distinguish between sleeping, awake, rocking, or calm states, and another prediction model that is trained to distinguish between light sleep and deep sleep. Fig. 9 Training is discussed further. Rocking represents a state where the infant moves more than a first threshold amount and calm represents a state where the infant has calmed down and is moving less than a second threshold amount.
[0109] At block 404 of method 400, the monitor application 113 receives an activity determined based on the inertial measurements from the prediction model. The determined activity can be from a list of predefined activities.
[0110] For example, the prediction model 120 provides a prediction from one of the trained categories to the monitor application 113, such as eating on the left hand side, eating on the right hand side, sleeping, awake and lying on his back playing, being held, or sitting. An illustration of the activities determined by the prediction model is shown in FIG. Figure 5 shown.
[0111] Figure 5 A graph 500 is depicted showing activity determined by a predictive model based on motion detected from an accelerometer and a gyroscope, in accordance with certain aspects of the present disclosure. Graph 500 includes waveform 501, awake state 510, eating state 511, and sleeping state 512. Graph 500 indicates activity determined by a predictive model based on motion detected from an accelerometer and a gyroscope. Figure 2 The sensor information shown in graph 200 determines the activity.
[0112] Waveform 501 represents the output of a predictive model, such as predictive model 120, over time. In the aspect depicted by graph 500, predictive model 120 is trained to determine one of three activities: awake, eating, or sleeping. Graph 500 includes awake state 510 corresponding to awake, eating state 511 corresponding to eating or breastfeeding, and sleeping state 512 corresponding to sleeping. Thus, at a given point in time, the output represented by waveform 501 is in one of states 510-512.
[0113] In the chart 500, the time unit is hours. Thus, it can be seen that the prediction model 120 has predicted that the baby has slept for various time periods ranging from a few hours to seven hours or more. The point indicated by the feeding state 511 represents feeding, which represents a typical feeding time of 20-30 minutes as measured by the prediction model.
[0114] States 510-512 can also represent annotated training information provided to prediction model 120. For example, data indicating infant activity can be obtained via monitor 110 or another device and provided with a predicted state and a user input prompt. For example, monitor 110 can prompt an operator such as a caregiver with a prompt such as "The monitor has determined that the infant is sleeping. True or false?"
[0115] Back to Figure 4 At block 405 of method 400, monitor application 113 provides the determined activity to an output device, such as an external device or a user interface. At block 405, monitor application 113 typically outputs the activity as described with respect to block 304.
[0116] As discussed, a predictive model such as predictive model 120 or predictive model 190 is trained prior to use. Predictive model 120 can be trained using supervised or unsupervised learning. With supervised learning, the predictive model is provided with annotated or labeled training data that indicates actual activities of an infant, such as whether the infant is sleeping, awake, or eating. The training data is provided to the predictive model, which creates a loss function and compares the loss function to the actual, annotated, or labeled values.
[0117] In one aspect, additional training can be performed locally by an operator of the monitor application 113. For example, the training data can be augmented at runtime by a device such as a caregiver. For example, the monitor 110 can prompt the operator "The activity monitor thinks your baby is sleeping, is this correct?" In this way, the prediction model 120 can be continuously updated and improved over time. The prediction model 120 can be specifically tailored to the tendencies of a particular baby. For example, one baby may be a lighter sleeper than another baby, resulting in the prediction for one baby needing to be adjusted for use on the second baby.
[0118] Figure 6 Depicted are examples of sensor output from an accelerometer annotated with predicted states in accordance with certain aspects of the present disclosure. Figure 6 Included is a diagram 601. Diagram 601 shows the accelerometer output of the sensor attached to the body of the baby on the x, y and z axes. Diagram 601 is annotated with identified or predicted activity states 602, 603, 604 and 605. Activity state 602 represents the state in which the baby is eating. It can be seen that the movement detected by the accelerometer is relatively small compared to other states such as 604, which represents the time when the baby is lying on his back and playing with the baby gym. Activity state 603 represents changing a diaper. It can be seen that state 605 shows a little more movement than nothing, depicting when the baby is sleeping. The aspects described herein can be trained to predict activity states, such as the activity states depicted by 602-605, and other activity states such as lying on one's stomach and lying on one's side.
[0119] Figure 7 Depicted are examples of sensor output from an accelerometer indicating sleep and wakefulness, in accordance with certain aspects of the present disclosure. Figure 7 Included is a chart 701 depicting activity states 702 and 703. Activity state 702 indicates data representing an infant subject slowly waking up and accumulating to eventually crying. Activity state 703 indicates that the infant is awake. Aspects described herein can be trained to distinguish between activity states 702 and 703.
[0120] Figure 8 Depicted are examples of sensor outputs indicative of detected infant breathing rate according to certain aspects of the present disclosure.When an infant subject is sleeping, an accelerometer can detect small movements in one or more directions based on frequency changes and identify those movements as breathing. Figure 8 Included are graphs 801 and 802. Graph 801 depicts a waveform associated with respiratory rate. Graph 802 depicts a waveform derived from the waveform in graph 801 but filtered by a bandpass filter (e.g., allowing 0.1-1 Hz frequencies to pass). As can be seen, waveform 803 is easily discernible. Waveform 803 shows a respiratory rate of approximately 28 breaths per minute.
[0121] Fig. 9 is a flow chart of an exemplary method for training a prediction model to determine activity from an infant's movements according to certain aspects of the present disclosure. The prediction model 120 can be generated by a monitor application 113 running on the monitor 110 or a computer such as Fig.21 Similarly, an external device such as the activity classification server 180 can train the prediction model 190 using the method 900 .
[0122] At block 901, the monitor application 113 receives training data including time series data indicating inertial measurements measured over a set of time periods, the training data annotated with training labels indicating corresponding activities for each time period. In some cases, the training data labels are provided by a caregiver. For example, a caregiver can indicate what an infant is doing at a particular time, and the monitor application 113 can combine the indication with accelerometer or gyroscope data to provide training data to train one of the prediction models.
[0123] For example, the training data consists of multiple instances of the training data. Each instance includes measurements of motion. The measurements of motion include acceleration, such as an xyz triplet, or measurements of angular velocity from a gyroscope. Therefore, each instance represents a snapshot in time of motion.
[0124] The training data is annotated with labels corresponding to actual measured activities, such as whether the infant is sleeping or awake. For example, each instance of the training data is annotated with a label indicating the actual determined activity of the infant, specifically, the activity performed by the infant at a time corresponding to the instance of the training data (e.g., when the measurement was taken).
[0125] In one aspect, the training data can include instances representing time periods. For example, an instance can represent a period of one second. Thus, the instance can include multiple samples of motion data with an associated activity annotation.
[0126] At block 902, the monitor application 113 provides the annotated training data to the predictive model. The monitor application 113 provides the predictive model 120 with one instance of the training data at a time.
[0127] At block 903 , the monitor application 113 iteratively adjusts the prediction model based on a loss function based on a comparison of the predictions to the training labels by determining, by the prediction model, whether the data corresponding to the time period corresponds to one of the activities.
[0128] Specifically, the prediction model 120 outputs an activity from a set of predefined activities. For example, the set of activities can be eating on the left side, eating on the right side, sleeping, playing while awake but lying on the upper side, being held, and sitting. The loss function is a comparison of the predicted activity (e.g., the activity corresponding to the time series data determined by the prediction model) with the activity indicated by the training labels. The predicted activity can be represented numerically or otherwise.
[0129] At block 904, the monitor application 113 reduces the value of a loss function based on a comparison of the activity determined from the prediction model with the labeled activity. The loss function calculates the difference between the determined activity and the labeled activity (e.g., the activity actually performed by the infant who was the subject of the training data at that time). The prediction model 120 attempts to minimize the loss function at each iteration and adjusts its internal parameters accordingly.
[0130] The monitor application 113 repeats steps 902-904 and subsequent instances of the training data. The prediction model continues to update and reduce the value of the loss function with each iteration.
[0131] After a sufficient amount of training data, the prediction model learns to distinguish between sleeping and not sleeping. For example, after a sufficient number of examples of deep sleep and light sleep, the prediction model can learn to distinguish between deep sleep and light sleep. Other training examples include distinguishing when the baby is lying on its back and when it is lying on its stomach, and whether the baby is breastfeeding, and if so, distinguishing which breast the baby is feeding on. The baby's orientation is useful for distinguishing between breastfeeding, bottle feeding, and eating solid food. For example, certain orientations (such as completely vertical or completely horizontal) are less likely than other orientations.
[0132] At block 905, the monitor application 113 tests the trained model using a set of test data separate from the training data and measures the accuracy level of the predictions. The test data is separate from the training data used for training purposes as described with respect to blocks 901-904. The test data is not provided to the prediction model 120 until testing. In this way, the test data is new to the prediction model 120 and therefore cannot affect the predictions of the model. Therefore, the test data can be used to rigorously test whether the training is effective. If the prediction model fails in a threshold number of tests, the model is further trained.
[0133] In one aspect, the activity classification system 100 can be integrated with a system that can measure urine in a diaper, temperature, humidity, a baby's bowel movements, or other data. For example, the integrated sensor can include a sensor 101, such as an accelerometer 102 and a gyroscope 103, and additional sensors such as temperature, humidity, etc. The sensor 101, in particular the processor 105, can send additional measurements such as temperature measurements and humidity measurements to the monitor 110. In turn, then, the monitor application 113 can determine when the baby's diaper needs to be changed and provide an alarm, for example, via an output device. On the other hand, the monitor 110 can receive data sent from a sensor that measures the color of a color bar. The monitor 110 can detect the amount of urine present in the baby's diaper by the color of the color bar. The monitor 110 can output an alarm to a caregiver via an output device 111.
[0134] In some cases, activity monitor 110 can use a state machine to determine the activity state of the infant. An exemplary state machine is Fig.10 The state machine can be implemented by the monitor 110, the activity classification server 180, the mobile device 199, or another device. The predicted state can be output to the caregiver periodically.
[0135] Fig.10 is a state diagram of an exemplary state machine for determining an activity level in accordance with aspects of the present disclosure. Fig.10 A state machine 1000 is depicted, including various states and transitions between states.
[0136] More specifically, state machine 1000 includes four states: rocking 1010, awake 1011, sleeping 1012, and calm 1013. Other states are possible. State machine 1010 also includes examples of transitions between states. For example, transition 1020 is a transition between rocking 1010 and awake 1011. Transition 1021 is a transition between rocking 1010 and sleeping 1012. Transition 1022 is a transition between awake 1011 and sleeping 1012. Transition 1023 is a transition between awake 1011 and calm 1013. Finally, transition 1024 is a transition between sleeping 1012 and calm 1013.
[0137] Transitions 1020 - 1024 can be triggered by detection of infant activity, such as by using method 300 or 400 .
[0138] In some cases, measurements from an accelerometer or gyroscope can result in a brief transition in state. One example is when a baby is sleeping but is rocked for a short period of time (e.g., five minutes) during sleep. Such a short period may be incorrectly marked as awake, causing the caregiver to become concerned or frustrated with the monitor. In another example, the "sleeping" state is preceded by a certain amount of quiet time, which is actually the baby falling asleep, such as cooling down. In addition, during the day, periods of low activity may be incorrectly measured as sleeping.
[0139] In some cases, the user of the activity monitor may not expect or know how to operate such activity state detection. Therefore, in some cases, the activity monitor can use a smoothing filter to eliminate short or seemingly odd transitions. The smoothing filter helps reduce confusion caused by a transition that is later replaced by another transition. For example, if the activity monitor determines that the baby is sleeping, then rocks for a short period of time and continues to sleep, the baby is not in the process of waking up. Conversely, the activity monitor should not report the rocking period.
[0140] Fig.11 An example of such a smoothing filter is depicted in Figure 12-Figure 15Examples of the smoothing filter are further described.
[0141] Fig.11 is a flow chart of an exemplary method 1100 performed by a smoothing filter according to certain aspects of the present disclosure. For discussion purposes, reference is made to Figure 12-Figure 15 To illustrate Fig.11 .
[0142] Figure 12-Figure 15 is a diagram illustrating exemplary operations performed by a smoothing filter according to aspects of the present disclosure.
[0143] At block 1101 , the smoothing filter changes all active segments designated as "quiet" to a subsequent state. Fig.12 Graphs 1210 and 1250 are depicted, each depicting states ("awake," "rocking," "calm," and "sleeping") over time. More specifically, graph 1210 depicts prediction 1211, which represents the output of the activity monitor at block 1101. Output 1251 represents the output after block 1101 is completed.
[0144] As can be seen, chart 1210 includes regions 1212, 1213, and 1214, which indicate transitions to a calm state. As depicted in region 1252, periods where the predicted state transitions from calm to sleep are reclassified as sleep. As depicted in output 1251 following region 1252, transitions from calm to wakefulness identified in regions 1213 and 1214 are converted to wakefulness.
[0145] At block 1102, the smoothing filter changes all activity segments designated as "rock" to "sleep." Fig.13 Graphs 1310 and 1350 are depicted, each depicting states over time. More specifically, graph 1310 depicts prediction 1311, which represents the input to the smoothing filter at block 1102. Graph 1351 represents the output after block 1101 is completed.
[0146] It can be seen that regions 1312, 1313 and 1314, each region represents a brief period of time when the state is "rocking" before and after the period of "sleeping". Regions 1312-1314 are removed or "eliminated" by the smoothing filter. It can be seen that graph 1351 depicts a constant "sleeping" state.
[0147] At block 1103 , the smoothing filter changes all segments that are (i) designated as “sleeping” and (ii) below a length threshold and (iii) during daytime to “awake”. Fig.14Graphs 1410 and 1450 are depicted, each depicting states over time. More specifically, graph 1410 depicts prediction 1411, which represents the input to the smoothing filter at block 1103. Graph 1451 represents the output after block 1103 is completed.
[0148] As can be seen, regions 1412-1414 are regions where the predicted state changes from awake to sleeping to awake, remaining in the sleeping state for only a short period of time. Such activity periods may be incorrectly identified as sleeping because they are too short to be true sleep. Therefore, the smoothing filter changes these states to awake, resulting in graph 1451. Examples of length (or time) thresholds include 10 minutes, 20 minutes, or 30 minutes.
[0149] At block 1104 , the smoothing filter changes all segments that are (i) designated as “awake” and (ii) below a length threshold and (iii) during the night to “sleeping”. Fig.15 Graphs 1510 and 1550 are depicted, each depicting states over time. More specifically, graph 1510 depicts prediction 1511, which represents the input to the smoothing filter at block 1104. Graph 1551 represents the output after block 1104 is completed. The output depicted in graph 1551 can be output to a user interface.
[0150] As can be seen, regions 1512-1515 represent regions predicted to be "awake" within the "sleeping" region. Given that the infant has slept for a relatively long period of time and then continues to sleep, such activity periods may be incorrectly identified as awake because they are too short to be truly awake. Therefore, the smoothing filter changes these states to sleeping, resulting in graph 1551.
[0151] Alternatively or additionally, the monitor application 113 can display the infant's historical sleep information in a graphical format or on a calendar display to enable a caregiver to view sleeping trends or routines.
[0152] Fig.16 and Fig.17 Depicted are exemplary user interfaces for viewing an infant's sleep patterns in accordance with certain aspects of the present disclosure. Fig.16 The user interface 1600 and the chart 1601 are included. The user interface 1600 includes a sleep routine display that indicates for multiple days of the week and each time period of the day whether the infant subject is predicted to be asleep or awake. The user interface 1700 includes Fig.17 The total sleep of the baby per day is shown in graph 1701. Graph 1701 can be filtered by week, month or lifetime.
[0153] The monitor application 113 can also output feeding information derived from the baby's activities. For example, the output device 111 can provide information about when the last feeding occurred, how long the feeding event lasted, and whether the breastfeeding event occurred on the left or right breast. The monitor application 113 can also enable the caregiver to input information from bottle feeding events. The monitor application 113 can display the baby's historical feeding information in a graphical format or on a calendar display so that the caregiver can view feeding trends or routines.
[0154] Fig.18 Depicted are exemplary user interfaces for viewing an infant's feeding patterns in accordance with certain aspects of the present disclosure. Fig.18 It includes a user interface 1800, a sleep or feed button 1801, and a chart 1802. The chart 1802 read together with the feed button 1801 shows the time periods of each day for multiple days when the infant subject is sleeping, feeding, or neither.
[0155] Fig.19 Included is a user interface 1900 depicting a graph 1901 and text information 1902. The user interface 1900 allows filtering by week, month, or lifetime. The graph 1901 shows the hours used for feeding or breastfeeding each day. The text information 1902 provides specific feedback to the operator, such as averages.
[0156] Fig. 20 A user interface 2000 is included, which depicts a text information box 2001 and recent information 2002. The text information box 2001 provides detailed information to the operator of the device, such as the number of feedings for the day, the total time of breastfeeding for the day, and observations. The recent information 2002 displays the time and duration of the last feeding.
[0157] Fig.21 A block diagram of a processor configured to determine activity of a wearer of a sensor according to certain aspects of the present disclosure is depicted. Some or all components of computing system 2100 may belong to Figure 1105 or processor 112. For example, monitor application 113 and prediction model 120 can operate on computing system 2100. Computing system 2100 includes one or more processors 2102 communicatively coupled to one or more memory devices 2114. Processor 2102 runs computer executable program code stored in memory device 2114 in the form of non-transitory computer executable instructions, accesses information stored in memory device 2114, or both. Examples of processor 2102 include a microprocessor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or any other suitable processing device. Processor 2102 can include any number of processing devices, including one processing device.
[0158] Memory device 2114 includes program code 2120 and program data 2130. Program code 2120 can include code executed by monitor application 113, prediction model 120, prediction model 190, a mobile application running on mobile device 199, or any other application described herein. Program data 2130 can include training data for a prediction model, accelerometer or gyroscope measurements, or program data for any application described herein, such as monitor application 113, prediction model 120, or prediction model 190.
[0159] Memory device 2114 includes any suitable computer-readable medium, such as an electronic storage device, an optical storage device, a magnetic storage device, or other storage device capable of providing computer-readable instructions or other program code to the processor. Non-limiting examples of computer-readable media include disks, memory chips, ROM, RAM, ASICs, optical storage, tapes or other magnetic storage, or any other medium from which a processing device can read instructions. Instructions may include processor-specific instructions generated by a compiler or interpreter from code written in any suitable computer programming language, including, for example, C, C++, C#, Visual Basic, Java, Python, Perl, JavaScript, and ActionScript.
[0160] The computing system 2100 may also include a number of external or internal devices, such as input devices or output devices. For example, the computing system 2100 is shown as having an input / output ("I / O") interface 2108 that can receive input from an input device or provide output to an output device. A bus 2106 can also be included in the computing system 2100. The bus 2106 can communicatively couple one or more components of the computing system 2100 and allow communication between these components.
[0161] The computing system 2100 runs to configure the processor 2102 to perform the above Figure 1-4 9-11. The program code of the monitor application 113, which can be in the form of non-transitory computer executable instructions, can reside in the memory device 2114 or any suitable computer-readable medium and can be executed by the processor 2102 or any other suitable processor or processors. The operation of such program code configures or causes (multiple) processors to perform the operations described herein with respect to the processor 2102. In additional or alternative aspects, the above program code can be stored in one or more memory devices that the computing system 2100 can access from a remote storage device via a data network. The processor 2102 and any process can use the memory device 2114. The memory device 2114 can store, for example, additional programs or data used by applications (such as the monitor application 113) running on the processor 2102.
[0162] The computing system 2100 can also include at least one network interface 2104. The network interface 2104 includes any device or group of devices suitable for establishing a wired or wireless data connection to one or more data networks. Non-limiting examples of the network interface 2104 include an Ethernet network adapter, a WiFi network, a modem, and / or the like. The computing system 2100 can communicate with one or more other computing devices or computer-readable data sources via a data network using the network interface 2104.
[0163] Numerous specific details are set forth herein to provide a comprehensive understanding of the claimed subject matter. However, those skilled in the art will appreciate that the claimed subject matter may be practiced without these specific details. In other cases, methods, devices, or systems known to those of ordinary skill in the art are not described in detail in order to avoid obscuring the claimed subject matter.
[0164] Unless otherwise specifically noted, it should be understood that terms such as "process," "compute," "determine," and "identify" utilized throughout the specification discussion refer to the actions or processes of a computing device, such as one or more computers or similar electronic computing devices or multiple electronic computing devices that manipulate or transform data represented as physical electronic or magnetic quantities within a memory, register, or other information storage device, transmission device, or display device of a computing platform.
[0165] The systems discussed herein are not limited to any particular hardware architecture or configuration. The computing device can include any suitable arrangement of components that provide results conditioned on one or more inputs. Suitable computing devices include multi-purpose microprocessor-based computer systems that access stored software that programs or configures the computing system from a general-purpose computing device to a special-purpose computing device that implements one or more aspects of the present subject matter. Any suitable programming, scripting, or other type of language or combination of languages can be used to implement the teachings contained herein in software that will be used to program or configure the computing device.
[0166] The foregoing description of some examples has been presented only for the purpose of illustration and description, and the foregoing description is not intended to be exhaustive or to limit the present disclosure to the precise form disclosed. Many modifications and changes of the present disclosure will be apparent to those skilled in the art without departing from the spirit and scope of the present disclosure.
Claims
1. A computer-implemented method for determining an activity, the method comprising: receiving a plurality of inertial measurement results in three dimensions over a period of time from an inertial measurement sensor; Calculate statistics derived from inertial measurements; providing the plurality of inertial measurements and statistical data to a prediction model; receiving, from a prediction model, an activity determined from a plurality of predefined activities based on the inertial measurements, wherein the plurality of predefined activities include awake, sleeping, rocking, and restful; In response to identifying the determined activity as calm and the previously determined activity as awake, sleeping, or rocking, adjusting the determined activity to a corresponding one of awake, sleeping, or rocking; In response to identifying that the determined activity is rocking and the previously determined activity was sleeping, changing the determined activity to sleeping; and Provide the identified activities to the output device.
2. The method according to claim 1, wherein: The statistical data includes (i) a variance of the plurality of inertial measurement results, or (ii) a root mean square of the plurality of inertial measurement results.
3. The method according to claim 1, wherein: The plurality of predefined activities include one or more of (i) sleeping, (ii) awake, (iii) lying face down, (iv) lying face up, (v) resting, or (vi) rocking.
4. The method according to claim 1, wherein: Rocking refers to movement exceeding a first threshold amount, and wherein calm refers to movement less than a second threshold amount.
5. The method according to claim 1, further comprising: determining a plurality of activities based on the prediction model, each activity corresponding to a different time period in the plurality of time periods; In response to identifying that the determined activity is (i) sleeping, (ii) below a time threshold, and (iii) occurring during daytime, changing the determined activity to awake; as well as In response to identifying that the determined activity is (i) awake, (ii) below a time threshold, and (iii) occurring during night time, the determined activity is changed to sleeping.
6. The method of claim 1 , wherein the plurality of predefined activities include (i) whether the infant is breastfeeding, (ii) whether the infant is bottle-feeding, or (iii) the orientation of the infant during feeding, the method comprising: receiving a plurality of orientation or angular velocity measurements from a gyroscope; as well as providing the plurality of orientation or angular velocity measurements to a prediction model, Therein, the determined activity is based at least in part on angular velocity measurements.
7. The method according to claim 1, further comprising: receiving additional training data from a monitor device, the additional training data comprising time series data indicating inertial measurements measured over additional time periods, the additional training data annotated with training labels indicating activities of a plurality of activities; and The prediction model is updated based on the additional training data and the loss function, the updating comprising: determining, by the prediction model, a prediction indicating whether the data corresponding to the time period corresponds to one of the plurality of activities, and A value of a loss function is reduced based on a comparison of the predictions to the training labels.
8. The method of claim 7, wherein the plurality of activities comprises one or more of (i) awake, (ii) sleeping, (iii) rocking, or (iv) resting.
9. The method according to any one of claims 1 to 5 and 7, further comprising: Storing time series data for multiple time periods in memory; determining a plurality of activities based on the prediction model, each activity corresponding to a different time period among the plurality of time periods; and In response to determining that the predetermined time threshold has passed, a log report is generated and displayed on a user interface, the log report including the determined activity for each of the plurality of time periods.
10. The method according to claim 1, further comprising: receiving, via the wireless transceiver, a software update including a new prediction model; and Update the forecast model with the new forecast model.
11. A baby monitoring system comprising: Inertial measurement sensors; and A processor configured to perform operations including: receiving a plurality of inertial measurements from an inertial measurement sensor; Calculate statistics derived from inertial measurements; providing the plurality of inertial measurements and statistical data to a prediction model; receiving from a predictive model a determined activity of the infant according to a plurality of predefined activities based on the plurality of inertial measurements, wherein the plurality of predefined activities include awake, sleeping, rocking, and calm; In response to identifying the determined activity as calm and the previously determined activity as awake, sleeping, or rocking, adjusting the determined activity to a corresponding one of awake, sleeping, or rocking; In response to identifying that the determined activity is rocking and the previously determined activity was sleeping, changing the determined activity to sleeping; and Provide the identified activities to the output device.
12. The infant monitoring system of claim 11, wherein: The processor is further configured to perform operations including: receiving, from the prediction model, a breathing rate determined by the prediction model; and Provides the respiration rate to an output device.
13. The infant monitoring system of claim 11, wherein: The processor is further configured to perform operations including receiving from the predictive model an indication of a volume of liquid consumed by the infant during a feeding.
14. The infant monitoring system of claim 11, wherein: The processor is further configured to perform operations including receiving from a prediction model an estimate of (i) a time until a next meal or (ii) a volume of liquid expected to be consumed at the next meal, wherein the estimate is based on the plurality of inertial measurements.
15. The infant monitoring system of claim 11, further comprising: A wireless transmitter is configured to transmit the plurality of inertial measurement results to an external device, wherein the external device comprises: an additional wireless transceiver configured to receive the plurality of inertial measurements from the wireless transmitter; and an additional processor configured to execute instructions such that the additional processor: receiving, from an additional wireless transceiver, the plurality of inertial measurements obtained from an inertial measurement sensor; providing the plurality of inertial measurements to an additional prediction model; and The determined activity of the infant is received from the additional predictive model.
16. The infant monitoring system of claim 11, wherein: The processor is further configured to perform operations including: receiving a plurality of angular velocity measurements from a gyroscope; and providing the plurality of angular velocity measurements to a prediction model, Wherein the prediction model calculates the determined activity based on the plurality of angular velocity measurements.
17. The infant monitoring system of claim 15, wherein: The external device is external to the inertial measurement sensor and the processor, and wherein the inertial measurement results are sent to the external device via a data network connection.
18. The infant monitoring system of claim 11, wherein: Providing the plurality of inertial measurements to a predictive model includes sending the plurality of inertial measurements to a server device over a network, and wherein receiving the determined activity further includes receiving the determined activity from the server device over the network.
19. The infant monitoring system of claim 11, wherein: The processor is further configured to perform operations including: storing inertial measurements for a plurality of time periods in a memory; determining a plurality of activities based on the prediction model, each activity corresponding to a different time period among the plurality of time periods; and In response to determining that the predetermined time threshold has passed, a log report including the plurality of activities and corresponding time periods is generated.
20. The infant monitoring system of claim 11, wherein the processor is further configured to execute instructions such that the processor: The plurality of inertial measurements are transmitted over a data network to a server device configured to record a plurality of measurements from a plurality of infant monitoring systems.
21. The infant monitoring system of claim 11, wherein: The processor is further configured to execute instructions so that the processor: determining a plurality of activities based on the prediction model, each activity corresponding to a different time period in the plurality of time periods; In response to identifying that the determined activity is (i) sleeping, (ii) below a time threshold, and (iii) occurring during daytime, changing the determined activity to awake; and In response to identifying that the determined activity is (i) awake, (ii) below a time threshold, and (iii) occurring during night time, the determined activity is changed to sleeping.
22. The infant monitoring system of claim 11, further comprising an additional sensor, wherein: The processor is configured to execute instructions so that the processor: receiving additional measurements of (i) a change in conductivity, (ii) a change in an optical property, (iii) a change in capacitance, (iv) a change in color, or (v) a change in a detected volatile organic compound from an additional sensor; determining an indication of the presence of (i) urine or (ii) a bowel movement based on the additional measurement results; and The indication is provided to an output device.
23. The infant monitoring system of claim 11, further comprising: Optical sensor, configured as: Obtain color measurement results; and The color measurement result is sent to a processor, wherein the processor is further configured to perform operations including: determining a wetness indication of the absorbent article based on the color measurement; and The determined activity and the humidity indication are provided to an output device.
24. The infant monitoring system of claim 11, further comprising a battery operable to power the processor and the inertial measurement sensor.
25. A computer-implemented method for training a predictive model as claimed in any one of claims 1 and 11 to determine an activity, the method comprising: receiving training data including time series data indicating inertial measurements measured over a plurality of time periods, the training data annotated with a plurality of training labels indicating a corresponding activity in each of a plurality of activities; and Iteratively adjusting the prediction model based on a loss function based on a comparison of the predictions to corresponding training labels, wherein iteratively adjusting the prediction model comprises: determining, by a prediction model, whether the data corresponding to the time period corresponds to a prediction of one of the plurality of activities, and The prediction model is modified so that subsequent values of the loss function are reduced in subsequent iterations.
Citation Information
Patent Citations
Liver and tumor segmentation method and system based on multitask deep convolution network
CN107784647A
Apparatus and method for product and signaling device matching
US20120116337A1
Infant caregiver system
US20160287074A1
