Automated Detection of Body Behavior Events and Corresponding Regulation of a Drug Dispensing System

Abstract

The present invention provides an automated drug dispensing system that provides a trigger drug administration message upon detection of an inferred event for which a drug administration message is to be sent. The event may be the start, onset, or expected onset of a feeding event detected from a user's posture by an event detection module based on a set of sensor readings. The message may be a signal to a drug dispensing device and / or a reminder message to the user, an ancillary message to a caregiver, a health professional, or another person. The drug administration message may include a signal for input to an insulin management system or an input to a meal sensing artificial pancreas. The event may also include a drinking event, a smoking event, a personal hygiene event, and / or a drug-related event.

Description

[0001] Cross - Reference to Related Applications

[0002] This application claims the benefit and priority of the following patent applications: U.S. Provisional Patent Application No. 62 / 753,819, filed October 31, 2018; U.S. Patent Application No. 16 / 667,641, filed October 29, 2019; and U.S. Patent Application No. 16 / 667,650, filed October 29, 2019. Technical Field

[0003] The present disclosure generally relates to drug dispensing or administration, and more particularly to methods and devices for using sensors to track patient activity and derive patient activity related to food intake for providing reminders to a patient or caregiver regarding drug needs and / or signaling a drug dispensing device to dispense a drug. Background Art

[0004] For some medical conditions such as type 1 diabetes, the timing and administration of drugs such as insulin depend on various factors such as the patient's current blood glucose level and whether the patient is eating or drinking, and the components consumed. Thus, knowing when someone has started or is about to start eating or drinking is of great significance for the treatment of a diabetic individual on an insulin regimen. Further quantifying other parameters of the eating or drinking activity such as the duration or speed of eating or drinking can also be of great significance since the drug regimen may vary based on the duration and / or speed.

[0005] Millions of people have type 1 diabetes, a condition in which a person's body cannot produce insulin. The human body breaks down the carbohydrates consumed into blood glucose, which serves as energy for the body. The body needs to convert this blood glucose from the bloodstream into glucose in body cells and this conversion uses the hormone insulin. People with type 1 diabetes cannot produce enough insulin on their own to regulate blood sugar levels and may require insulin injections to maintain safe blood sugar levels.

[0006] Insulin can be provided by a micro - dosing system that injects a specific amount of insulin over time. For example, a type 1 diabetes patient may need to regularly check their blood glucose level and manually inject the correct dose of the required insulin at the start of a meal so that their body can convert the glucose that enters their bloodstream due to the meal into glucose stored in body cells. Both over - dosing and under - dosing can lead to adverse conditions and long - term complications. Micro - dosing systems or manual management of insulin injection are treatment regimens and typically require variable timing and dosing. This can make managing the disease difficult, especially in the case of a young child patient.

[0007] One approach is a “hybrid” closed-loop drug delivery system that takes continuous or near-continuous blood glucose readings and autonomously micro-doses the patient based on those readings, except when the patient is eating or drinking. To account for the latter situation, the hybrid system receives patient input indicating when the patient is about to start eating or drinking. In the absence of this additional information, the drug delivery system will respond too slowly because of the significant delay between glucose readings and insulin diffusion. Manual meal announcements place a great burden on the patient and poor compliance results in diminished blood glucose control. Missing a pre-meal insulin bolus is a significant contributor to poor blood glucose control in type 1 diabetic patients.

[0008] Improved methods and apparatus are needed for automated detection of eating or drinking events and for signaling the drug delivery system and / or sending reminders to the patient, caregiver, and health professional. SUMMARY OF THE INVENTION

[0009] An automated drug delivery system may include a sensor that detects motion and other physical inputs related to a user of the automated drug delivery system; a processor that executes program code and processes data received from the sensor, including a set of sensor readings, where at least one of the sensor readings in the set measures motion of a body part of the user; an event detection module that determines a user's posture from the set of sensor readings; a storage device for event-specific parameters initialized for a food intake event; and a storage device for an event state value, where the event state value is one of an out-of-event state or an in-event state, and where the event state value is initialized to the out-of-event state. The program code may include: a) program code for determining a first potential posture of the user (including the posture type of the first potential posture) from the set of sensor readings, where some of the posture types are members of a first set of posture types; b) program code for determining a confidence level associated with the first potential posture, where the confidence level relates to the level of confidence in correctly determining the posture type of the first potential posture; c) program code for modifying and recording, where if the confidence level is above or at a threshold and the posture type is a member of the first set of posture types, the event state value is modified from the out-of-event state to the in-event state and the first potential posture is recorded as the first posture of the food intake event; d) program code for determining an inferred event for which a drug administration message is to be sent based on the confidence level and / or the event state value; and e) program code for outputting a drug administration message regarding the drug administration need to the user.

[0010] An automated drug dispensing system may output secondary messages regarding drug administration needs to additional destinations other than the user. The additional destinations may include the communication devices of the user's friends, the user's healthcare provider, and / or first responders. The automated drug dispensing system may update the drug log and / or inventory in response to an inferred event, a signal input to an insulin management system, and / or a signal input to a meal sensing artificial pancreas.

[0011] In some embodiments, the event detection system includes sensors that detect movement and other physical inputs related to the user. The event detection system may process the movement and other physical inputs to identify the user's posture. Further processing methods include:

[0012] Providing a storage device for event-specific parameters initialized for a food intake event;

[0013] Providing a storage device for an event status value, where the event status value is one of an out-of-event status or an in-event status, and the event status value is initialized to the out-of-event status;

[0014] Using a processor in the event detection system to determine a set of sensor readings, where at least one sensor reading in the set of sensor readings measures the movement of a body part of the user;

[0015] Determining a first potential posture of the user from the set of sensor readings, including the posture type of the first potential posture, where some posture types are members of a first set of posture types;

[0016] Determining a confidence level associated with the first potential posture, where the confidence level relates to the level of confidence in correctly determining the posture type of the first potential posture;

[0017] If the confidence level is above or at a threshold and the posture type is a member of the first set of posture types, then:

[0018] (a) Modifying the event status value from the out-of-event status to the in-event status; and

[0019] (b) Recording the first potential posture as the first posture of the food intake event; and

[0020] Outputting a message regarding drug administration needs to the patient.

[0021] The method may further include providing a storage device for additional event-specific parameters, the additional event-specific parameters including eating events, smoking events, personal hygiene events, and / or medication-related events. The external trigger time may be determined based on when the food intake event is inferred to have started, when the food intake event is inferred to be in progress, and / or when the food intake event is inferred to have ended. The computer-based actions in response to the food intake event may be one or more of the following: (1) obtaining other information associated with the data representing the food intake event to be stored in a memory; (2) interacting with the user to provide information or a reminder; (3) interacting with the user to prompt the user for input; (4) sending a message to a remote computer system; and / or (5) sending a message to another person.

[0022] The method may include recording the food intake event in a food log and / or updating an inventory database in response to the food intake event.

[0023] A system for sensing a wearer's activity may include:

[0024] at least one sensor of an electronic device worn by the wearer, the at least one sensor for sensing the wearer's activity or a part thereof, including the movement of a body part of the wearer;

[0025] a storage device for event-specific parameters initialized for a food intake event;

[0026] a storage device for an event status value within the electronic device, where the event status value is one of an event-outside status or an event-in-progress status, and where the event status value is initialized to the event-outside status;

[0027] a processor in the electronic device, the processor determining a set of sensor readings from the at least one sensor; and

[0028] program code stored in the electronic device or in components of a system in communication with the electronic device, executable by the processor in the electronic device or another processor, including:

[0029] a) program code for determining a first potential posture of the wearer (including the posture type of the first potential posture) from the set of sensor readings, where some of the posture types are members of a first set of posture types;

[0030] b) program code for determining a confidence level associated with the first potential posture, where the confidence level relates to the level of confidence in correctly determining the posture type of the first potential posture;

[0031] c) Program code for determining whether a confidence level is above or at a threshold and the pose type is a member of a first set of pose types, and for modifying an event state value from an out-of-event state to an in-event state and recording a first potential pose as a first pose of a food intake event when the confidence level is above or at the threshold and the pose type is a member of the first set of pose types;

[0032] d) Program code for determining additional poses of a wearer from a set of additional sensor readings, each additional pose having a corresponding pose type;

[0033] e) Program code for determining whether an event state value is in an in-event state, and for recording the first pose and the additional poses as a pose sequence of a food intake event and deriving event-specific parameters from at least some of the poses of the pose sequence when the event state value is in the in-event state; and

[0034] f) Outputting a message regarding a drug administration requirement to a patient.

[0035] The system may include a control for changing an electronic device to a higher performance state when the event state value is modified from an out-of-event state to an in-event state, where the higher performance state includes one or more of additional power supplied to sensors, reduced latency of a communication channel, and / or increased sensor sampling rate. The sensors may include one or more accelerometers that measure movement of the wearer's arm and a gyroscope that measures rotation of the wearer's arm.

[0036] Using pose sensing technology, an event detection system may trigger an external device to collect additional information. In a particular implementation, the external device is a near field communication (NFC) reader and detects various objects having NFC tags thereon. In cases where these objects are food / drink related, the event detection system may determine what the pose is related to. For example, a food / drink container may have an NFC tag embedded in the product packaging, and a food intake monitoring system may automatically determine that the pose is related to an eating event and then signal the NFC reader to turn on and read a nearby NFC tag, thereby reading the NFC tag on the consumed product so that the pose and the event are associated with a specific product.

[0037] In other variations, other wireless technologies may be used. In some variations, the external device is a module integrated within a housing that also houses the event detection system.

[0038] An event detection system may include sensors that detect motion and other physical inputs related to a user. The event detection system may process the motion and other physical inputs to identify the user's pose, and may also use historical data, machine learning, rule sets, or other techniques for processing data to derive inferred events related to the user sensed by the sensors. For example, the sensors may detect an audio signal near the user's mouth, and the event detection system may use the audio signal to infer events such as events related to the user's eating or drinking activities. Other non-handheld sensors may also be used, such as motion sensors, temperature sensors, audio sensors, etc.

[0039] Exemplary poses may be a food intake pose, a sucking pose, or some other pose. The inferred events may be an eating event, a smoking event, a personal hygiene event, a drug-related event, or some other event that the user is inferred to be engaged in. A pose may represent some motion of the user, such as a hand pose that can be detected using a wrist-worn device that can be implemented non-invasively or minimally invasively and can operate without or with little user intervention.

[0040] When an event is inferred to have started, be in progress, and / or have ended, the event detection system may take actions related to the event, such as obtaining other information to be stored in a memory associated with data representing the event; interacting with the user to provide information or a reminder or prompt the user for input; sending a message to a remote computer system; sending a message to another person, such as a friend, a healthcare provider, a first responder; or other actions. In a particular example, once the event detection system infers that an event has started, it signals an auxiliary sensor system or an auxiliary processing system to take actions, such as collecting more information, sending a communication, or processing a task. The event detection system may create a data record of the event after detecting a new event, and populate the data record with details of the event that the event detection system is able to determine (such as details related to the pose involved in the event). The auxiliary sensor system or the auxiliary processing system may populate the data record with auxiliary data related to the event (such as the auxiliary data described herein).

[0041] The event detection system and the auxiliary sensor system and / or the auxiliary processing system may be used as part of a monitoring system having one or more applications such as food logging, inventory tracking / supplementation, production line monitoring / QC automation, medication adherence, insulin therapy, supporting meal-aware artificial pancreas, and other applications.

[0042] The sensing device monitors and tracks food intake events and details. A suitably programmed processor controls aspects of the sensing device to capture data, store data, analyze data, and provide appropriate feedback related to food intake. More generally, the method may include detection, identification, analysis, quantification, tracking, processing, and / or influencing related to food intake, eating habits, eating patterns, and / or triggers for food intake events, eating habits, or eating patterns. The feedback may be targeted at influencing food intake, eating habits, or eating patterns and / or triggers for these. The feedback may also be targeted at reminding the user to take one or more actions. The sensing device may also be used to track and provide feedback other than food-related behaviors, and more generally to track behavioral events, detect behavioral event triggers and behavioral event patterns, and provide appropriate feedback. The event detection system may be implemented using hardware and / or software.

[0043] The following detailed description, along with the accompanying Figure 1 drawings will provide a better understanding of the nature and advantages of the present invention.

[0044] The present invention is provided to introduce some concepts that will be further described in the following detailed description in a simplified form. The present invention is not intended to identify the main features or essential features of the claimed subject matter, nor is it intended to be used to help determine the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Various embodiments in accordance with the present disclosure will be described with reference to the drawings, in which:

[0046] Figure 1 An event monitoring system is shown.

[0047] Figure 2 A process for providing user intervention is shown.

[0048] Figure 3 is an illustrative example of an environment according to at least one embodiment.

[0049] Figure 4 is an illustrative example of an environment according to at least one embodiment, the environment including communication with at least one additional device via the Internet.

[0050] Figure 5 is an illustrative example of an environment according to at least one embodiment, in which a food intake monitoring and tracking device communicates directly with a base station or access point.

[0051] Figure 6 is an illustrative example of a high-level block diagram of a monitoring and tracking device according to at least one embodiment.

[0052] Figure 7An illustrative example of a block diagram of a monitoring and tracking device according to at least one embodiment.

[0053] Figure 8 Shows an example of a machine classification system according to at least one embodiment of the present disclosure.

[0054] Figure 9 Shows an example of a machine classification training subsystem according to at least one embodiment of the present disclosure.

[0055] Figure 10 Shows an example of a machine classification detector subsystem according to at least one embodiment of the present disclosure.

[0056] Figure 11 Shows an example of a machine classification training subsystem that uses non-temporal data and other data.

[0057] Figure 12 Shows an example of a machine classification detector subsystem that uses non-temporal data and other data.

[0058] Figure 13 Shows an example of a training subsystem of an unsupervised classification system according to at least one embodiment of the present disclosure.

[0059] Figure 14 Shows an example of a detector subsystem of an unsupervised classification system according to at least one embodiment of the present disclosure.

[0060] Figure 15 Shows an example of a classifier integration system.

[0061] Figure 16 Shows an example of a machine classification system that includes a cross-correlation analysis subsystem.

[0062] Figure 17 Shows, according to one embodiment, an advanced functional diagram of a monitoring system similar to Figure 1 a variant.

[0063] Figure 18 Shows, according to one embodiment, an advanced functional diagram of a monitoring system that requires user intervention.

[0064] Figure 19 Shows an advanced functional diagram of a drug delivery system.

[0065] Figure 20 An illustrative example of a machine learning system that may be used with other elements described in the present disclosure. Detailed Description

[0066] In the following description, various embodiments will be described. For purposes of explanation, specific configurations and details are set forth in order to provide a thorough understanding of the embodiments. However, it will also be apparent to those skilled in the art that the embodiments may be practiced without specific details. In addition, well-known features may be omitted or simplified in order not to obscure the described embodiments.

[0067] Various examples of devices are provided herein that a person will use to monitor, track, analyze food intake, the intake process and timing, and other related aspects of what a person eats, drinks, and otherwise consumes and provide feedback thereon for various purposes such as providing dietary information and feedback. Data related to the food intake process may include the timing of the eating process, the speed of eating, the time since the last food intake event, what was eaten, an estimate of the components of what was eaten, etc. Such devices may be integrated with a drug delivery system.

[0068] Overview

[0069] As will be described in more detail herein, a patient management system including a novel digital health application and a wearable sensing device that interacts with the application may provide a fully autonomous artificial pancreas system and may significantly improve the quality of life of a person with type 1 diabetes. The patient management system may remind the patient to take or administer their medications, but may also provide hybrid or autonomous management of drug delivery such as, for example, insulin delivery. The patient management system may promote mindful eating and proper hydration.

[0070] Through a combination of fine motor detection and artificial intelligence techniques, the patient management system detects high-impact moments and allows the individual to better manage their health, all based on insights automatically and non-invasively captured by analyzing their wrist movements and other sensor inputs. Response actions may include sending bolus reminders to the patient and / or their caregiver. The early meal detection capability provides unique insights into eating behavior and is a key component enabling the autonomous artificial pancreas system.

[0071] The patient management system may have an open data and software architecture. This may allow other companies, researchers, and developers to leverage the patient management system and the data it generates via an API that allows seamless integration with third-party applications and platforms.

[0072] In certain applications, the patient management system serves as a meal-time medication reminder, sending messages to patients to notify them that it is time to start insulin administration. The patient management system may have multi-level messaging capabilities. For example, in the case where a message is sent to a patient to notify them that an action needs to be taken and the patient does not respond, the patient management system may send a message to a caregiver to indicate that the patient has or has not taken the expected action in response to the detection of the start of a feeding event or has not taken action for meal-time insulin administration. The patient management system and its ability to provide automated real-time or near-real-time information on the patient's eating behavior, combined with novel control processes, can at least partially enable a fully autonomous closed-loop artificial pancreas system, thus potentially enabling a truly "set it and forget it" closed-loop insulin delivery system.

[0073] The patient management system may also be useful for people who are concerned about their diet for other reasons. People with type 1 diabetes typically receive insulin therapy in which they administer appropriate insulin doses based on their food intake and other factors. Although the cause of type 1 diabetes may not be directly linked to a person's eating behavior, people with type 1 diabetes need to carefully track their food intake in order to manage their insulin therapy. Such patients would also benefit from easier-to-use and more discreet methods of tracking food intake. In some embodiments of the patient management system, the sensing device is part of a feedback-driven automated insulin delivery therapy system. Such a system may include continuous monitoring of the patient's glucose level, a precision insulin delivery system, and the use of insulin with a faster absorption rate, which would further benefit from information that can be extracted from automated and seamless food intake tracking, such as tracking of carbohydrate and sugar intake. These devices may also be useful for health programs and the like.

[0074] Data Collection

[0075] Data can be obtained from some fixed devices with sensors and electronics, some mobile devices with sensors and electronics that a person can easily move and carry with them, and / or wearable devices with sensors and electronics that a person attaches to their body or clothing or that are part of the person's clothing. Generally speaking, such devices are referred to herein as sensing devices. The data may be raw sensor data provided by sensors capable of outputting data, or the data may be processed, sampled, or organized in some way to become data derived from the output of the sensors.

[0076] In this document, a person who owns such a device and whose consumption is monitored is referred to as a user. However, it should be understood that the device may be used without modification in cases where the person consuming, the person monitoring, and the person evaluating the feedback do not have to be the same person. In this document, what is consumed is referred to as food intake. However, it should be understood that these devices can be used to more generally track consumption and consumption patterns. A behavior tracking / feedback system as described herein may include one or more wearable devices and may also include one or more additional devices that are not worn. These additional devices may be carried by the wearer or placed nearby so that they can communicate with the wearable devices. The behavior tracking / feedback system may also include remote elements, such as remote cloud computing elements and / or remote storage means for user information.

[0077] The wearable device may be worn at different locations on the wearer (i.e., the person monitoring their behavior), and the wearable device may be programmed or configured to take into account these differences as well as differences between different wearers. For example, a right-handed person may wear the device around their right wrist, while a left-handed person may wear the device around their left wrist. Users may also have different preferences regarding orientation. For example, some users may want the control buttons to be on one side, while other users may prefer the control buttons to be on the opposite side. In one embodiment, the user may manually enter wrist preferences and / or device orientation.

[0078] In another embodiment, wrist preference and / or device orientation can be determined by asking the user to perform one or more predefined gestures and monitoring sensor data from the wearable device corresponding to the user performing the predefined gesture or group of gestures. For example, the user may be asked to move their hand towards their mouth. Then the changes in the accelerometer sensor readings across one or more axes can be used to determine the wrist and device orientation. In yet another example, the behavior tracking / feedback system can process sensor readings from the device when the user has worn the wearable device for a certain duration. Optionally, the behavior tracking / feedback system can also combine the sensor readings with other data or metadata about the wearer to infer the wrist and device orientation. For example, the behavior tracking / feedback system can monitor the user for a day and record the accelerometer sensor readings across one or more axes.

[0079] Since the movement of the lower arm is restricted by the elbow and the upper arm, based on the wrist and device orientation, some accelerometer readings will be more frequent than others. Then the information from the accelerometer can be used to determine the wrist and / or device orientation. For example, the mean, minimum, maximum, and / or standard deviation of the accelerometer readings can be used to determine the wrist and / or device orientation.

[0080] In some embodiments, the sensing device can sense the start / end of a food intake event, the speed of eating, the speed of drinking, the number of bites, the number of sucks, the estimation of fluid intake, and / or the estimation of portion size without user interaction. Operating with less human intervention, no human intervention, or only intervention that is not obvious to others will allow these devices to adapt well to different meal scenarios and different social situations. Sensing may include capturing details of the food before consumption, as well as user actions known to be associated with eating, such as repeated rotation of the upper arm or other hand-to-mouth movements. Sensors may include accelerometers, gyroscopes, cameras, and other sensors.

[0081] Using these devices can provide low usage conflicts for a person to detect, quantify, track the person's food intake components and the person's food intake behavior and provide feedback related thereto. Such methods have the potential to prevent, treat, and in some cases even cure diet-related diseases. Such devices can improve efficacy, accuracy, and compliance, reduce the usage burden, and enhance social acceptance. These devices can operate autonomously with no or minimal human intervention and do not interfere with a person's normal activities or social interactions in an invasive or other significantly negative way or violate a person's privacy. These devices are capable of handling a wide variety of meal scenarios and dining environments in a discreet and socially acceptable manner and are capable of estimating and tracking food intake components and amounts and other aspects of eating behavior. These devices can provide the person with real-time and non-real-time feedback on their eating behavior, habits, and patterns.

[0082] It is well known and should be understood that certain eating behaviors can be associated with, triggered by, or otherwise affected by physical, mental, or environmental conditions such as, for example, hunger, stress, sleep, addiction, disease, physical location, social stress, and exercise. These characteristics can form inputs for the processing performed by or used in these devices.

[0083] A food intake event generally refers to a situation, circumstance, or action in which a person eats, drinks an edible substance, or otherwise takes an edible substance into his or her body. Edible substances can include, but are not limited to, solid foods, liquids, soups, beverages, snacks, medications, vitamins, pharmaceuticals, herbal supplements, finger foods, pre-packaged foods, raw foods, meals, appetizers, main courses, desserts, candies, breakfasts, sports drinks, or energy drinks. Edible substances include, but are not limited to, substances that may contain toxins, allergens, viruses, bacteria, or other components that may be harmful to a person or to a group or subgroup of people. In this document, for ease of reading, food is used as an example of an edible substance, but it should be understood that other edible substances may be used instead of food unless otherwise specified.

[0084] Eating habits and patterns generally relate to how people consume food. Eating habits and patterns can include, but are not limited to, the speed of eating or drinking, bite size, amount of chewing before swallowing, speed of chewing, frequency of food intake events, amount of food consumed during a food intake event, body position during a food intake event, possible movement of the body or a particular body part during a food intake event, mental or physical state during a food intake event, and utensils or other devices used to hold, pick up, or consume food. The speed of eating or drinking may be reflected in the time between successive bites or sips.

[0085] Triggers generally relate to the reasons behind the occurrence of a food intake event, the amount consumed, and how it is consumed. Triggers for food intake events and eating habits or patterns can include, but are not limited to, hunger, stress, social pressure, fatigue, addiction, discomfort, medical needs, physical location, social context or environment, smell, memory, or physical activity. A trigger can coincide with the food intake event of which it is a trigger. Alternatively, a trigger can occur outside of the food intake event window and may occur before or after a food intake event at a time that may or may not be directly related to the time of the food intake event.

[0086] In some embodiments of the sensing device or system, fewer than all of the features and functions presented in this disclosure are implemented. For example, some embodiments may focus solely on the detection and / or processing and tracking of food intake and not be intended to guide the user to modify his or her food intake or not track, process, or guide eating habits or patterns.

[0087] In many examples herein, the context is that an electronic device is provided to a user who wears the electronic device, either alone or while communicating with nearby support devices that may or may not be worn (such as a smartphone for performing operations to offload the worn electronic device). In such examples, a person wears the electronic device and that person is referred to as the "wearer" in these examples, and the system includes the worn device and may include other components that are not worn and are nearby and preferably remote components that are capable of communicating with the worn device. Thus, the wearer wears the electronic device, and the electronic device includes sensors that sense the environment around the wearer. The sensing can be of environmental characteristics, body characteristics, movement, and other sensing signals as described elsewhere herein.

[0088] In many examples, the functionality of an electronic device may be implemented by hardware circuitry or program code instructions or a combination thereof, the program code instructions being configurable to be executed by a processor in the electronic device. In cases where it is stated that the processor does something, it may be that the processor does that thing by executing instructions read from an instruction memory, where the instructions provide for the execution of that thing. In this regard, the program code instructions may be configurable such that the processor (or host device) executes certain methods, processes or functions as defined by the executed instructions. Although others may be involved, a common example here is the wearer of the electronic device using the electronic device to monitor their own movements, such as postures, behavioral events including sequences of postures, activities, the start of an activity or behavioral event, the stop of an activity or behavioral event, etc. In cases where it is described that a processor executes a particular process, it may be that part of that process is carried out separately in a distributed processing manner from the wearable electronic device. Thus, the description of a process executed by a processor of an electronic device need not be limited to a processor within the wearable electronic device, but may also be a processor in a support device that communicates with the wearable electronic device.

[0089] Patient Management System

[0090] A patient management system may include a novel digital health application running on a device such as a smart phone or a dedicated device communicable with a wearable sensing device worn by a person. The wearable sensing device interacts with the application to provide hybrid or autonomous management of insulin administration and reminders.

[0091] In an example of the use of such a system, a person (referred to herein as the "patient") has been diagnosed with type 1 diabetes, which requires the provision of external insulin to the patient. The insulin may be given in the form of injectable insulin or an implantable micro-delivery device attached to the patient's body to introduce a measured amount of insulin into the patient's body. The dosage and timing of administration may be a function of when the patient is eating, starting to eat, about to eat, or has been eating and will continue to eat. A wearable device capable of discerning postures from motion and sensor data outputs may be able to determine, without specific patient intervention, when an eating event has started or is about to start and the speed, duration and possible end of the eating event. Based on that determination, the wearable device (or an auxiliary device communicable with the wearable device, such as a fully functional smart phone) will send a signal to the implantable micro-delivery device indicating some details of the eating / drinking, and / or will send a message to the patient, caregiver and / or health professional.

[0092] For example, a wearable device may determine that a patient has started eating, and based on the speed of eating and the determined possible duration of the event, may signal an implantable insulin micro-delivery device with some information about the eating event, which the delivery device may use to start delivering insulin to the patient. Additionally or alternatively, the wearable device may send a message related to the eating event and the measured parameters. For example, the wearable device may communicate with a nearby smartphone (which runs an application paired with the wearable device) and send a message (possibly as a cellular network text message (e.g., an SMS message)) to a pre-stored number assigned to the patient, where the message might say, "A food event has been detected. Be sure to activate your implantable insulin micro-delivery device to administer insulin."

[0093] If the patient is the operator of the smartphone, the application may send a message directly to the patient without having to send a network text message. In some cases, it may be useful for the message to have more than one recipient. For example, in cases where the patient requires or uses caregiver assistance either due to age (young or old) or other reasons, the wearable device messaging may also be directed to the caregiver. Where useful or required, this information may be provided to healthcare professionals, perhaps for monitoring patient protocol compliance.

[0094] As used herein, for ease of reading, an event may be described as an "eating event" rather than an "eating event and / or drinking event", but it should be understood that unless otherwise specified, the teachings herein related to eating events are equally applicable to drinking events.

[0095] Posture sensing technology can be used to automatically detect posture events, such as postures indicating when someone is eating or drinking, using motion sensors (accelerometers / gyroscopes) in a wrist-worn wearable device or a ring without user prompting or interaction. This detection can be done in real-time to infer key insights about consumption activities, such as for example the start time of the consumption event, the end time of the consumption event, the method of consumption, metrics associated with the speed of consumption, metrics associated with the amount consumed, and metrics associated with the frequency of consumption, the location of consumption, etc. Posture sensing technology can be used for other activities and behaviors, such as smoking, dental hygiene, hand hygiene, etc.

[0096] Posture sensing technology can use motion sensors (accelerometers / gyroscopes) in a wrist-worn wearable device or a ring to automatically (i.e., without user intervention) detect when a person is eating or drinking based on their hand posture. This detection can be performed in real time. The worn device may be combined with the processing and communication functions of a detached device (such as may be found in a portable communication device). The processing of the detached device may be used for more complex processing tasks, such as inferring insights about consumption activities, such as for example start time, end time, method of consumption, metrics associated with consumption speed, metrics associated with the amount consumed, metrics associated with consumption frequency, consumption location, etc.

[0097] The start of each meal can be a critical moment for a person with type 1 diabetes. In an example of disease management, at the start of each meal, the patient (or caregiver, etc.) needs to decide whether to inject insulin and how much insulin to inject and needs to take an action (e.g., the action of injecting insulin) based on that decision.

[0098] A patient management system can provide early meal detection capabilities and provide strong "persuasion" with information to help the patient and / or insulin delivery device make decisions and take actions. For example, the patient management system may send a real-time reminder to administer insulin at the start of a meal. Forgetting to bolus is one of the main causes of poor blood glucose control, especially in adolescents. The patient management system can prompt the patient to check their blood glucose level at the start and / or end of a meal and can support simple logging. The logging may be processed by the wearable component of the system, and it may be as discrete as the wrist posture of the patient performing a gesture indicating that the required action has been performed (e.g., administering insulin or checking blood glucose level) or other movements that can be used as data input actions.

[0099] Healthcare providers can survey their "current" patients to obtain accurate and actionable insights (e.g., assessing estimated carbohydrates, mood, difficulty of help needed).

[0100] A patient management system can more generally track eating and drinking activities with second-level accuracy, making it possible, for example, to correlate blood glucose levels and bolus (insulin administration) actions back to the accurate eating time. This can be used for personalized nutrition research and the formulation of personalized nutrition plans. This information can be a component of a personalized diabetes care pathway.

[0101] A patient management system can be used to study the impact of real-time bolus reminders on medication adherence and blood glucose control. Data sharing can be performed in real time or not in real time. Data sharing can be with caregivers, healthcare professionals, or provided to researchers and medical device developers as part of a larger dataset under appropriate privacy protection.

[0102] Part of the patient management system is an application that may provide a user interface for the patient. The application can provide the ability to send real-time bolus reminders to the patient and a monitoring service that allows real-time alerts (notifications or texts) to be sent to one or more destinations (e.g., phone numbers, email addresses, URLs, etc.) of remote caregivers and others. This can, for example, allow parents to remotely monitor the eating activities of their young child with type 1 diabetes and responses to messages or alerts.

[0103] In some methods, the patient manually notifies the insulin delivery system when he or she is eating or about to eat. Based on this input, the insulin delivery system then begins insulin delivery in one or more doses. This is not ideal because the user needs to take the action of notifying the insulin delivery system. It would be better if no user action were required. Alternatively, the insulin delivery system can infer the occurrence of a food intake event by monitoring changes in the patient's blood glucose level. Direct blood glucose level measurement may be impractical. Continuous glucose monitoring devices can be used to automatically and periodically measure interstitial fluid glucose levels, and these levels are typically used as surrogate indicators of blood glucose levels. However, interstitial fluid glucose level measurements can be delayed by 20 minutes or more. This is too late to achieve good blood glucose control, so interstitial fluid measurements may not be ideal for notifying the insulin delivery system.

[0104] Automatic detection of food intake events is needed to notify the insulin delivery system and allow the insulin delivery system to operate without or with reduced human intervention. The patient management system can generally be used as a medication reminder system, or more specifically, an insulin therapy system.

[0105] An example of the patient management system will now be described.

[0106] When an actual or imminent start of a food intake event is detected, a message / alert can be sent to the patient to remind him or her to take his / her medication. The medication can be insulin or other medications, such as medications that need to be taken before, during, or after a meal. In some cases, it may be desirable to have a delay between the detection of the actual or imminent start of the food intake event and the time of sending the alert, and this can be tuned as needed. For example, instead of sending the alert at the start of the food intake event, the alert can be sent after the system has detected five, ten, or some other specific number of bites or sucks consumed. In another example, the alert may be sent one minute, five minutes, or some other specific fixed time after the system has detected the actual or imminent start of the food intake event. The timing of the alert can also depend at least in part on the confidence level of the patient management system that the actual or imminent start of the food intake event has occurred. The alert may be sent when the patient management system has detected a condition that historically preceded a feeding event or some time later. Based on historical data recorded by the patient management system from the patient's past feeding events, the patient management system may be able to determine that notification or reminder is needed ten minutes before a meal, at the start of the meal, or during the meal. The patient management system may also be programmed to handle "nap" events, where the patient has been notified with an alert and the patient indicates that the patient management system should resend the reminder at a defined time in the near future, such as after a defined time period after the initial alert or reminder or after a defined number of bites after the initial alert or reminder.

[0107] When the patient management system sends a message / alert to the patient, one or more persons or one or more systems can also be notified via secondary messaging. The actual, possible, or imminent start of the food intake event can be notified to other persons or systems. If the message / alert to the patient includes a mechanism for the patient to respond, the patient's response can be notified to other persons or systems. If the patient fails to respond, other persons or systems can also be notified. This can be helpful in terms of supporting the patient's parents and / or caregivers.

[0108] In some cases, the patient management system may be programmed to receive different inputs from the patient. For example, the patient management system may have a user interface for receiving from the patient an indication that the meal has ended, that the meal will have zero, many, or some measure of carbohydrates.

[0109] As with patient messaging, for this secondary messaging, there may be a delay between the detection of the actual or imminent start of a food intake event and the time of sending a warning. Instead of sending a warning at the start of the food intake event, it may be sent after the system has detected a specific number of bites or sucks have been consumed. There may be a delay between sending the warning to the patient and notifying someone else or a system. If the message / warning to the patient includes a mechanism for the patient to respond, there may be a delay between the patient responding to the message / warning and notifying someone else or a system of the person's response or failure to respond.

[0110] One or more other people may be notified via a message sent over a cellular network. For example, a text message on his or her phone or other mobile device or wearable device.

[0111] An insulin therapy system will now be described.

[0112] Detection of the actual, possible, or imminent start of a food intake event as described herein may be used to notify an insulin delivery system. Upon receiving a signal indicating the actual, possible, or imminent start of a food intake event, the insulin delivery system may calculate or estimate an adequate dose of insulin to be administered and a schedule for insulin delivery.

[0113] The insulin delivery system may use other parameters and inputs to calculate or estimate the dose and frequency. For example, the insulin delivery system may use current or previous glucose level readings, fluctuations in glucose level readings, parameters derived from glucose level readings, or insulin-on-board (i.e., insulin administered at an earlier time but still having an effect in the patient's body). Examples of parameters derived from glucose level readings may be the current slope of the glucose level reading, the maximum, average, minimum, etc. of glucose level readings in a specific time window prior to the current food intake event.

[0114] The insulin delivery system may also include parameters related to the meal activity itself, such as the duration of the meal, the rate of eating, the amount consumed. The insulin delivery system may also use other sensor inputs, such as heart rate, blood pressure, body temperature, degree of hydration, degree of fatigue, etc., and may obtain these values from its own sensors or some of these values from other devices that the patient may use for this or other purposes.

[0115] The insulin delivery system may also include other inputs such as current or past physical activity levels, current or past sleep levels, and current or past stress levels. The insulin delivery system may also include specific personal information such as gender, age, height, weight, etc. The insulin delivery system may also include information related to the patient's insulin needs. This may be information input or configured by the patient, caregiver, or health record or healthcare maintenance system. Information related to the patient's insulin needs may also be derived from historical data collected and stored by the insulin delivery system. For example, the amount of insulin delivered by the insulin delivery system during a time period prior to the current food intake event.

[0116] In another embodiment, the insulin delivery system may consider the amount of insulin and the delivery schedule associated with one or more prior food intake events that occurred at or around the same time of day and / or the same day of the week or within a defined time window near the time of day. The insulin delivery system may also consider the patient's current location.

[0117] Additional parameters related to the food intake event may also be used to inform the insulin delivery system. The insulin delivery system may use such parameters to calculate or estimate a sufficient dose of insulin to be delivered and / or the schedule of insulin delivery. Such parameters may include, but are not limited to, the duration of eating or drinking, the amount of food or beverage consumed, the speed of eating, the amount of carbohydrates consumed, the method of eating, or the type of eating utensils or containers used. The food intake tracking and feedback system may calculate some of these additional parameters (e.g., duration or speed) without any user intervention. In other cases, user intervention, input, or confirmation by the user may be necessary.

[0118] The insulin delivery system may also use parameters related to past food intake events to calculate or estimate a sufficient insulin dose and insulin delivery schedule. For example, the insulin delivery system may consider the duration of one or more past food intake events and / or the average speed of eating of one or more past food intake events. In certain embodiments, the insulin delivery system may only consider parameters related to past food intake events that occurred within a specific time window prior to the current food intake event. In certain embodiments, the insulin delivery system may only consider parameters from past food intake events that occurred at or around the same time of day and / or the same day of the week as the current food intake event. In certain embodiments, the insulin delivery system may only consider parameters from one or more past food intake events that occurred at or near the same location as the current food intake event.

[0119] In some embodiments, an insulin delivery system may examine the effect of past insulin doses and delivery schedules on blood glucose readings or sensor outputs of other surrogates for blood glucose readings, such as interstitial fluid glucose readings, to calculate or estimate an insulin dose and determine a delivery schedule for a current food intake event.

[0120] The insulin delivery system may continuously or periodically process its logic and accordingly update the logic for insulin administration and / or insulin delivery scheduling based on one or more of the above parameters.

[0121] The patient management system may be generalized from insulin to the administration of drugs or substances that need to be administered in combination with food intake events, especially when the amount of the drug or other substance involves parameters associated with food intake events.

[0122] A filter may be applied to determine whether a meal is part of an eligible / applicable meal category.

[0123] The patient management system may be programmed to perform continuous or periodic and frequent assessments during a meal and upregulate or downregulate the schedule or insulin amount based on changes in the observed blood glucose level readings (or their surrogates, such as interstitial fluid glucose level readings). In a particular embodiment, insulin delivery may be suspended if the glucose level reading is below a specific threshold, or if a prediction algorithm embodied in the executable program code executed by the patient management system outputs a prediction that the glucose reading will be below a specific level at a future time if the current amount and schedule are executed.

[0124] Thus, the patient management system may send notifications and signals to the insulin delivery device while taking into account the start of eating, the rate of eating, the expected end of eating, the eating duration, and other factors. For example, the patient management system may instruct the insulin delivery device to deliver insulin at the start of eating and adjust it during eating using various inputs such as heart rate, eating rate, body temperature, etc. Thus, the patient management system may be part of a meal-aware, autonomous or semi-autonomous artificial pancreas. BRIEF DESCRIPTION OF THE DRAWINGS

[0126] Figure 1 A high-level functional diagram of a diet tracking and feedback system according to one embodiment is shown. The system for diet tracking and feedback may partially include one or more of the following: a food intake event detection subsystem 101, one or more sensors 102, a tracking and processing subsystem 103, a feedback subsystem 106, one or more data storage units 104, and a learning subsystem 105 that may perform non-real-time analysis. In some embodiments, Figure 1The components shown are implemented in electronic hardware, while in other embodiments, some components are implemented in software and executed by a processor. Some functions may share hardware and processor / memory resources, and some functions may be distributed. Functions may be implemented entirely within a sensor device (such as a wrist-worn wearable device), or functions may be implemented across a sensor device, a processing system (such as a smartphone) with which the sensor device communicates, and / or a server system that processes some function remote from the sensor device.

[0127] For example, a wearable sensor device may make measurements and communicate the measurements to a mobile device, which may process the data received from the wearable sensor device and use this information, possibly combined with other data inputs, to activate the tracking and processing subsystem 103. The tracking and processing subsystem 103 may be implemented on the mobile device, on the wearable sensor device, or on another electronic device. The tracking and processing subsystem 103 may also be distributed across multiple devices, such as, for example, across a mobile device and a wearable sensor device. Communication with a server may occur via the Internet, and the server may further process the data. Data or other information may be stored in a suitable format, distributed across multiple locations, or stored centrally in a recorded form or after some degree of processing. Data may be stored temporarily or permanently.

[0128] Figure 1 The first component of the system shown is the food intake event detection subsystem 101. The role of the food intake event detection subsystem 101 is to identify the start and / or end of a food intake event and convey the actual, possible, or impending occurrence of the event. The event may be, for example, an event related to a particular activity or behavior. Other examples of events that may be detected by the event detection subsystem 101 may be an operator on a production line or elsewhere performing a specific task or executing a specific process. Another example may be a robot or robotic arm performing a specific task or executing a specific process in a production department or elsewhere.

[0129] Generally speaking, the device detects what may be the start or a possible start of a food intake event, but as long as the device reasonably determines such a start / possible start, the device will function adequately. For clarity, this detection is referred to as the "presumed start" of a food intake event, and when various processes, operations, and components will perform some action or behavior related to the start of a food intake event, these various processes, operations, and components will accept the presumed start as the start, even if the presumed start is sometimes not actually the start of a food intake event.

[0130] In one embodiment, the detection and / or signaling of the occurrence that begins the identification of a food intake event coincides with the beginning of the identification of the food intake event. In another embodiment, it may occur at a time after the beginning of the identification of the food intake event. In another embodiment, it may occur at a time before the beginning of the identification of the food intake event. It is generally desirable for the signaling to be close to the beginning of the identification of the food intake event. In some embodiments of the present disclosure, it may be advantageous for the detection and / or signaling of the beginning of the identification of the food intake event to occur before the beginning of the food intake event. This may be useful, for example, if a message or signal is to be sent to the user, healthcare provider, or caregiver before the food intake event begins as a guidance mechanism to help direct the user's food intake decisions or eating habits.

[0131] Methods for event detection may include, but are not limited to, detection based on: monitoring the movement or position of the body or a particular part of the body; monitoring the movement, position, or posture of the arm; monitoring the movement, position, or posture of the hand; monitoring the movement, position, or posture of the fingers; monitoring the swallowing pattern; monitoring the movement of the mouth and lips; monitoring saliva; monitoring the movement of the cheeks or jaw; monitoring biting or teeth grinding; monitoring signals from the mouth, throat, and digestive system. The methods for detection may include visual, audio, or any other type of sensory monitoring of the person and / or his or her surrounding environment.

[0132] The monitored signals may be generated by a diet tracking and feedback system. Alternatively, they may be generated by a separate system but be accessible by the diet tracking and feedback system through an interface. Machine learning and other data analysis techniques may be applied to detect the beginning or possible beginning of a food intake event based on the monitored input signals.

[0133] In one example, a food intake detection system 101 may monitor the output of an accelerometer and / or gyroscope sensor to detect a possible biting posture or a possible sucking posture. Such postures may be determined by a posture processor that uses machine learning to extract postures from the sensor readings. The posture processor may be part of the processor of a wearable device or be located in another part of the system.

[0134] As described elsewhere herein, pose detection machine learning techniques can be used to detect a biting pose or a sucking pose, although other techniques are possible. The food intake detection system 101 can also assign a confidence level to the detected biting pose or sucking pose. The confidence level corresponds to the likelihood that the detected pose is indeed a biting or sucking pose. The food intake detection system can determine the start of a food intake event based on the detection of the pose and its confidence level without any additional input. For example, when the confidence level of the biting or sucking pose exceeds a pre-configured threshold, the food intake event detection system 101 can decide that the start of a food intake event has occurred.

[0135] Alternatively, when a possible biting or sucking pose has been detected, the food intake event detection system 101 can use additional input to determine the start or possible start of a food intake event. In one example, the food intake event detection system 101 can monitor other poses that are temporally close to determine whether the start of a food intake event has occurred. For example, when a possible biting pose is detected, the food intake event detection system 101 can wait for the detection of another biting pose within a specific time window after the detection of the first pose and / or with a specific confidence level before determining that the start of a food intake event has occurred.

[0136] Upon such detection, the food intake detection system 101 can place one or more circuits or components in a higher performance mode to further improve the accuracy of pose detection. In another example, the food intake event detection system 101 can consider the time of day or the user's location to determine whether the start or possible start of a food intake event has occurred. The food intake event detection system can use machine learning or other data analysis techniques to improve the accuracy and reliability of its detection capabilities. For example, training data obtained from the user and / or from other users at an earlier time can be used to train a classifier. The training data can be obtained by requesting user confirmation when a possible biting or sucking pose has been detected. Then a labeled data record can be created and stored in a memory readable by the pose processor, the labeled data record including features related to the pose and other situational features such as the time of day or location. Then a classifier can be trained on a labeled data set consisting of a multi-labeled data record set of the labeled data records, and the trained classifier model can then be used in the food intake event detection system to more accurately detect the start of a food intake event.

[0137] In another embodiment, the food intake detection subsystem may autonomously predict the likely onset of a food intake event using triggering factors. Methods for autonomously detecting the likely onset of a food intake event based on triggering factors may include, but are not limited to, monitoring a person's sleep pattern, monitoring a person's stress level, monitoring a person's activity level, monitoring a person's location, monitoring the people around a person, monitoring a person's vital signs, monitoring a person's degree of hydration, monitoring a person's degree of fatigue. In some cases, the food intake detection subsystem may monitor one or more specific trigger signals or trigger events over a longer time period and apply machine learning or other data analysis techniques in conjunction with the non-real-time analysis and learning subsystem 105 to predict the likely occurrence of the onset of a food intake event.

[0138] For example, in the absence of any additional information, it may be difficult to predict when a user will eat breakfast. However, if the system has records of the user's wake-up times and days of the week over multiple days, the system may use this historical pattern to determine the likely time for the user to eat breakfast. These records may be determined by the system, may have feedback from the user regarding their accuracy, or these records may be determined by the user and input via the system's user interface. The user interface may be the wearable device itself or, for example, a smartphone application. Thus, the system may process the correlations in the historical data to predict the time or time window when the user is most likely to eat breakfast based on the current day of the week and at what time the user wakes up. The non-real-time analysis and learning subsystem 105 may also use other trigger signals or trigger events to predict the time when the user will eat breakfast.

[0139] In another example, the non-real-time analysis and learning system 105 may record the stress level of a user over a specific time period. The stress level may be determined, for example, by monitoring and analyzing the user's heart rate or specific parameters related to the user's heart rate. The stress level may also be determined by analyzing the user's voice. The stress level may also be determined by analyzing the content of the user's messages or electronic communications. Other methods for determining the stress level are possible. The non-real-time analysis and learning system 105 may also record the occurrence of food intake events and certain characteristics of the food intake events, such as the speed of eating, the amount of food consumed, the time interval between food intake events, etc., over the same time period. Then, by analyzing the historical data of the stress level, the occurrence of food intake events, and the characteristics of the food intake events, and by examining the correlations in the historical data of the stress level, the occurrence of food intake events, and the characteristics of the food intake events, it may be possible to predict the likelihood that the user will start a food intake event in a specific time window in the future, or to predict at what time window in the future the user will most likely start a food intake event. It may also be possible to predict the characteristics of the food intake event, such as, for example, the speed of eating or the amount consumed.

[0140] In certain embodiments, the non-real-time analysis and learning subsystem 105 may use historical data from different users or a combination of data from other users and from the wearer, and use similarities (such as age, gender, medical conditions, etc.) among one or more of the different users and wearers to predict a likely start of a food intake event of the wearer.

[0141] In still other examples, the non-real-time analysis and learning subsystem 105 may use methods similar to the methods described herein to predict when a user is most likely to relapse into a binge eating episode or is most likely to start eating a convenience snack.

[0142] A variety of sensors may be used for such monitoring. The signals being monitored may be generated by the dietary tracking and feedback system. Alternatively, they may be generated by a separate system but be accessible by the dietary tracking and feedback system for processing and / or used as trigger signals. Machine learning and other data analysis techniques may also be applied to predict certain other characteristics of a likely intake event, such as the type and / or amount of food that may be consumed, the rate at which a person may eat, the satisfaction a person will derive from consuming the food, and the like.

[0143] Machine learning processes performed as part of gesture recognition may use external data to further refine their decisions. This may be done by the non-real-time analysis and learning subsystem process. The data analysis process may, for example, consider food intake events detected by a gesture-sensing based food intake detection system and a gesture-sensing based tracking and processing system, thus forming a second layer of machine learning. For example, over a certain time period, food intake events and characteristics associated with these food intake events, such as eating rate, amount of food consumed, food composition, etc., are recorded while also tracking other parameters that are not directly linked or may not have an obvious link to the food intake events. This may be, for example, location information, the time of day a person wakes up, stress levels, certain patterns of a person's sleep behavior, calendar event details (including time, event location, and list of participants), phone call information (including time, duration, phone number, etc.), email metadata (such as time, duration, sender, etc.). The data analysis process then identifies patterns and correlations. For example, it may determine a correlation between the number of calendar events during the day and the characteristics of food intake events in the evening. This may be because a user is more likely to start eating a snack when arriving home or to have a larger and / or more hurried dinner when the number of calendar events for the day exceeds a certain threshold. With subsystem 105, it is possible to predict food intake events and characteristics from other signals and events that do not have an obvious link to food intake. The processing and analysis subsystem may use additional situational metadata (such as location, calendar information, day of the week, or time of day) to make such determinations or predictions.

[0144] Optionally, the processing and analysis of one or more sensor inputs and / or one or more images over a longer time period using machine learning or other data analysis techniques can also be used to estimate the duration of a food intake event or to predict that a food intake event is likely to end or is approaching its end.

[0145] In another embodiment, some user input 108 may be necessary or desirable to appropriately or more accurately detect the start and / or end of a food intake event. Such user input may be provided in addition to external input and the input received from sensor 102. Alternatively, one or more user inputs may be used in place of any sensor input. User input may include, but is not limited to, activating the device, pressing a button, touching or moving the device or a particular part of the device, taking a picture, issuing a voice command, making a selection on a screen, or entering information using hardware and / or software (which may include, but is not limited to, a keyboard, a touch screen, or voice recognition technology). If one or more user inputs are required, it is important to conceive and implement the user interaction in a way that minimizes the negative impact on normal human activities or social interactions.

[0146] The food intake event detection subsystem 101 can combine multiple methods to autonomously detect or predict the actual, likely, or impending start and / or end of a food intake event.

[0147] Another component of the system is the tracking and processing subsystem 103. In a preferred embodiment of the present disclosure, this subsystem interfaces with the food intake event detection subsystem 101 at 109 and is activated when it receives a signal from the food intake event detection subsystem 101 that an event has actually, likely, or is about to start, and is deactivated when it receives a signal from the food intake event detection subsystem 101 that an event has actually, likely, or is about to end or at some time after that. When a food intake event is detected as starting, the device may trigger the activation of other sensors or components of the food intake tracking system and may also trigger the deactivation of these sensors or components when a food intake event is detected as ending.

[0148] Application: Food Logging

[0149] In existing food logging methods, if a user forgets to enter an entry or for another reason (intentionally or unintentionally) does not enter an entry, there is no record or history of the eating event. This results in an incomplete, inaccurate food diary that is of significantly reduced usability for treatment purposes. In addition, the content recorded in existing food logging methods is typically limited to self-reporting of ingredients and amounts. There is no information about important characteristics of how the food is consumed (e.g., the speed, duration of the meal).

[0150] Figure 1The monitoring system shown can be used to automate food logging or reduce conflicts in food logging. In one embodiment of the present disclosure, the event detection subsystem 101 uses information inferred from motion sensors such as accelerometers or gyroscopes to detect and monitor eating or drinking events based on the hand posture of the subject. In other embodiments, different sensor inputs can be used to infer eating or drinking events. Other sensors can include, but are not limited to, heart rate sensors, pressure sensors, proximity sensors, glucose sensors, optical sensors, image sensors, cameras, photometers, thermometers, ECG sensors, and microphones.

[0151] Record the output of the event detection subsystem 101 indicating the occurrence of an eating or drinking event of the subject. The event detection subsystem 101 can perform additional processing to obtain additional relevant information about the event, such as start time, end time, a measure representing the eating or drinking speed of the subject, and a measure representing the amount consumed. This additional information can also be recorded.

[0152] The detection of the occurrence of an eating or drinking event can be recorded as an entry in the food log. Additional information associated with the eating or drinking event that can be obtained from the event detection subsystem can also be recorded as part of the consumption event entry in the food log. This can replace manually entering each eating event.

[0153] The information to be recorded about an eating or drinking event can include the time of the event, the duration of the event, the location of the event, a measure related to the consumption speed, a measure related to the amount consumed, the eating method, the utensils used, etc.

[0154] Application: Medication Adherence

[0155] Since the event system is capable of monitoring events and postures and determining consumption, this can be used to automatically monitor a medication administration schedule that defines when medication or other actions need to be taken with what food. This can be done with a specific meal category (such as breakfast), at a specific time of day, etc. The medication administration schedule may specify whether the patient should take the medication while eating or drinking. Optionally, the medication administration schedule can also specify how much food or liquid the patient should consume.

[0156] For example, when medication needs to be taken at a specific time of day, the medication compliance system can monitor the time and issue a warning when it is time to take the medication. It can then also activate the event detection subsystem (if it is not already activated) and start monitoring the output of the event detection subsystem. It waits for a notification confirmation from the user that he / she has taken the medication.

[0157] If a confirmation is received and if the medication administration protocol specifies that the medication needs to be taken with food or liquid, the medication compliance system monitors the output from the eating / drinking event detection subsystem and determines whether the rules specified by the medication administration protocol are met. This can be as simple as confirming that an eating or drinking event has occurred either with or shortly after medication ingestion. In cases where the medication administration protocol specifies that a minimum amount of food or fluid needs to be consumed, the medication compliance system can monitor additional output from the event detection subsystem (metrics related to the amount consumed) to confirm that this condition is met. It is also possible to determine whether different rules / logic of the medication administration protocol have been satisfied.

[0158] If a confirmation is not received, the medication compliance subsystem can issue a second notification. Additional notifications are also possible. After a pre-configured number of notifications, the medication compliance subsystem can issue an alert. The alert can be issued as a text message to the user or can be sent to a remote server over the Internet or via a cellular connection (e.g., to a hospital, caregiver).

[0159] If the medication needs to be taken with a specific meal category, the medication compliance system can monitor the output of the eating detection subsystem. When an eating event is detected, it will use logic to determine the applicable meal category. If the meal category matches the category described in the medication administration protocol, it will issue a notification to remind the user to take his / her medication. If the medication administration protocol specifies that food or liquid should accompany medication consumption, the monitoring logic outlined above can be implemented to determine that the medication administration protocol has been followed.

[0160] The medication compliance system can monitor the output of the eating event detection subsystem to determine whether the start of an eating event has occurred and to determine whether the eating event belongs to the applicable meal category. When an eating event is detected and the meal category matches the category described in the medication administration protocol, certain actions may be taken, such as it can activate an object information retrieval system (e.g., NFC tag, imaging) to gather more information about the object with which the user is interacting. In this way, it can obtain information about the medication from an NFC tag attached to the pillbox or medication container. In another use, it can verify whether the medication matches the medication specified by the medication administration protocol and / or issue a notification to remind the user to take his / her medication. If the medication administration protocol specifies that food or liquid should accompany medication consumption, the monitoring logic outlined above can be implemented to determine that the medication administration protocol has been followed.

[0161] The system may also incorporate details about the medication obtained from an object information collection system or via different notification methods, record confirmation in response to a notification from the user that he / she has taken the medication, request additional information about the medication from the user, and / or send follow-up questions to the user at a preconfigured time after the user has taken the medication to obtain additional input. (For example, a query about how the user feels, a query about the level of pain, a prompt to measure blood glucose levels).

[0162] Additional Embodiments

[0163] In another embodiment of the present disclosure, the tracking and processing subsystem may be activated and / or deactivated independently of any signal from the food intake detection subsystem. It may also be possible to track and / or process certain parameters independently of any signal from the food intake detection subsystem, while tracking and / or processing of other parameters may be initiated only when a signal is received from the food intake event detection subsystem.

[0164] The sensor input may be the same as or similar to the input sent to the food intake event detection subsystem. Alternatively, different and / or additional sensor inputs may be collected. The sensors may include, but are not limited to, accelerometers, gyroscopes, magnetometers, image sensors, cameras, optical sensors, proximity sensors, pressure sensors, odor sensors, gas sensors, global positioning system (GPS) circuitry, microphones, galvanic skin response sensors, thermometers, ambient light sensors, UV sensors, electrodes for electromyogram (“EMG”) potential detection, bioimpedance sensors, spectrometers, glucose sensors, touchscreens, or capacitive sensors. Examples of sensor data include motion data, body temperature, heart rate, pulse, galvanic skin response, blood or body chemistry, audio or video recordings, and other sensor data, depending on the sensor type. The sensor input may be communicated to the processor wirelessly or via wires in analog or digital form, with gating and / or clock circuitry in between or provided directly.

[0165] The processing methods used by the tracking and processing subsystem may include, but are not limited to, data manipulation, algebraic calculations, geotagging, statistical calculations, machine learning, computer vision, speech recognition, pattern recognition, compression, and filtering.

[0166] The data collected may optionally be stored temporarily or permanently in a data storage unit. The tracking and processing subsystem may use an interface to the data storage unit to place data or other information into the data storage unit and retrieve data or other information from the data storage unit.

[0167] In a preferred embodiment of the present disclosure, data collection, processing, and tracking occur autonomously and do not require any special user intervention. The parameters tracked may include, but are not limited to, the following: location, temperature of the surrounding environment, ambient light, ambient sound, biometric information, activity level, image capture of food, food name and description, portion size, fluid intake, calorie and nutritional information, mouthful count, bite count, suck count, duration between successive bites or sucks, and duration of the food intake event. The parameters tracked may also include, for each bite or suck, the duration that the user's hand, arm, and / or eating utensil is close to the user's mouth, and the duration that the components of the bite or suck remain in the user's mouth before swallowing. The method may vary based on what sensor data is available.

[0168] In other embodiments of the present disclosure, some user intervention may be required or may be desirable to achieve, for example, greater accuracy or to input additional details. User intervention may include, but is not limited to, activating the device or a specific function of the device, holding the device in place, taking a photo, adding a voice annotation, recording a video, making a correction or adjustment, providing feedback, making a data entry, taking a measurement of the food or a food sample. The measurement may include, but is not limited to, non-destructive techniques, such as obtaining one or more spectrograms of the food item, or chemical methods that may require sampling the food.

[0169] The tracking and processing subsystem 103 typically processes the sensor data and user input in real time or near real time. There may be some latency, for example, to conserve power or to avoid certain hardware limitations, but in some embodiments, the processing occurs during the food intake event, or in the case of tracking outside of the food intake event, approximately at the time when the sensor or user input is received.

[0170] In certain specific implementations or in certain situations, there may not be real-time or near-real-time access to the processing unit required to perform some or all of the processing. This may be due, for example, to power consumption or connectivity limitations. Other motivations or reasons are also possible. In such cases, these inputs and / or partially processed data may be stored locally until later access to the processing unit becomes available.

[0171] In a particular embodiment of the present disclosure, sensor signals tracking the movement of a person's arm, hand, or wrist can be sent to a tracking and processing subsystem 103. The tracking and processing subsystem 103 can process and analyze such signals to identify food that has been consumed or is likely to have been consumed in a single bite or liquid that has been consumed in a single suck. The tracking and processing subsystem 103 can also process and analyze such signals to identify and / or quantify other aspects of the eating behavior, such as, for example, the time interval between bites or sucks, the speed of the hand-to-mouth movement, etc. The tracking and processing subsystem 103 can also process and analyze such signals to identify certain aspects of the eating method, such as, for example, whether the person is eating with a fork or spoon, drinking from a glass or can, or consuming food without using any utensils.

[0172] In a particular example, it is possible that the wearer rotates his or her wrist in one direction when bringing a dining utensil or hand to the mouth during a bite and in another direction when sucking liquid. The amount of rotation of the wearer's wrist as it moves toward or away from the mouth and the duration for which the wrist is held at a higher angle of rotation may also be different for drinking and eating postures. Other metrics can be used to distinguish between eating and drinking postures or to distinguish differences in the eating method. Combinations of multiple metrics can also be used. Other examples of metrics that can be used to distinguish between eating and drinking postures or to distinguish differences in the eating method include, but are not limited to, the change in roll angle over time or approximate time from the start or approximate start of the posture until the hand reaches the mouth, the change in roll angle over time or approximate time from when the hand approaches the mouth until the end or approximate end of the posture, the variance of accelerometer or gyroscope readings across one or more axes during the duration that the hand is approaching the mouth or during a duration centered on when the hand is closest to the mouth or during a duration that may not be centered on when the hand is closest to the mouth but includes the time when the hand is closest to the mouth, the variance of the magnitude of accelerometer readings during the duration that the hand is approaching the mouth or during a duration centered on when the hand is closest to the mouth or during a duration that may not be centered on when the hand is closest to the mouth but includes the time when the hand is closest to the mouth, the maximum value of the magnitude of accelerometer readings during the duration that the hand is approaching the mouth or during a duration centered on when the hand is closest to the mouth or during a duration that may not be centered on when the hand is closest to the mouth but includes the time when the hand is closest to the mouth. The magnitude of the accelerometer reading can be defined as the square root of the acceleration in each orthogonal direction (e.g., sensing accelerations in the x, y, and z directions and calculating SQRT(ax 2 + ay 2 + az 2 ))。

[0173] The position of the hand relative to the mouth can be determined, for example, by monitoring the pitch of the worn device and thus the pitch of the wearer's arm. The time corresponding to the peak of the pitch can be used as the moment when the hand is closest to the mouth. The time when the pitch starts to rise can be used, for example, as the start time of the gesture. The time when the pitch stops descending can be used, for example, as the end time of the gesture.

[0174] Other definitions of the closest mouth position, start of movement, and end of movement are also possible. For example, alternatively, the time when the roll changes direction can be used to determine the time when the arm or hand is closest to the mouth. Alternatively, the time when the roll stops changing in a particular direction or at a particular speed can be used to determine the start time of the movement towards the mouth.

[0175] The tracking and processing subsystem can also process and analyze such signals to determine the appropriate or preferred time to activate other sensors. In one particular example, the tracking and processing subsystem can process and analyze such signals to determine the appropriate or preferred time to activate one or more cameras to capture one or more static or dynamic images of the food. By using sensors that track the movement of the arm, hand, finger, or wrist and / or the orientation and position of the camera to activate the camera and / or automate the image capture process, the complexity, power, and power consumption of the image capture and image analysis system can be significantly reduced, and in some cases, better accuracy can be achieved. This also significantly reduces any privacy infringement issues as it is then possible to more precisely control the timing of image capture and make it conform to the camera focusing on the food.

[0176] For example, the processor may analyze the motion sensor inputs from an accelerometer, gyroscope, magnetometer, etc., and thus may identify the optimal time to activate the camera based on when the processor determines that the field of view of the camera encompasses the food to be photographed and capture the photo and trigger the camera at that time. In one example, the processor determines the start of an eating event and signals the wearer to capture an image of the food being eaten, and also determines the end of the eating event and signals the wearer again to capture an image of what is left of the food or the plate, etc. Such images can be processed to determine the consumption amount and / or confirm the consumption amount determined by the processor. In some embodiments, image processing can be used as part of the feedback for training the machine learning used by the processor.

[0177] In some embodiments, the system can use sensors that track the movement of the wearer's arm or hand and activate the camera only when the system determines based on motion sensing that the arm or hand is close to the mouth. In another example, the system can activate the camera at some time between the start of the movement towards the mouth and the time when the arm or hand is closest to the mouth. In yet another example, the system can activate the camera at some time between the time when the arm or hand is closest to the mouth and the end of the movement away from the mouth.

[0178] As mentioned above, the position of the hand relative to the mouth can be determined by monitoring the upward pitch indicating the start time of the movement towards the mouth and the downward pitch indicating the end time of the movement. Other definitions of the most recent mouth position, start of movement, and end of movement are also possible.

[0179] The position of the hand relative to the mouth can be determined, for example, by monitoring the pitch of a worn device and thus the pitch of the wearer's arm. The time corresponding to the peak of the pitch can be used as the moment when the hand is closest to the mouth. The time when the pitch starts to rise can be used, for example, as the start time of the gesture. The time when the pitch stops falling can be used, for example, as the end time of the gesture.

[0180] The processing and analysis of sensor signals tracking the movement of the user's arm, hand, or wrist can be combined with other proposed methods (such as image capture of food when it enters the mouth) to build in redundancy and improve the robustness of the diet tracking and feedback system. For example, through the processing and analysis of the user's arm, hand, or wrist movement, information related to bite counts and bite patterns will still be retained even if the camera is blocked or tampered with.

[0181] One or more of the sensor inputs can be static or streaming images obtained from one or more camera modules. Such images may require some degree of processing and analysis. Among other methods, the processing and analysis methods can include one or more of the following methods: compression, deletion, resizing, filtering, image editing, and computer vision techniques to identify objects (such as specific foods or dishes) or features (such as portion sizes).

[0182] In addition to measuring bite counts and suck counts, the processor may also analyze details such as termination and duration to determine bite and suck sizes. Measuring the time that the wearer's hand, eating utensil, or fluid container is close to their mouth can be used to derive a "close to mouth" duration, which in turn is used as an input to generate an estimate of the size of a bite or suck. The amount of wrist rotation during sucking may be useful for hydration tracking.

[0183] Measuring the amount of rotation of the wrist during one or more time periods within the start and end of a gesture can also be used to estimate the size of a bite or suck. For example, the system can measure the amount of rotation of the wrist from a time after the start of the gesture to the time when the arm or hand is closest to the mouth. The time corresponding to the peak of the pitch can be used as the moment when the hand is closest to the mouth. The time when the pitch starts to rise can be used, for example, as the start time of the movement towards the mouth. The time when the pitch stops descending can be used, for example, as the end time of the movement away from the mouth. Other definitions of the closest mouth position, start of movement, and end of movement are also possible. For example, the time when the roll changes direction can alternatively be used as the time when the arm or hand is closest to the mouth. The time when the roll stops changing in a particular direction or at a particular speed can be used as the start time of the movement towards the mouth. One or more static or streaming images can be analyzed and / or compared by the tracking and processing subsystem for one or more purposes, including but not limited to the identification of food items, the identification of food ingredients, the identification or derivation of nutritional information, the estimation of portion sizes, and the inference of certain eating behaviors and eating patterns.

[0184] As an example, computer vision techniques (optionally combined with other image manipulation techniques) can be used to identify food categories, specific food items, and / or estimate portion sizes. Alternatively, the Mechanical Turk process or other crowdsourcing methods can be used to manually analyze the images. Once the food category and / or specific food item has been identified, this information can be used to retrieve nutritional information from one or more food / nutrition databases.

[0185] As another example, information about the user's eating or drinking speed can be inferred by analyzing and comparing multiple images captured at different times during a food intake event. As yet another example, images (optionally combined with other sensor inputs) can be used to distinguish between table meals and finger foods or snacks. As yet another example, the analysis of one image taken at the start of a food intake event and another image taken at the end of the food intake event can provide information about the amount of food actually consumed.

[0186] In general, sensor data is received by a processor that analyzes the sensor data, possibly also analyzing previously recorded data and / or metadata about the person from whom the sensor data was sensed. The processor performs calculations (such as those described herein) to derive a sequence of sensed gestures. The sensed gestures may be one of the gestures described elsewhere herein, as well as the relevant data about the sensed gestures, such as the time of occurrence of the sensed gesture. The processor analyzes the sequence of sensed gestures to determine the start of a behavioral event, such as the start of an eating event.

[0187] The determination of the start of a feeding event can be based on a sequence of sensed postures, but it can also be based on the detection of a single event (which may have a scenario not based on postures). For example, if the system detects a biting posture with a reasonably high confidence level, the processor may consider the detection of that individual posture as the start of a feeding event. The processor can also analyze the sequence of sensed postures to determine the end of a behavioral event. The determination of the end of a feeding event can also be based on the absence of the detected event. For example, if a biting posture is not detected within a given time period, the processor may assume that the feeding event has ended.

[0188] Knowing the start and end of a behavioral event allows the processor to more accurately determine postures, since these postures are incorporated into a scenario and / or the processor can enable additional sensors or put one or more sensors or other components into a higher performance state, such as in the examples described elsewhere in this document. Knowing the start and end of a behavioral event also allows for power savings, because in some cases it is possible to put the worn device into a lower power consumption mode outside of certain behavioral events. Additionally, aggregating individual postures into events (which may be combined with previously recorded data about similar behavioral events from the same user or other users in the past) allows the processor to derive meaningful characteristics about the behavioral event. For example, the eating speed during breakfast, lunch, and dinner can be determined in this way. As another example, if the processor has the state of the current behavior and the current behavior is brushing teeth, postures that might otherwise appear to be eating or drinking postures will not be interpreted as eating or drinking postures, so sucking during toothbrushing will not be interpreted as consumption of a liquid. Behavioral events can be general events (eating, walking, brushing teeth, etc.) or more specific (eating with a spoon, eating with a fork, drinking from a glass, drinking from a can, etc.).

[0189] While it is possible to decode indirect postures (such as detecting an indicative posture and then determining the object that the sensed person is pointing at), of interest are postures that are directly part of the detected event itself. Some postures are accidental postures, such as postures associated with operating the device, in which case the accidental postures may be excluded from consideration.

[0190] In a particular example, the system uses a set of sensors to determine the start of a feeding event with a certain confidence level, and if the confidence level is higher than a threshold, the system activates additional sensors. Thus, an accelerometer sensor may be used to determine the start of a feeding event with a high confidence level, but the gyroscope is placed in a low-power mode to extend battery life. The accelerometer alone can detect postures indicating a possible bite or suck (e.g., upward arm or hand movement or hand or arm movement generally in the direction of the mouth) or postures generally indicating the start of a feeding event. Upon detecting a first posture that generally indicates a possible start of a feeding event, additional sensors (e.g., a gyroscope, etc.) may then be enabled. If a subsequent bite or suck posture is detected, the processor determines the start of a feeding event with a higher confidence level.

[0191] Event Detection

[0192] Knowing the start / end of a behavioral event allows the processor to place one or more sensors or other components in a higher performance state during the duration of the behavioral event. For example, when the start of a behavioral event has been determined, the processor can increase the sampling rate of the accelerometer and / or gyroscope sensors used to detect postures. As another example, when the start of a behavioral event has been determined, the processor can increase the update rate at which sensor data is sent to the electronic device 219 for further processing to reduce latency.

[0193] Referring again to Figure 1 , in addition to the tracking and processing subsystem, Figure 1 the system of may also include a non-real-time analysis and learning subsystem 105. The non-real-time analysis and learning subsystem 105 can perform analysis on larger data sets that take a longer time to collect (such as historical data across multiple food intake events and / or data from a larger population). The methods used by the non-real-time analysis and learning subsystem 105 may include, but are not limited to, data manipulation, algebraic calculations, geotagging, statistical calculations, machine learning and data analysis, computer vision, speech recognition, pattern recognition, compression, and filtering.

[0194] Among other things, the methods used by the non-real-time analysis and learning subsystem 105 may include data analysis of larger data sets collected over a longer time period. As an example, one or more data inputs may be captured over a longer time period and across multiple food intake events to train a machine learning model. Such data inputs are hereinafter referred to as training data sets. It is generally desirable that the time period (hereinafter referred to as the training period) for collecting the training data set be long enough such that the data collected represents a person's typical food intake.

[0195] Among other things, the training dataset may include one or more of the following food intake-related information: the number of bites per food intake event, the total bite count, the duration of the food intake event, the speed of food intake or the time between subsequent counts, the classification of food intake components, such as separating solid foods from liquids or separating table meals from snacks or finger foods. This information can be derived from one or more sensor inputs.

[0196] The training dataset may also include images of each or most of the items consumed during each food intake event within the training period. Computer vision and / or other methods can be used to process these images to identify food categories, specific food items, and estimate portion sizes. This information can then in turn be used to quantify the calorie content and / or macronutrient composition, such as the amounts of carbohydrates, fats, proteins, etc., of the food items.

[0197] In cases where the food is not fully consumed, it may be desirable to take a photo of the food item at the start of the food intake event and a photo at the end of the food intake event to derive the portion of the food that was actually consumed. Other methods (including but not limited to manual user input) can be used to add portion size information to the data in the training dataset.

[0198] The training dataset may also include metadata that does not directly quantify food intake and / or eating behaviors and patterns, but can indirectly provide information that can be associated with food intake events and / or eating behaviors, and / or can be a trigger for food intake events to occur, or can affect eating habits, patterns, and behaviors. Among other things, such metadata may include one or more of the following: gender, age, weight, socioeconomic status, temporal information about the food intake event (such as date, time of day, day of the week), information about the location of the food intake event, vital sign information, hydration level information, and other physical, mental, or environmental conditions, such as hunger, stress, sleep, fatigue level, addiction, disease, social stress, and exercise.

[0199] One or more training datasets can be used to train one or more machine learning models, and then one or more components of the diet tracking and feedback system can use the one or more machine learning models to predict certain aspects of food intake events and eating patterns and behaviors.

[0200] In one example, a trainable model can be trained to predict the occurrence of a food intake event based on the tracking of one or more metadata that can influence the occurrence of the food intake event. Other characteristics related to a possible food intake event can also be predicted, such as the type and / or amount of food that may be consumed, the rate at which a person may eat, the duration of the food intake event, and / or the satisfaction that a person will derive from consuming the food. Among other things, the metadata can include one or more of the following: gender, age, weight, socioeconomic status, temporal information about the food intake event (such as date, time of day, day of the week), information about the location of the food intake event, vital sign information, hydration level information, and other physical, mental, or environmental conditions, such as hunger, stress, sleep, fatigue level, addiction, disease, social stress, and exercise.

[0201] In another example, machine learning and data analysis can be applied to derive metrics that can be used outside of a training period to estimate calorie or other macronutrient intake, even if there is limited or no available food intake sensor input or imagery. The metadata can be used to further adjust the value of such metrics based on additional situational information. Among other things, the metadata can include one or more of the following: gender, age, weight, socioeconomic status, temporal information about the food intake event (such as date, time of day, day of the week), information about the location of the food intake event, information about common food categories, vital sign information, hydration level information, calendar event information, phone call logs, email logs, and other physical, mental, or environmental conditions, such as hunger, stress, sleep, fatigue level, addiction, disease, social stress, and exercise.

[0202] An example of such a metric would be "calories per bite". By combining the bite count with calorie information obtained from image processing and analysis, the "calories per bite" metric can be established from one or more training datasets. Then this metric can be used outside of the training period to estimate calorie intake based solely on the bite count, even if there are no available images or only limited available images.

[0203] Another metric could be "typical bite size". By combining the bite count with portion size information obtained from image processing and analysis, the "typical bite size" metric can be established from one or more training datasets. Then this metric can be used outside of the training period to estimate portion size based solely on the bite count, even if there are no available images or only limited available images. It can also be used to identify differences between reported food intake and measured food intake based on the bite count and the typical bite size. The differences can indicate that the user did not report all of the food items that he or she consumed. Or alternatively, it can indicate that the user did not consume all of the food that he or she reported.

[0204] The biting action may be determined by a processor reading an accelerometer and a gyroscope sensor or more generally by reading a motion sensor that senses the movement of a body part of the wearer. Then, by counting the bites, the total number of bites can be inferred. Additionally, the processor may use the time series of the bites to infer an eating pattern.

[0205] The non-real-time analysis and learning subsystem 105 can also be used to track, analyze, and help visualize larger historical data sets, track progress against specific fixed or configured goals, and help establish such goals. It can also be used to identify and track streaks and compare performance with friends or a larger (optionally anonymous) group of people.

[0206] In addition, in some embodiments, in addition to other data manipulation and processing techniques, the non-real-time analysis and learning subsystem 105 can apply machine learning and data analysis techniques to predict the proximity or likelihood of certain health problems, diseases, and other medical conditions. In such cases, training typically requires historical food intake and / or eating behavior data captured over a long time period and across a large population. It is also desirable for the training data set to include additional metadata such as age, weight, gender, geographic information, socioeconomic status, vital signs, medical record information, calendar information, phone call logs, email logs, and / or other information. The prediction can then be used to help guide health outcomes and / or prevent or delay the onset of certain diseases such as, for example, diabetes.

[0207] The non-real-time and learning subsystem 105 can also be used to learn and extract more information about other aspects, including but not limited to one or more of the following: the user's diet and food preferences, the user's dining preferences, the user's restaurant preferences, and the user's food consumption. The food intake tracking and feedback system can use such information to make specific recommendations to the user. The food intake tracking and feedback system described herein can also interface or integrate with other systems, such as a restaurant reservation system, an online food or meal ordering system, and other systems that facilitate, streamline, or automate the process of food or meal ordering or reservation.

[0208] The non-real-time and learning subsystem 105 can also be used to monitor food intake over a long time period and detect any particularly long events of no food intake activity. Among other things, such events can indicate that the user has stopped using the device, has intentionally or unintentionally tampered with the device, there is a functional defect in the device, or a medical condition such as, for example, the user has fallen, died, or lost consciousness. The detection of a particularly long event of no food intake activity can be used to send a notification or alert to the user, one or more of his caregivers, a monitoring system, an emergency response system, or a third party that may have a direct or indirect interest in being informed of the occurrence of such an event.

[0209] Figure 1 Another component of the system shown is the feedback subsystem 106. The feedback subsystem 106 provides one or more feedback signals to the user or anyone else to whom such feedback information may be relevant. The feedback subsystem 106 can provide real-time or near-real-time feedback related to a particular food intake event. Real-time or near-real-time feedback generally refers to feedback given approximately at the time of the food intake event. This can include feedback given during the food intake event, feedback given before the start of the food intake event, and feedback given at some time after the end of the food intake event. Alternatively or in addition, the feedback subsystem can provide feedback to the user that is not directly related to a particular food intake event.

[0210] The feedback methods used by the feedback subsystem can include, but are not limited to, haptic feedback, whereby a haptic interface is used that applies force, vibration, and / or motion to the user; audio feedback, where a speaker or any other audio interface can be used; or visual feedback, whereby a display, one or more LEDs, and / or a projected light pattern can be used. The feedback subsystem can use only one feedback method or a combination of more than one feedback method.

[0211] The feedback subsystem can be implemented in hardware, software, or a combination of hardware and software. The feedback subsystem can be implemented on the same device as the food intake event detection subsystem 101 and / or the tracking and processing subsystem 103. Alternatively, the feedback subsystem can be implemented in a device separate from the food intake event detection subsystem 101 and / or the tracking and processing subsystem 103. The feedback subsystem 106 can also be distributed across multiple devices, some of which can optionally house Figure 1 parts of some of the other subsystems shown.

[0212] In one embodiment, the feedback subsystem 106 can provide feedback to the user to signal the actual, possible, or impending start of a food intake event. The feedback subsystem 106 can also provide feedback to the user during the food intake event to remind the user of the fact that a food intake event is occurring, improve present awareness, and / or encourage mindful eating. The feedback subsystem can also provide guidelines on recommended portion sizes and / or food composition, or provide alternative suggestions for eating. The alternative suggestions can be default suggestions, or they can be customized suggestions that have been programmed or configured by the user at different times.

[0213] The feedback signal can include, but is not limited to, periodic haptic feedback signals on a wearable device, audible alerts, display messages, or one or more notifications pushed to the display of his or her mobile phone.

[0214] When a signal indicating the start of a food intake event is received or at some time after, the user may confirm that a food intake event is indeed occurring. The confirmation can be used, for example, to trigger logging of the event or can cause the system to prompt the user for additional information.

[0215] In another embodiment of the present disclosure, the feedback subsystem initiates feedback during a food intake event only when one or more of the tracked parameters reach a specific threshold. As an example, if the time between successive bites or sucks is being tracked, feedback to the user can be initiated when that time (possibly averaged over multiple bites or sucks) is shorter than a fixed or programmed value to encourage the user to slow down. Similarly, feedback can be initiated when a fixed or programmed bite or suck count is exceeded.

[0216] In a feedback subsystem that provides feedback during a food intake event, the feedback provided by the feedback subsystem typically pertains to details of that specific food intake event. However, other information can also be provided by the feedback subsystem, including but not limited to information related to previous food intake events, biometric information, mental health information, activity or health level information, and environmental information.

[0217] In yet another embodiment of the present disclosure, the feedback subsystem 106 can send one or more feedback signals outside of a specific food intake event. In an example of such an embodiment, environmental temperature and / or other parameters that can affect hydration needs or otherwise directly or indirectly measure the degree of hydration can be tracked. Such tracking can occur continuously or periodically, or otherwise independently of a specific food intake event. If one or more such parameters exceed a fixed or programmed threshold, a feedback signal can be sent to, for example, encourage him / her to take steps to improve hydration. The feedback subsystem 106 may evaluate its inputs and determine that the preferred time to send feedback is not during a food intake event, but rather after the food intake event has ended. Some of the inputs to the feedback subsystem 106 may come from the food intake event, but some inputs may come from other monitoring that is not measured directly as a result of the food intake event.

[0218] The decision to send a feedback signal can be independent of any food intake tracking, such as in the embodiment described in the previous paragraph. Alternatively, such a decision can be linked to food intake tracking across one or more food intake events. For example, in one embodiment of the present disclosure, the system described above can be modified to also directly or indirectly track a person's fluid intake. For different environmental temperature ranges, this embodiment can have pre-programmed fluid intake requirement thresholds. If, for the measured environmental temperature, the person's fluid intake that may be tracked and accumulated over a specific time period does not meet the threshold for the environmental temperature, the system can send a feedback signal to advise the person to increase his or her fluid intake level.

[0219] Similarly, among other parameters, feedback signals or recommendations related to food intake may be associated with activity level, sleep level, social context or environment, health or disease diagnosis, and tracking of health or disease monitoring.

[0220] In yet another embodiment of the present disclosure, the feedback subsystem 106 may initiate a feedback signal when it has detected that a food intake event has started or is approaching or is likely. In such an embodiment, the feedback may be used, for example, as a cue to remind the user to record in a log a food intake event or certain aspects of a food intake event that cannot be automatically tracked, or to influence or guide a person's food intake behavior and / or the amount or composition of the food consumed.

[0221] The information provided by the feedback subsystem 106 may include, but is not limited to, information related to eating patterns or habits; information related to specific edible substances, such as, for example, name, description, nutritional composition; reviews, ratings, and / or images of food items or dishes; information related to triggers for food intake; information related to triggers for eating patterns or habits; biometric or environmental information; or other information that may be directly or indirectly related to a person's general food intake behavior, health, and / or wellness.

[0222] The feedback subsystem 106 may include a display of images of food items or dishes that have been consumed or are likely to be consumed. Additionally, the feedback subsystem 106 may include additional information about the food items or dishes, such as, for example, an indication of how healthy they are, nutritional composition, background story or preparation details, ratings, personalized feedback, or other personalized information.

[0223] In certain embodiments of the present disclosure, the information provided by the feedback subsystem 106 may include non-real-time information. The feedback subsystem 106 may, for example, include feedback based on the processing and analysis of historical data and / or data that has been accumulated within a larger user group. The feedback subsystem 106 may also provide feedback independent of the tracking of any specific parameters. As an example, the feedback subsystem 106 may provide general food, nutrition, or health information or guidelines.

[0224] In certain embodiments of the present disclosure, the user may interact with the feedback subsystem 106 and provide an input 116. For example, the user may suppress or customize some or all of the feedback signals.

[0225] Among other things, non-real-time feedback may include historical data, trend overviews, personal records, streaks, performance compared to a goal or compared to friends or other people or groups, notifications of alarm trends, feedback from friends, social networks and social media, caregivers, nutritionists, doctors, etc., coaching advice, and guidelines.

[0226] Data or other information can be stored in the data storage unit 104. It can be stored in its original format. Alternatively, it can be stored after having been subjected to some degree of processing. The data can be stored temporarily or permanently. The data or other information can be stored for a variety of reasons, including but not limited to temporarily storing while waiting for a processor or other system resources to become available; temporarily storing for combination with other data that may only be available at a later time; storing for feedback to the user in original or processed format through the feedback subsystem 106; storing for later consultation or review; storing for diet analysis and / or health guidance purposes; storing for statistical analysis across a larger population or of a larger data set; storing for performing pattern recognition methods or machine learning techniques on a larger data set.

[0227] The stored data and information or portions thereof can be accessed by the users of the system. It may also be possible that the stored data and information or portions thereof can be shared with or accessed by third parties. Third parties can include but are not limited to friends, family members, caregivers, healthcare providers, nutritionists, health coaches, other users, companies that develop and / or sell systems for diet tracking and guidance, companies that develop and / or sell components or subsystems of systems for diet tracking and guidance, and insurance companies. In some cases, it may be desirable to anonymize the data before making it available to third parties.

[0228] Figure 2 Some components disposed in an electronic system for diet tracking and guidance according to one embodiment of the present disclosure are shown. The electronic system includes a first electronic device 218, a second electronic device 219 (which can be a mobile device), and a central processing and storage unit 220. A typical system may have a calibration function to allow calibration of sensors and processors.

[0229] Figure 2 Variations of the illustrated system are also possible and are within the scope of the present disclosure. For example, in one variation, the electronic device 218 and the electronic device 219 can be combined into a single electronic device. In another variation, the functions of the electronic device 218 can be distributed across multiple devices. In some variations, part of the Figure 2 functions shown as part of the electronic device 218 can alternatively be included in the electronic device 219. In some other variations, part of the Figure 2 functions shown as part of the electronic device 219 can alternatively be included in the electronic device 218 and / or the central processing and storage unit 220. In yet another variation, there may be no central processing and storage unit 220, and all processing and storage can be performed locally on the electronic device 218 and / or the electronic device 219. Other variations are also possible.

[0230] Figure 3 is shown inFigure 2 Example of an electronic system. The electronic device 218 can be, for example, a wearable device 321 worn around the wrist, arm, or finger. The electronic device 218 can also be implemented as a wearable patch attachable to the body or embeddable in clothing. The electronic device 218 can also be a module or add-on device attachable, for example, to another wearable device, jewelry, or clothing. The electronic device 219 can be, for example, a mobile device 322 such as a mobile phone, tablet, or smartwatch. Other implementations of the electronic device 219 and the electronic device 218 are also possible. The central processing and storage unit 220 generally consists of one or more computer systems or servers and one or more storage systems. The central processing and storage unit 220 can be, for example, a remote data center 324 accessible via the Internet using an Internet connection 325. The central processing and storage unit 220 is often shared among and / or accessed by multiple users.

[0231] The wearable device 321 can communicate with the mobile device 322 via a wireless network. The wireless protocols for communicating between the wearable device 321 and the mobile device 322 via a wireless network can include, but are not limited to, Bluetooth, Bluetooth Low Energy (also known as Bluetooth Smart), Bluetooth Mesh, ZigBee, Wi-Fi, Wi-Fi Direct, NFC, Cellular, and Thread. Proprietary or wireless protocols, modified versions of standardized wireless protocols, or other standardized wireless protocols can also be used. In another implementation of the present disclosure, the wearable device 321 and the mobile device 322 can communicate via a wired network.

[0232] The mobile device 322 can communicate wirelessly with a base station or access point (“AP”) 323, which is connected to the Internet via an Internet connection 325. Via the Internet connection 325, the mobile device 322 can transfer data and information from the wearable device 321 to one or more central processing and storage units 220 residing at a remote location such as, for example, a remote data center. Via the Internet connection 325, the mobile device 322 can also transfer data and information from one or more central processing and storage units 220 residing at a remote location to the wearable device 321. Other examples are also possible. In some implementations, the central processing and storage unit 220 may not be at a remote location but may reside at or near the same location as the wearable device 321 and / or the mobile device 322. The wireless protocols for communication between the mobile device 322 and the base station or access point 323 can be the same as those between the mobile device and the wearable device. Proprietary or wireless protocols, modified versions of standardized wireless protocols, or other standardized wireless protocols can also be used.

[0233] Figure 2The electronic system can also send data, information, notifications, and / or instructions to additional devices connected to the Internet, and / or receive data, information, notifications, and / or instructions from additional devices connected to the Internet. Such devices can be, for example, the tablet computers, mobile phones, laptops, or computers of one or more caregivers, members of a doctor's office, tutors, family members, friends, people the user is connected with on social media, or other people to whom the user has given authorization to share information. Figure 4 An example of such a system is shown in Figure 4 In the example shown, the electronic device 441 is wirelessly connected to a base station or access point 440, which is connected to the Internet via an Internet connection 442. Examples of the electronic device 441 can include, but are not limited to, tablet computers, mobile phones, laptops, computers, or smart watches. Via the Internet connection 442, the electronic device 441 can receive data, instructions, notifications, or other information from one or more central processing and storage units that may reside at a local or remote location, such as, for example, a remote data center. The communication capabilities can include the Internet connection 442 or other communication channels. The electronic device 441 can also send information, instructions, or notifications to one or more computer servers or storage units 439. The central processing and storage unit 439 can forward the information, instructions, or notifications to the mobile device 436 via the Internet 438 and a base station or access point (“AP”) 437.

[0234] Other examples are possible. In some embodiments, the central processing and storage unit 439 may not be in a remote location, but may reside at or near the same location as the wearable device 435 and / or the mobile device 436. Figure 4 The electronic device 441 is shown as being wirelessly connected to a base station or access point. A wired connection between the electronic device 441 and a router connected to the Internet via the Internet connection 442 is also possible.

[0235] Figure 5 Another embodiment of the present disclosure is shown in Figure 5 In this embodiment, the wearable device 543 can directly exchange data or other information with the central processing and storage system 546 via a base station or access point 544 and the Internet without having to go through the mobile device 545. The mobile device 545 can exchange data or other information with the wearable device 543 via the central processing and storage system 546 or via a local wireless or wired network. The central processing and storage system 546 can exchange information with one or more additional electronic devices 550.

[0236] Figure 6Shows some components disposed in an electronic device 218 according to one embodiment. The electronic device 218 generally includes, in part, one or more sensor units 627, a processing unit 628, a memory 629, a clock or crystal 630, radio circuitry 634, and a power management unit (“PMU”) 631. The electronic device 218 may also include one or more camera modules 626, one or more stimulation units 633, and one or more user interfaces 632. Although not shown, other components such as capacitors, resistors, inductors may also be included in the electronic device 218. Among other things, the power management unit 631 may include one or more of the following: a battery, a charging circuit, a voltage regulator, hardware for disabling power to one or more components, a power plug.

[0237] In many embodiments, the electronic device 218 is a size - limited, power - sensitive, battery - operated device with a simple and limited user interface. In power - constrained situations, the electronic device 218 may be programmed to conserve power outside of behavioral events. For example, the processor in the electronic device 218 may be programmed to determine the start of a behavioral event (such as a feeding event), then power on additional sensors, place certain sensors in a higher - performance mode, and / or perform additional calculations until the processor determines the end of the behavioral event, at which point the processor may disconnect the additional sensors, place certain sensors back in a lower - performance mode, and omit the additional calculations.

[0238] For example, the processor may be programmed to disable all motion - detection - related circuitry except for the accelerometer. The processor may then monitor the accelerometer sensor data, and if the data indicates an actual or significant food - intake activity, such as a biting or sucking gesture, the processor may activate additional circuitry, such as a data - recording mechanism. The processor may use the accelerometer sensor data to monitor the pitch of the wearer's arm.

[0239] For example, the processor may measure the pitch of the wearer's arm until the pitch exceeds a specific threshold, which may be a threshold indicating hand or arm movement towards the wearer's mouth. Once this is detected, the processor may change state (such as by changing the memory location reserved for that state from “inactive” or “outside event” to “in - action” or “during event”) and activate additional circuitry or activate a higher - performance mode of a specific circuit or component. In another embodiment, other accelerometer sensor data characteristics may be used, such as the first integral of acceleration (velocity) or the second integral of acceleration (distance traveled), as determined by one or more accelerometer axes, or characteristics related to and / or derived from the first and / or second integrals of acceleration. Machine - learning processes may be used to detect specific motions and translate those motions into gestures.

[0240] The processor may detect the end of a food intake event by considering whether a certain amount of time has passed since the last biting or sucking motion or when there is other data (metadata about the wearer, motion detection sensor data, and / or historical data for the wearer, or a combination of these). Based on these circumstances, the processor makes a determination that a food intake event is unlikely and then changes the state of the electronic device to an inactive monitoring state, possibly a lower power consumption mode.

[0241] The lower power consumption mode may be achieved by the processor reducing the sampling rate of the accelerometer and / or gyroscope, powering off the gyroscope, reducing the update rate at which sensor data is transmitted from an electronic device (such as electronic device 218) to a supporting device (such as electronic device 219), and compressing data before transmitting the data from the sensing electronic device to the supporting electronic device.

[0242] In some embodiments of the present disclosure, Figure 5 . As one example, the processing unit, memory, radio circuitry, and PMU functionality may be combined in whole or in part in a single wireless microcontroller unit ("MCU"). Other combinations are also possible. Similarly, Figure 5 Components shown as single components in the example may be implemented as multiple components. As an example, processing functionality may be distributed across multiple processors. Similarly, data storage functionality may be distributed across multiple memory components. Other examples of distributed implementations are also possible.

[0243] In another embodiment of the present disclosure, radio circuitry may not be present, and instead data or information may be transferred to and / or from the electronic device 218 using a different interface, such as, for example, a USB interface and cable.

[0244] The stimulation unit 633 can provide feedback to the user of the electronic device. The stimulation unit 633 can include, but is not limited to, a tactile interface that applies force, vibration, or motion to the user, a speaker or headphone interface that provides sound to the user, and a display that provides visual feedback to the user.

[0245] In certain embodiments, processing and analysis of signals from sensors embedded in the electronic device 218 can detect when the electronic device has been disabled, tampered with, removed from the body, or not in use. This can be used to save power, or to send notifications to the user, a friend, or another person who may have a direct or indirect interest in being notified when the electronic device 218 is not being used properly.

[0246] Description Detection / Prediction of Start / End of Food Intake Events

[0247] In a preferred embodiment, the electronic device 218 is worn around the wrist, arm, or finger and has one or more sensors that generate data needed to detect the start and / or end of a food intake event. The electronic device 218 can also be integrated into a patch that can be attached to a person's arm or wrist. The electronic device 218 can also be a module or add-on device that can be attached to another device worn around the wrist, arm, or finger. In addition to other sensors, the sensors for detecting the start and / or end of a food intake event can include one or more of the sensors described herein.

[0248] The raw sensor output can be stored locally in the memory 629 and processed locally on the processing unit 628 to detect whether the start or end of a food intake event has occurred. Alternatively, one or more sensor outputs can be sent in raw or processed format to the electronic device 219 and / or the central processing and storage unit 220 for further processing and to detect whether the start or end of a food intake event has occurred. Regardless of where the food intake detection is processed, the sensor output in raw or processed format can be stored within the electronic device 218, within the electronic device 219, and / or within the central processing and storage unit 220.

[0249] The one or more sensors that generate data needed to detect the start and / or end of a food intake event can be internal to the electronic device 218. Alternatively, one or more of the sensors responsible for detecting the start of a food intake event can be external to the electronic device 218 but capable of relaying relevant information directly to the electronic device 218 via direct wireless or wired communication with the electronic device 218 or indirectly via another device. It is also possible that the electronic device 218 and one or more external sensors can relay information to the electronic device 219 but not directly to each other.

[0250] In the case of indirect communication via another device (such as a mobile phone or other portable or fixed device), such a third device can receive data or information from one or more external sensor units, optionally process such data or information, and forward the raw or processed data or information to the electronic device 218. Communication to and from the electronic device 218 can be wired or wireless or a combination of both.

[0251] Examples of sensors that may be external to the electronic device 218 can be one or more sensors embedded in a necklace or pendant worn around the neck, one or more sensors embedded in patches attached to different locations on the body, one or more sensors embedded in a complementary second wearable device worn around the other arm or wrist or on the fingers of the other hand, or one or more sensors integrated in the teeth. In some embodiments, the electronic device is worn on one hand or arm, but detects the movement of the other hand or arm. In some embodiments, the electronic device is worn on each hand.

[0252] The information obtained from the non-real-time analysis and learning subsystem 105 (optionally combined with information from one or more sensors 627) can also be used to predict the likely, impending, or actual start / end of a food intake event or to facilitate the detection of the likely, impending, or actual start / end of a food intake event.

[0253] It is generally desirable for the detection and / or prediction of the start and / or end of a food intake event to occur autonomously without user intervention. For example, if the actual, likely, or impending start of a food intake event is autonomously predicted or detected, this information can be used as a trigger to activate or power on specific components or circuits that are only needed during the food intake event. This can help save power and extend the battery life of the electronic device 218. The prediction or detection of the actual, likely, or impending start of a food intake event can also be used to issue a prompt or reminder to the user. A prompt can be sent to the user, for example, to remind him / her to take further actions, including but not limited to logging the food intake event or taking a photo of the food. One or more prompts may be dispersed during the duration of the food intake event upon detection of the start of the food intake event to remind the user that a food intake event is occurring and to improve present awareness and / or encourage mindful eating. For example, one or more stimulation units 633 can be used to send the prompt or reminder through discrete tactile feedback. Other methods using one or more user interfaces 632 (such as one or more LEDs, display messages, or audio signals) are also possible. Alternatively, a mobile device can be used to communicate a prompt, reminder, or other information (such as portion size recommendations or alternative suggestions for eating) to the user.

[0254] If the actual, likely, or impending end of a food intake event is autonomously predicted or detected, this information can be used as a trigger to power off or at least place one or more circuits or components of the electronic device that are only needed during the food intake event in a lower power consumption mode. This can help save power and extend the battery life of the electronic device. The detection of the actual, likely, or impending end of a food intake event can also be used to modify or pause the feedback provided to the user by one or more of the stimulation units 633, the user interface 632, and / or the mobile device.

[0255] In some embodiments of the present disclosure, the detection or prediction of the actual, possible, or impending start and / or end of a food intake event may not be fully autonomous. For example, a user may be required to make a specific arm, wrist, hand, or finger gesture to signal to the electronic device the actual, possible, or impending start and / or end of a food intake event. The arm, wrist, hand, or finger gesture is then detected by one or more sensors within the electronic device. It is generally desirable that the one or more arm, wrist, hand, or finger gestures required to indicate the start and / or end of a food intake event can be performed in a refined and discrete manner. Other methods may also be used. For example, the user may be required to press a button on the electronic device to indicate the start and / or end of a food intake event. Voice activation commands that utilize a microphone built into the electronic device may also be used. Other methods are possible.

[0256] Description of Tracking of Eating Behaviors and Patterns

[0257] In a particular embodiment, the electronic device is worn around the wrist, arm, or finger and has one or more sensors that generate data for measuring and analyzing eating behaviors, patterns, and habits. The sensors for measuring and analyzing certain eating behaviors and patterns may include one or more of the sensors described herein.

[0258] Relevant metrics that can be used to quantify and track eating behaviors and eating patterns may include, but are not limited to, the time between successive bites or sucks, the distance between the plate and the user's mouth, the speed of arm movement towards and / or away from the user's mouth, and the number of bites or sucks during a single food intake event derived from the total count of arm movements corresponding to bites or sucks, specific chewing behaviors and characteristics, the time between taking a bite and swallowing, and the amount of chewing before swallowing.

[0259] Figure 6 An example of the components of such an electronic device is shown. As shown, the raw sensor output may be stored locally in the memory 629 and processed locally on the processing unit 628. Alternatively, one or more sensor outputs may be sent in raw or processed format to the electronic device and / or the processing unit 628 for further processing and analysis. Regardless of where the eating behaviors and patterns are processed and analyzed, the sensor output in raw or processed format may be stored within the electronic device, within an auxiliary electronic device (such as a mobile phone), and / or within the processing unit 628.

[0260] In some embodiments, the generation, collection, and / or processing of data facilitating the measurement and analysis of eating behaviors, patterns, and habits can be continuous, periodic, or otherwise independent of the start and / or end of a food intake event. Alternatively, the generation, collection, and / or processing of data facilitating the measurement and analysis of eating behaviors and patterns can occur only during a food intake event or otherwise be associated with a particular food intake event. It is also possible that some sensor data is generated, collected, and / or processed continuously, periodically, or otherwise independent of the start and / or end of a food intake event, while other sensor data is acquired during a food intake event or otherwise associated with a food intake event.

[0261] One or more sensors that generate data for measuring and analyzing eating behaviors and eating patterns can be internal to the electronic device. Alternatively, one or more of the sensors that generate data for measuring and analyzing eating behaviors and eating patterns can be external to the electronic device but capable of relaying relevant information directly to the electronic device through direct wireless or wired communication with the electronic device or indirectly through another device.

[0262] In the case of indirect communication through another device (such as a mobile phone or other portable or fixed device), such a third device is capable of receiving data or information from an external sensor unit, optionally processing such data or information, and forwarding the raw or processed data or information to the tracking device. Communication to and from the electronic device can be wired, wireless, or a combination of both.

[0263] Examples of sensors that can be external to the electronic device can be one or more sensors embedded in a necklace or pendant worn around the neck, one or more sensors embedded in patches attached to different locations on the body, one or more sensors embedded in a complementary second wearable device worn around the other arm or wrist or on a finger of the other hand, or one or more sensors integrated in a tooth.

[0264] Description of Use of Camera Module and Image Capture

[0265] Although the use of cameras to capture images of food has been proposed in the prior art, they typically rely on the user taking a photo with his or her mobile phone or tablet. Unfortunately, image capture using a mobile phone or tablet presents significant usage conflicts, may not be socially acceptable in certain dining situations, or can interfere with the authenticity of the dining experience. It is often undesirable or inappropriate for the user to need to take out his or her mobile phone, unlock the screen, open a mobile application, and take a photo using the camera built into the mobile phone.

[0266] If user intervention is required, it is generally desirable to perform the user intervention in a fine and discrete manner and with as few conflicts as possible. To minimize usage conflicts, it is often desirable to initiate image capture directly from an electronic device.

[0267] Although the examples provided herein use image capture of food and meal scenarios as examples, it should be understood after reading this disclosure that the methods and apparatuses described herein can be applied to image capture of objects and scenarios other than food and meal scenarios. For example, a camera without a viewfinder can have applications outside the field of food event capture.

[0268] In some embodiments, the electronic device is worn around the wrist, arm, or finger and includes one or more camera modules 626. The one or more camera modules 626 can be used for the capture of still images according to one embodiment of the present disclosure and for the capture of one or more video streams according to another embodiment of the present disclosure. In yet another embodiment of the present disclosure, a combination of still and streaming images is also possible.

[0269] One or more camera modules can also be included in devices worn at different locations around the body, such as a necklace or pendant worn around the neck, or a device attached to or integrated with the user's clothing, where the camera or camera module is preferably aimed forward so that it can be in line of sight with the food being consumed.

[0270] In some embodiments, activation of the camera module and / or image capture by the camera module may require some degree of user intervention. Among other things, user intervention can include pressing a button; issuing a voice command into a microphone built into the electronic device or mobile device; making a selection using a display integrated in the electronic device or mobile device; making a specific arm, wrist, hand, or finger gesture; guiding the camera so that the object of interest is within the field of view of the camera; removing an obstacle that may be in the line of sight between the camera and the object of interest; and / or adjusting the position of the object of interest so that it is within the field of view of the camera. Other user intervention methods or combinations of multiple user intervention methods are also possible.

[0271] In one embodiment of the present disclosure, a camera module is built into an electronic device (such as a wearable device) that may not have a viewfinder or may not have a display that can give feedback to the user about an area located within the camera's field of view. In such a case, the electronic device may include a light source that projects a visible light pattern onto a surface or onto an object to indicate to the user an area located within the camera's field of view. One or more light-emitting diodes (LEDs) can be used as the light source. Other light sources (including but not limited to lasers, halogen, or incandescent light sources) are also possible. Among other things, the user can use the visible light pattern to adjust the position of the camera, adjust the position of the object of interest, and / or remove any object that obscures the line of sight between the object of interest and the camera.

[0272] The light source can also be used to convey other information to the user. As an example, the electronic device can use inputs from one or more proximity sensors, process these inputs to determine whether the camera is within an appropriate distance range from the object of interest, and use one or more light sources to convey to the user that the camera is within the appropriate distance range, that the user needs to increase the distance between the camera and the object of interest, or that the user needs to decrease the distance between the camera and the object of interest.

[0273] The light source can also be used in combination with an ambient light sensor to convey to the user whether the ambient light is insufficient or too strong for proper quality image capture.

[0274] The light source can also be used to convey information including but not limited to a low battery condition or a functional defect.

[0275] The light source can also be used to convey dietary guidance information. As an example, among other things, the light source may indicate whether too little or too much time has passed since the previous food intake event, or can convey to the user how well he / she is doing compared to a specific dietary goal.

[0276] The signaling mechanism for transmitting a specific message using one or more light sources can include but is not limited to one or more of the following: a specific light intensity or light intensity pattern, a specific light color or light color pattern, a specific spatial or temporal light pattern. Multiple mechanisms can also be combined to convey a single specific message.

[0277] In another embodiment of the present disclosure, the camera module may be built into an electronic device (such as a wearable device) that does not have a viewfinder or a display that can give feedback to the user about the area located within the camera's field of view. As an alternative or supplement to using a light source, one or more images captured by the camera module (possibly combined with inputs from other sensors embedded in the electronic device) may be sent to a processing unit within the electronic device, a processing unit within a mobile device, and / or processing unit 628 for analysis and to determine whether an object of interest is within the appropriate field of view and / or appropriate focus range of the camera. The analysis result may be communicated to the user using one of the feedback mechanisms available in the electronic device, including but not limited to haptic feedback, visual feedback using one or more LEDs or a display, and / or audio feedback.

[0278] In some other embodiments of the present disclosure, the electronic device may capture one or more images without any user intervention. The electronic device may capture static or streaming images continuously, periodically, or otherwise independently of any food intake event. Alternatively, the electronic device may activate one or more of the camera modules in its camera module only approximately at or during the time of a food intake event. As an example, the electronic device may activate one or more of the camera modules in its camera module and capture one or more images only after a food intake event has been detected as starting and before a food intake event has been detected as ending. It may use one or more of the camera modules in its camera module to capture more images of an entire food item or dish or a portion of one or more food items or dishes.

[0279] In some embodiments, one camera may be used to capture one or more images of food items located on a plate, table, or other stationary surface, and a second camera may be used to capture one or more images of food items held by the user (such as finger foods or beverages). In cases where user intervention is not desired and the position, field of view area, or focus range of a single camera is not suitable for capturing all possible meal scenarios, it may be desirable to use more than one camera.

[0280] In an exemplary embodiment, the position, orientation, and viewing angle of the camera are such that image or video capture can occur without user intervention. In such an embodiment, the wearable device can use a variety of techniques to determine the appropriate timing for image or video stream capture such that it can capture the food being consumed or a portion of the food. It can also select to capture multiple image or video streams for this purpose. Techniques for determining the appropriate timing can include, but are not limited to, the sensing of proximity, the sensing of acceleration or motion (or the absence thereof), and location information. Such sensor information can be used alone or in combination with pattern recognition or data analysis techniques (or a combination of both) to predict the optimal timing for image or video capture. Techniques can include, but are not limited to, the training of machine learning-based models.

[0281] The captured static and / or streaming images typically require some degree of processing. Processing can include, but is not limited to, compression, deletion, resizing, filtering, image editing, and computer vision techniques to identify objects (such as a specific food or dish) or features (such as portion size). Processing units that can be used to process static or streaming images from one or more camera modules, regardless of whether the one or more camera modules are internal to the electronic device, include, but are not limited to, a processing unit within the electronic device, a processing unit within a mobile device, and / or a processing unit that can reside at the same location as where the electronic device is used or alternatively can reside at a remote location (e.g., in a cloud server) (in which case it can be accessed via the Internet). Image processing can also be distributed among a combination of the above processing units.

[0282] Examples of local processing can include, but are not limited to: selecting one or more static images from multiple images or one or more video streams; compressing the image or video stream; applying computer vision algorithms to one or more images or video streams.

[0283] Local processing can include compression. In terms of compression, the compressed image can be transmitted as part of a time-critical transaction, while its uncompressed version can be saved for later transmission.

[0284] One or more static or streaming images can be analyzed and / or compared for one or more purposes, including, but not limited to, the detection of the start and / or end of a food intake event, the identification of food items, the identification of food ingredients, the identification or derivation of nutritional information, the estimation of portion size, and the inference of certain eating behaviors and eating patterns.

[0285] As an example, computer vision techniques (optionally in combination with other image manipulation techniques) can be used to identify food categories, specific food items, and / or estimate portion sizes. Alternatively, Mechanical Turk processes or other crowdsourcing methods can be used to manually analyze the images. Once the food category and / or specific food item has been identified, this information can be used to retrieve nutritional information from one or more food / nutrition databases.

[0286] As another example, information about the user's eating or drinking speed can be inferred by analyzing and comparing multiple images captured at different times during a food intake event. As yet another example, images (optionally in combination with other sensor information) can be used to distinguish between a sit-down meal and finger foods or snacks. As yet another example, the analysis of one image taken at the start of a food intake event and another image taken at the end of the food intake event can provide information about the amount of food actually consumed.

[0287] Description of User Feedback

[0288] In a preferred embodiment of the present disclosure, the electronic device 218 is worn around the wrist, arm, or finger and has one or more stimulation units and / or user interfaces that allow feedback to be provided to the user or wearer of the electronic device. In different embodiments of the present disclosure, the electronic device 218 can be implemented as a wearable patch that can be attached to the body or embedded in clothing.

[0289] The feedback generally includes food or food intake-related feedback. Feedback methods can include, but are not limited to, tactile feedback, visual feedback using an LED or display, or audio feedback. In one such embodiment, the electronic device 218 can have a tactile interface that vibrates once or more times at the detection of the start and / or end of a food intake event. In another embodiment, the electronic device 218 can have a tactile interface that vibrates once or more times when the tracking and processing subsystem identifies that the wearer of the device is consuming food and exhibiting eating behavior that exceeds certain programmed thresholds, such as eating too fast, too slow, or too much. Alternatively, the tactile interface can vibrate once or more times during a food intake event, independent of any specific eating behavior, for example to alert the wearer to the fact that a food intake event is occurring and / or to improve present moment awareness and encourage mindful eating. Other feedback methods are possible, and different metrics or criteria can be used to trigger the activation of such feedback methods.

[0290] In various embodiments of the present disclosure, feedback is provided to a user by a device separate from electronic device 218. One or more stimulation units and / or user interfaces required to provide feedback to the user may be external to electronic device 218. As an example, one or more stimulation units and / or user interfaces may be within electronic device 219, and one or more of the stimulation units and / or user interfaces within electronic device 219 may be used to provide feedback, as an alternative or supplement to the feedback provided by electronic device 218. Examples may include but are not limited to displaying a message on a display of electronic device 219, or emitting an audible alert by an audio subsystem embedded within electronic device 219.

[0291] Alternatively, feedback may be provided by a device separate from both electronic device 218 and electronic device 219, but capable of receiving data from at least one of these devices, at least directly or indirectly.

[0292] As a supplement or alternative to feedback provided at or around the time of a food intake event, Figure 2 or Figure 3 the system may also provide feedback that may span multiple food intake events or may not be associated with a particular food intake event or group of food intake events. Examples of such feedback may include but are not limited to food composition and nutritional information, a summary of historical data, an overview of one or more tracked parameters over an extended time period, the progress of one or more tracked parameters, personalized diet guidance and recommendations, benchmarking of one or more tracked parameters compared to peers or other users with similar profiles.

[0293] Detailed Description of Specific Embodiments

[0294] In a particular embodiment of the present disclosure, electronic device 218 is a wearable device in the form factor of a bracelet or wristband, worn around the wrist or arm of the user's dominant hand. Electronic device 219 is a mobile phone, and central processing and storage unit 220 is one or more computer servers and data storage located at a remote location.

[0295] Figure 7 A possible implementation of a wearable bracelet or wristband according to aspects of the present invention is shown in. Wearable device 770 may optionally be implemented using a modular design, where individual modules include one or more subsets of these components and all functions. The user may choose to add specific modules based on their personal preferences and needs.

[0296] Wearable device 770 may include a processor, program code memory, and program code (software) stored therein and / or within electronic device 219 to optionally allow the user to customize a subset of the functions of wearable device 770.

[0297] The wearable device 770 relies on the battery 769 and the power management unit (“PMU”) 760 to deliver power at an appropriate supply voltage level to all electronic circuits and components. The power management unit 760 may also include a battery recharging circuit. The power management unit 760 may also include hardware, such as switches, that allows power to be cut off to specific electronic device circuits and components when not in use.

[0298] When there are no ongoing behavioral events, most of the circuits and components in the wearable device 770 are turned off to save power. Only the circuits and components that detect or contribute to predicting the start of a behavioral event may remain enabled. For example, if no movement is detected, all sensor circuits except the accelerometer may be turned off and the accelerometer may be placed in a low-power motion wake-up mode or another lower-power mode that consumes less power than the high-performance activity mode. The processing unit may also be placed in a low-power mode to save power. When movement or a specific motion pattern is detected, the accelerometer and / or the processing unit may switch to a higher-power mode and additional sensors, such as, for example, a gyroscope and / or a proximity sensor, may also be enabled. When the potential start of an event is detected, memory variables for storing event-specific parameters, such as pose type, pose duration, etc., may be initialized.

[0299] In another example, when movement is detected, the accelerometer switches to a higher-power mode, but the other sensors remain off until data from the accelerometer indicates that the start of a behavioral event may have occurred. At this point in time, additional sensors, such as a gyroscope and a proximity sensor, may be enabled.

[0300] In another example, when there are no ongoing behavioral events, both the accelerometer and the gyroscope are enabled, but at least one of the accelerometer or the gyroscope is placed in a lower-power mode compared to its normal power consumption mode. For example, the sampling rate may be reduced to save power. Similarly, the circuits required to transfer data from the electronic device 218 to the electronic device 219 may be placed in a lower-power mode. For example, the radio circuit 764 may be completely disabled. Similarly, the circuits required to transfer data from the electronic device 218 to the electronic device 219 may be placed in a lower-power mode. For example, it may be completely disabled until it has been determined that the possible or likely start of a behavioral event has occurred. Alternatively, it may remain enabled but in a low-power state to maintain the connection between the electronic device 218 and the electronic device 219 but not transfer sensor data.

[0301] In yet another example, if, based on certain metadata, it is determined that the occurrence of a particular behavioral event, such as a food intake event, is unlikely, then all motion detection-related circuitry, including the accelerometer, may be turned off. This may be desirable, for example, to further conserve power. Among other things, the metadata used to make this determination may include one or more of the following: the time of day, location, ambient light level, proximity sensing, and detection of removal of the wearable device 770 from the wrist or hand, detection that the wearable device 770 is being charged. The metadata may be generated and collected within the wearable device 770. Alternatively, the metadata may be collected within the mobile phone or within another device that is external to the wearable device 770 and external to the mobile phone and that can directly or indirectly exchange information with the mobile phone and / or the wearable device 770. It is also possible that some of the metadata is generated and collected within the wearable device 770 while other metadata is generated and collected within a device external to the wearable device 770. In the case where some or all of the metadata is generated and collected external to the wearable device 770, the wearable device 770 may periodically or from time to time power up its radio circuitry 764 to retrieve metadata-related information from the mobile phone or other external device.

[0302] In yet another embodiment of the present invention, if certain metadata indicates that a particular behavioral event (such as, for example, a food intake event) is likely to occur, then some or all of the sensors may be turned on or placed in a higher power consumption mode. Among other things, the metadata used to make this determination may include one or more of the following: the time of day, location, ambient light level, and proximity sensing. Some or all of the metadata may be collected within the mobile phone or within another device that is external to the wearable device 770 and external to the mobile phone and that can directly or indirectly exchange information with the mobile phone and / or the wearable device 770. In the case where some or all of the metadata is generated and collected external to the wearable device 770, the wearable device 770 may periodically or from time to time power up its radio circuitry 764 to retrieve metadata-related information from the mobile phone or other external device.

[0303] The detection of the start of a behavioral event (such as, for example, a food intake event) may be signaled to the user via one of the available user interfaces on the wearable device 770 or on the mobile phone to which the wearable device 770 is connected. As an example, the tactile interface 761 within the wearable device 770 may be used for this purpose. Other signaling methods are also possible.

[0304] The detection of the start of a behavioral event (such as, for example, a food intake event) may trigger some or all sensors to be placed or remain in a high power mode or an active mode to track certain aspects of the user's eating behavior during a portion or all of the food intake event. One or more sensors may be powered down or placed in a lower power mode at or after the actual or perceived end of the detected behavioral event (the determined end of the behavioral event). Alternatively, it is also possible to power down or place one or more sensors in a lower power mode after a fixed or programmable time period.

[0305] Sensor data for tracking certain aspects of the user's behavior (such as, for example, the user's eating behavior) may be stored locally in the memory 766 of the wearable device 770 and processed locally using the processing unit 767 within the wearable device 770. It is also possible to transmit the sensor data using the radio circuitry 764 to a mobile phone or a remote computing server for further processing and analysis. It is also possible to perform some processing and analysis locally within the wearable device 770 and other processing and analysis on the mobile phone or on the remote computing server.

[0306] The detection of the start of a behavioral event (such as, for example, the start of a food intake event) may trigger the powering on and / or activation of additional sensors and circuitry (such as, for example, the camera module 751). The powering on and / or activation of the additional sensors and circuitry may occur at the same time as or at some time after the detection of the start of the food intake event. Specific sensors and circuitry may be turned on only at specific times during the food intake event when needed and otherwise may be turned off to conserve power.

[0307] It is also possible to power on or activate the camera module 751 only upon explicit user intervention (such as, for example, pressing and holding the button 759). Releasing the button 759 may disconnect the camera module 751 again to conserve power.

[0308] When the camera module 751 is powered on, the projection light source 752 may also be enabled to provide visual feedback to the user regarding the area located within the camera's field of view. Alternatively, the projection light source 752 may be activated only at some time after the camera module has been activated. In some cases, additional conditions may need to be met before the projection light source 752 is activated. Among other things, such conditions may include the determination of the direction in which the projection light source 752 may be aimed at the object of interest or the determination that the wearable device is not moving excessively.

[0309] In a particular implementation, partially pressing button 759 on wearable device 770 can power on camera module 751 and projection light source 752. Further pressing button 759 can trigger camera module 751 to capture one or more still images or one or more streaming images. In some cases, further pressing button 759 can trigger deactivation, modified brightness, modified color, or modified pattern of projection light source 752 before or coinciding with image capture. Releasing button 759 can trigger deactivation and / or power-off of projection light source 752 and / or camera module 751.

[0310] The image can be tagged with additional information or metadata, such as for example camera focus information, proximity information from proximity sensor 756, ambient light level information from ambient light sensor 757, timing information, etc. Such additional information or metadata can be used during the processing and analysis of food intake data.

[0311] Various light patterns are possible and can be formed in various ways. For example, it can include mirrors or such mechanisms that reflect projection light source 752 such that projection light source 752 produces one or more light rays that trace out a center or define a particular area, such as a cross, L-shape, circle, rectangle, multiple points or lines that frame the field of view or otherwise give the user visual feedback about the field of view.

[0312] One or more light-emitting diodes (LEDs) can be used as projection light source 752. Among other things, the user can use this visible light pattern to adjust the position of the camera, adjust the position of the object of interest, and / or remove any object that obscures the line of sight between the object of interest and the camera.

[0313] Projection light source 752 can also be used to convey other information to the user. As an example, the electronic device can use inputs from one or more proximity sensors, process these inputs to determine whether the camera is within an appropriate distance range from the object of interest, and use one or more light sources to convey to the user that the camera is within the appropriate distance range, the user needs to increase the distance between the camera and the object of interest, or the user needs to decrease the distance between the camera and the object of interest.

[0314] The light source can also be used in combination with an ambient light sensor to convey to the user whether the ambient light is insufficient or too strong for proper quality image capture.

[0315] The light source can also be used to convey information including but not limited to low battery conditions or functional defects.

[0316] The light source can also be used to convey dietary guidance information. As an example, among other things, the light source may indicate whether too little or too much time has elapsed since the previous food intake event, or may convey to the user how well he / she is doing compared to a particular dietary goal.

[0317] The signaling mechanism that uses one or more projected light sources to convey a specific message can include, but is not limited to, one or more of the following: a specific light intensity or light intensity pattern, a specific light color or light color pattern, a specific spatial or temporal light pattern. Multiple mechanisms can also be combined to convey a specific message.

[0318] The user can use the microphone 758 to add a specific or customized label or message to the food intake event and / or the image. The audio clip can be processed by a speech recognition engine.

[0319] In some embodiments, in addition to tracking at least one parameter directly related to food intake and / or eating behavior, an accelerometer (possibly in combination with other sensors) can also be used to track one or more parameters not directly related to food intake. Among other things, such parameters can include activity, sleep, or stress.

[0320] Specific Embodiment without Built-in Camera

[0321] In different embodiments, the electronic device 218 does not need to have any built-in image capture capabilities. The electronic device 218 can be a wearable device, such as a bracelet or wristband worn around the arm or wrist, or a ring worn around a finger. The electronic device 219 can be a mobile phone, and the central processing and storage unit 220 can be one or more computing servers and data storage located at a remote location.

[0322] In such embodiments, the food intake tracking and feedback system may not use images to extract information about food intake and / or eating behavior. Alternatively, the food intake tracking and feedback system can utilize the image capture capabilities available within other devices, such as, for example, the electronic device 219 or an electronic device external to the electronic device 218.

[0323] Upon detecting or predicting the start of a food intake event, the electronic device 218 can send a signal to the electronic device 219 or to an electronic device that otherwise houses image capture capabilities to indicate the actual, possible, or imminent start of a food intake event. This can trigger the electronic device 219 or an electronic device that otherwise houses image capture capabilities to enter a mode that will allow the user to capture an image with at least one fewer user step than its default mode or standby mode.

[0324] As an example, if the image capture capability is housed within the electronic device 219 and the electronic device 219 is a mobile phone, tablet, or similar mobile device, the electronic device 218 may send one or more signals to software installed on the electronic device 219 to indicate the actual, possible, or impending start of a food intake event. Upon receipt of one or more such signals, among other things, the software on the electronic device 219 may take one or more of the following actions: unlock the screen of the electronic device 219; open a mobile application related to the food intake and feedback subsystem; activate the camera mode of the electronic device 219; push a notification to the display of the electronic device 219 to assist the user with image capture; send a message to the electronic device 218 to alert, remind, and / or assist the user with image capture.

[0325] After an image capture is performed by the electronic device 219 or by an electronic device that otherwise houses the image capture capability, the electronic device 219 or the electronic device that otherwise houses the image capture capability may provide visual feedback to the user. Examples of visual feedback may include a pattern, shape, or overlay showing a recommended portion size, or a pattern, shape, or overlay shadow in one or more colors and / or having one or more brightness levels to indicate how healthy the food is. Other examples are possible.

[0326] Integration with Insulin Therapy System

[0327] One or more components of the food intake tracking and feedback system proposed in this disclosure may interface or integrate with an insulin therapy system. In a particular example, upon detection of the start of a food intake event, feedback may be sent to the wearer to remind him or her to perform a glucose level measurement and / or administer an appropriate dose of insulin. One or more additional reminders may be sent during the course of the food intake event.

[0328] Patients who have been diagnosed with type I or type II diabetes can also use the food intake tracking and feedback system or components thereof described in the present disclosure. For example, the components described in the present disclosure can be used to automatically detect when a person starts eating or drinking. Detection of the start of a food intake event can be used to send a message to the wearer when the food intake event starts or is about to start to remind him or her to perform a glucose level measurement and / or administer an appropriate dose of insulin. The messaging can be automatic and independent. Alternatively, the system can be integrated with a health system or a healthcare maintenance and reminder system. The health system or healthcare maintenance and reminder system can send a message to the wearer when it receives notice that the start of a food intake event has been detected. The health system or healthcare maintenance and reminder system can receive additional information about the food intake event, such as the number of bites or sips, the estimated amount of food consumed, the duration of the meal, the speed of eating, etc. The health system or healthcare maintenance and reminder system can send additional messages to the wearer based on the additional information during or after the food intake event.

[0329] In another example, specific information about the composition of the food intake can be used as an input (possibly in combination with one or more other inputs) to calculate the appropriate dose of insulin to be administered. Among other things, information about the composition of the food intake can include one or more of the following: the amount of carbohydrates, the amount of sugar, the amount of fat, the portion size, and the molecular food category (such as solid or liquid). Information related to food intake and eating patterns and behaviors, including real-time, near real-time, and historical information, can be included as inputs or parameters for calculating the insulin dose.

[0330] Among other things, other inputs that can be used as inputs or parameters for an algorithm for calculating the insulin dose can include one or more of the following: age, gender, weight, historical and real-time blood glucose levels, historical and real-time activity, sleep and stress levels, vital sign information, or other information indicating the physical or emotional health of the individual.

[0331] The calculation of the insulin dose can be performed completely manually by the user, completely autonomously by a closed-loop insulin therapy system, or semi-autonomously, where some or all of the calculations are performed by the insulin therapy system, but some user intervention is still required. Among other things, user intervention can include activation of the insulin therapy calculation unit; confirmation of the dose; intervention or suspension of insulin delivery in the event that the user detects or identifies an abnormality.

[0332] In a particular embodiment, the food intake tracking and feedback system described herein can send one or more notifications to one or more caregivers of the user when the actual, possible, or impending start of a food intake event is detected, as a supplement or alternative to sending the notification to the user.

[0333] At the start of a food intake event, optionally upon notification or signal from the system or from a caregiver, the user may send one or more images of the food or meal to one or more caregivers. The caregiver may analyze these images and send information back to the user regarding the composition of the food. Among other things, the information may include an estimate of certain macronutrient components (such as, for example, carbohydrates, sugars or fats), an estimate of calorific value, and advice regarding portion size.

[0334] In the case where the user is receiving insulin treatment, additional information (such as, for example, blood glucose level readings) may also be sent to the caregiver, and the information that the caregiver provides back to the user may also include advice regarding the insulin dose to be administered and the timing at which one or more such insulin doses should be administered. In certain specific implementations, the caregiver may not be a human but an artificial intelligence system.

[0335] Posture Recognition

[0336] In the various systems described herein, accurate determination of pose information may be important. For example, it would be useful to distinguish between poses associated with speaking and poses indicating the start of an eating event cycle. Some poses may be easy to detect, such as the pose of swinging the arms while walking, and thus measure walking speed and number of steps, but other poses may be more difficult, such as determining when the user takes a bite of food, takes a sip of a drink, bites their nails, etc. The latter can be used to evaluate precursor behaviors. For example, assume that a health maintenance and reminder system detects such a pattern, first a nail-biting pose, followed by a pose associated with stress eating five to ten minutes later. The user may program their health maintenance and reminder system to signal them two minutes after nail-biting so that the user is aware and more in tune with their otherwise unnoticed behavior. For this purpose, pose detection should be accurate and reliable. This can be a problem in cases where there is no simple correlation between the movement of an accelerometer in, for example, a wearable bracelet and stress eating. Part of the reason for this problem is that the poses that the health maintenance and reminder system is interested in are not easily derivable from simple sensor readings.

[0337] Being able to determine whether a user is taking a bite of food or sipping a drink and being able to distinguish between biting and sipping can be useful for providing appropriate weight management guidance. For example, a weight management monitoring and reminder system can monitor a user's food intake events from postures. A weight management monitoring and reminder system can also monitor a user's fluid intake events from postures. Studies have shown that drinking sufficient water at the start or near the start of a meal and further adequately throughout the meal can reduce food consumption and contribute to weight loss. A user, the user's counselor, the user's healthcare provider, or the provider of the weight management monitoring and reminder system can program the system to send a reminder when the user starts eating without drinking or when it detects that the user has not adequately drunk throughout the meal. The system can also monitor the user's fluid intake throughout the day and be programmed to send a reminder when the fluid intake level does not meet a preconfigured level at a specific time of day. For this, posture detection should be reliable and accurate. This can be a problem in cases where it is necessary to distinguish postures that have a lot of similarity, such as, for example, distinguishing an eating posture from a drinking posture.

[0338] In various embodiments described herein, a processing system (including program code, logic, hardware, and / or software, etc.) receives sensor data generated by an electronic device or other element based on a user's activity. The sensor data may represent a snapshot of readings at a specific time or may represent readings over a time span. The sensors may be accelerometers, gyroscopes, magnetometers, thermometers, photometers, etc. Based on the sensor data, the processing system uses stored rules and internal data (such as information about what sensors are used and past usage history) to identify behavior events, where a behavior event is a sequence of postures and these postures are determined according to a logical arrangement of sensor data with a start time, sensor readings, and an end time, as well as external data. A behavior event may be a high-level event, such as eating a meal, etc.

[0339] The determination of the boundaries of a posture (i.e., its start and end times) can be determined using the methods described herein. The data of the start time, sensor readings, and end time are collectively referred to herein as a posture envelope. The posture envelope may also include an anchor time, which is a data element that defines a single time associated with the posture envelope. The anchor time may be the midpoint between the start time and the end time, but may be based on some criteria of the sensor data based on the posture envelope. The anchor time may be outside the time span from the start time to the end time. Multiple anchor times for each posture are also possible.

[0340] A machine classifier, which is part of a processing system (but can also be a separate computer system and may be separated by some network), determines, based on a pose envelope, what class of poses might have generated the sensor data of that pose envelope and the details of that pose. For example, the machine classifier might output that the sensor data indicates or suggests that a person wearing a bracelet with sensors is walking, eating, or pointing at something.

[0341] When using such a system, if the pose can be accurately discerned, a health maintenance and reminder system (or other systems that use pose information) can accurately respond to the pose being made. In the following example, there is a set of sensors coupled to a machine classification system or at least an input from a set of sensors, and the machine classification system outputs pose data from the sensor readings considering rules and the stored data derived from training the machine classification system. A training subsystem might be used to train the machine classification system, thus forming the stored data derived from training. Each of these components might use different hardware or share hardware and can be located locally and / or remotely. Generally, when a pose is detected, the system can analyze the pose, determine what activities are likely actual, possible, or impending, and provide user feedback regarding those activities. For example, a vibration as a feedback signal to indicate that the user has previously set the system to alert the user when the user has been drinking for more than 45 minutes in a semi - continuous period or to alert the user that the user has reached the goal of the amount of walking to be completed in a session.

[0342] Figure 8 is an illustrative example of a typical machine classification system. Figure 8 The machine classification system of includes a training subsystem 801 and a detector subsystem 802. In some embodiments of the present disclosure, the machine classification system can include additional subsystems or Figure 8 a modified version of the subsystems shown. The training subsystem 801 uses training data input 803 and labels 804 to train a trained classifier model 805. The labels 804 might have been manually assigned by a human or might have been generated automatically or semi - automatically. Then, the trained classifier model 805 is used in the detector subsystem 802 to generate a classification output 806 corresponding to new unlabeled data input.

[0343] The stored sensor data includes a time component. The raw sensor readings are tagged with their reading times. Raw sensor data can be extracted from accelerometers, gyroscopes, magnetometers, thermometers, barometers, humidity sensors, ECG sensors, etc., and the temporal data can come from other sources. Other examples of time sources might be audio, speech, or video recordings.

[0344] Figure 9 and Figure 10Illustrative examples of a training subsystem 801 and a detector subsystem 802 in accordance with at least one embodiment of the present disclosure are shown. Temporal training data 907 and labels 912 are fed into Figure 8 a classifier training subsystem of

[0345] As explained in the examples herein, raw sensor data is processed to identify macro signature events. A macro signature event may define a posture including sensor data over a period of time. A detector subsystem or other system may create a posture envelope data set that includes a start time, an end time, one or more anchor times, metadata, and sensor data that occur within the temporal envelope of the posture from the start time to the end time.

[0346] For example, in the case of a posture recognition problem, a posture envelope detector may identify a specific time period in the raw temporal data that indicates a possible posture. The posture envelope detector also generates a temporal envelope that specifies the relevant time or time periods within the posture. Among other things, the information included in the temporal envelope may include the start time of the posture, the end time of the posture, one or more times that specify relevant posture sub - segments within the posture, one or more times that specify relevant posture anchor times (time points) within the posture, and possibly other metadata and raw sensor data from within the temporal envelope of the posture.

[0347] As an example of other metadata, assume that historical patterns indicate that the wearer will have a feeding session after a phone call from a specific phone number. The electronic device may signal this condition to the wearer to provide awareness of the pattern, which may help change behavior (if the wearer so decides).

[0348] Temporal training data 907 is fed into a posture envelope detector 908. The posture envelope detector 908 processes the temporal training data 907 and identifies possible instances of postures 909 and corresponding posture temporal envelopes from the temporal training data 907. The temporal training data 907 may include motion sensor data, and the posture envelope detector 908 may process the motion sensor data and identify a posture 909 based on changes in the pitch angle. In one embodiment, the posture envelope detector 908 may detect the start of a posture based on a detection that the pitch angle rises above a specified value, and detect the end of the event based on the pitch angle falling below the specified value. Other start and end criteria are possible. Examples of anchor points that may be detected by the posture envelope detector 908 and specified by the posture temporal envelope would be the time within a posture segment when the pitch angle reaches a maximum value. Other examples of anchor points are possible.

[0349] The pose envelope detector 908 can add additional criteria to further qualify the segment as a valid pose. For example, a threshold can be specified for the peak pitch angle or the average pitch angle within the segment. In another example, minimum and / or maximum limits can be specified over the duration of the entire segment or over the duration of sub-segments within the entire segment. Other criteria are possible. Hysteresis can be employed to reduce sensitivity to noisy jitter.

[0350] In other embodiments of the present disclosure, the pose envelope detector 908 can monitor other metrics derived from the inputs providing the temporal training data 907 and use these metrics to detect poses. Examples of other metrics include, but are not limited to, roll angle, yaw, first or higher order derivatives or first or higher order integrals of the motion sensor data. The temporal data can be or can include data other than the motion sensor data. In some embodiments of the present disclosure, the pose envelope detector 908 can monitor and use multiple metrics to detect poses or specify pose time envelopes.

[0351] The pose 909, along with the pose time envelope information, plus the temporal training data 907, is fed into the feature generator module 910. The feature generator module 910 uses the information from the temporal training data 907, the pose time envelope, or a combination of the information from the temporal training data 907 and the pose time envelope to compute one or more pose features. In some embodiments of the present disclosure, the feature generator module 910 computes one or more pose features based on the temporal training data 907 that falls within or during the time periods that fall within the pose time envelope. It is also possible that the feature generator module 910 computes one or more pose features based on the temporal training data 907 that does not fall within or only partially falls within the pose time envelope but is still related to the pose time envelope. An example would be pose features computed based on the temporal training data 907 during the time period immediately before the start of the pose time envelope or during the time period immediately after the end of the pose time envelope.

[0352] In some embodiments, the feature generator module 910 can create one or more features directly based on the pose time envelope information without using the temporal training data 907. Examples of such features can include, but are not limited to, the total duration of the pose time envelope, the elapsed time since the last previous pose, the time until the next pose, or the duration of a particular sub-segment within the total pose time envelope or event time envelope.

[0353] In one embodiment, the temporal training data 907 can be motion sensor data, and the features can include readings of pitch, roll, and / or yaw angles calculated among or within one or more sub - segments within or around the pose temporal envelope. The features can also include minimum, maximum, average, variance, first - order or higher - order derivatives, and first - order or higher - order integrals of various motion sensor data inputs calculated among or within one or more sub - segments within or around the pose temporal envelope. The features can also include distances traveled in a particular direction calculated along a particular sensor axis or among or within one or more sub - segments within or around the pose temporal envelope.

[0354] The temporal training data 907 can be or can include data other than motion sensor data, such as sensor signals from one or more of the sensors described herein. The sub - segments within or in which the feature generator module 910 calculates features can be selected based on the time points or time periods specified by the pose temporal envelope. The sub - segments can also be selected based on time points or time periods from multiple pose envelopes (such as, for example, adjacent poses or poses that may not be adjacent but are otherwise closely proximate).

[0355] Some embodiments can use multiple pose envelope detectors, either in parallel or non - parallel. Parallel pose envelope detectors can operate on different subsets of the sensor data, can use different thresholds or criteria to qualify poses, etc. For example, with respect to pose recognition based on motion sensor data inputs, one pose envelope detector can use the pitch angle, while a second parallel pose envelope detector can use the roll angle. One of the pose envelope detectors can be a primary pose envelope detector, and one or more additional pose envelope detectors can be used as secondary pose envelope detectors. The feature generation logic can process the poses generated by the primary pose envelope detector, but can gleam features derived using information from the pose temporal envelopes of nearby (in time) poses obtained from one or more secondary parallel envelope detectors.

[0356] The training data can include multiple pose envelope data sets, each having an associated label (such as a selection from a list of pose labels) representing a pose that is provided manually, in a test environment, or in some other way. This training data, along with the associated labels, can be used to train a machine classifier so that it can subsequently process pose envelopes of unknown poses and determine the pose label that most appropriately matches the pose envelope. Depending on the classification method used, the training set can be the raw data (unsupervised classification) that is cleaned but otherwise raw or a set of features derived from the cleaned but otherwise raw data (supervised classification).

[0357] Regardless of the classification method, defining appropriate data boundaries for each label is important for the performance of the classifier. In the case of a time issue (i.e., an issue where at least one of the data inputs in the data input has a time dimension associated therewith), defining appropriate data boundaries can be a challenge. This is especially true if the time dimension is variable or dynamic and if features associated with a particular segment of the variable time envelope or with the overall variable time envelope contribute substantially to the performance of the classifier.

[0358] An example of such a time issue is pose recognition, such as detecting eating or drinking poses from raw motion sensor data. The duration of a bite or a suck can vary from person to person and can depend on the meal scenario or details of the food consumed.

[0359] The output of the feature generator module 910 is a set of poses 911 with corresponding time envelopes and features. Before the poses 911 can be fed into the classifier training module 915, the labels 912 from the training data set need to be mapped to their corresponding poses. This mapping operation is performed by the label mapper module 913.

[0360] In some embodiments, the timestamp associated with the label 912 always falls within the time envelope of its corresponding pose. In such cases, the logic of the label mapper module 913 can be a lookup where the timestamp of each label is compared with the start and end times of each pose time envelope and each label is mapped to the pose whose timestamp of the label is greater than the start time of the corresponding pose time envelope and less than the end time of the corresponding pose time envelope. Poses without a corresponding label can be marked as "NEGATIVE", indicating that they do not correspond to any label of interest.

[0361] However, in other embodiments of the present disclosure, the timestamp of the label 912 may not always fall within the pose time envelope. This can be due to details of the process followed during the labeling process, timing uncertainties associated with the labeling process, unpredictability or variability of the actual raw data input, or artifacts of the pose envelope detector logic. In such cases, the label mapper may be modified to adjust the boundaries of the pose envelope.

[0362] The poses 914 characterized by features and labels can then be fed into the classifier training module 915 to produce a trained statistical model that can be used by the detector subsystem. The classifier training module 915 can use statistical models such as decision tree models, K-nearest neighbor models, support vector machine models, neural network models, logistic regression models, or other models suitable for machine classification. In other variations, as Figure 9 the structure of the table and the data format of the data used can vary and can be different from that shown in Figure 9 shown.

[0363] Figure 10 illustrates an exemplary example of the detector subsystem 802. As shown herein, unlabeled temporal data 1017 is fed into Figure 10 the detector subsystem. The detector subsystem includes pose envelope detector logic 1018 and feature generator logic 1020. Functionally, the pose envelope detector logic 1018 used by the detector subsystem is similar to the pose envelope detector logic used by the corresponding training subsystem. Similarly, the feature generator logic 1020 of the detector subsystem is functionally similar to the feature generator module 910 of its corresponding training subsystem. In some embodiments, the pose envelope detector logic 1018 may monitor and use multiple metrics to detect poses or specify pose time envelopes.

[0364] However, the implementations of the pose envelope detector logic 1018 and the feature generator logic 1020 may be different in the training subsystem and its corresponding detector subsystem. For example, the detector subsystem may be implemented on hardware with more limited power, in which case it may be necessary to optimize the pose envelope detector logic 1018 to operate at lower power consumption than its counterpart used in the corresponding training subsystem. The detector subsystem may also have more stringent latency requirements than the training system. If this is the case, then it may be necessary to design and implement the pose envelope detector logic 1018 used in the detector subsystem to have lower latency than its counterpart used in the corresponding training subsystem.

[0365] The output of the feature generator logic 1020 is fed into detector logic 1022, which classifies the pose based on the trained classifier module from its corresponding training subsystem. The classification output may include one or more labels. Optionally, the detector logic 1022 may also assign a confidence level to each label.

[0366] Classification of Combination of Temporal and Non-Temporal Data Inputs

[0367] In another embodiment, the input to the classification system may include a combination of temporal and non-temporal data. Figure 11 is an exemplary example of a training subsystem according to at least one embodiment of the present disclosure, where at least some of the data inputs in the data input are temporal and at least some of the data inputs in the data input are non-temporal. Other implementations are possible.

[0368] Non-temporal training data 1129 need not be processed by the pose envelope detector 1125 and the feature generator logic 1127. The non-temporal training data 1129 can be directly fed, along with the labels 1131, into the label mapper logic 1132. In some embodiments, the non-temporal training data can be processed by a separate feature generator module (i.e., the non-temporal feature generator module 1130) to extract specific non-temporal features of interest, which are then fed into the label mapper logic 1132. The label mapper logic 1132 can use a method similar to the methods already described herein for mapping labels to poses to assign the labels 1131, along with the non-temporal features 1136 attached to the labels, to poses.

[0369] Figure 12 is an illustrative example of a classification detector subsystem according to at least one embodiment of the present disclosure, where at least some of the data inputs in the data input are temporal and at least some of the data inputs in the data input are non-temporal.

[0370] Unsupervised Classification of Temporal Data Inputs

[0371] In yet another embodiment of the present disclosure, deep learning algorithms can be used for machine classification. Classification using deep learning algorithms is sometimes referred to as unsupervised classification. With unsupervised classification, statistical deep learning algorithms perform classification tasks directly based on the processing of data, thus eliminating the need for a feature generation step.

[0372] Figure 13 illustrates an illustrative example of a classifier training subsystem according to at least one embodiment of the present disclosure, where the classifier training module is based on a statistical deep learning algorithm for unsupervised classification.

[0373] The pose envelope detector 1349 calculates a pose 1350 with a corresponding pose time envelope based on the temporal training data 1348. The data segmenter 1351 assigns one or more appropriate data segments to each pose based on the information in the pose time envelope. As an example, the data segmenter 1351 can examine the start and end time information in the pose time envelope and assign one or more data segments corresponding to the total pose duration. This is only an example. Data segments can be selected based on different segments or sub-segments defined by the pose time envelope. Data segments can also be selected based on time periods outside the pose time envelope but directly or indirectly related to the pose time envelope. Examples can be the selection of a data segment corresponding to a time period immediately before the start of the pose time envelope, or the selection of a data segment corresponding to a time period immediately after the end of the pose time envelope. Other examples of time periods outside the pose time envelope but directly or indirectly related to the pose time envelope are also possible.

[0374] Feed the pose including the data segment, pose time envelope information, and label into the classifier training module 1356. In some embodiments of the present disclosure, only a subset of the pose time envelope information is fed into the classifier training module 1356. In some embodiments of the present disclosure, the pose time envelope information may be processed before being applied to the classifier training module 1356. One example may be to align the time reference of the pose time envelope with the start of the data segment rather than with the time base of the initial temporal training data stream. Other examples are possible. By adding time envelope information that further characterizes the data segment, the performance of the classifier training module can be improved.

[0375] For example, in terms of pose recognition of an eating pose based on motion sensor data input, feeding additional anchor time information (such as the time when the pitch angle, roll, or yaw reaches a maximum or minimum value) into the classifier training module can improve the performance of the trained classifier 1357 because the trained classifier 1357 can specifically analyze the training data around the anchor time and look for features and correlations. Other examples of time envelope information that can be fed into the classifier training module are possible.

[0376] Figure 14 An illustrative example of a classification detector subsystem according to at least one embodiment of the present disclosure is shown, and the classification detector subsystem can be used in combination with Figure 13 the classification training subsystem.

[0377] Classifier Integration

[0378] In some embodiments, multiple parallel classification systems based on pose envelope detection can be used. Figure 15 An example of a system with multiple parallel classifiers is shown in. The number of parallel classification systems can vary. Each classification system 1510, 1512, 1514 has its own training and detector subsystems, and performs pose envelope detection on different subsets of the training data 1502 and label 1504 inputs to detect poses, or different thresholds or criteria can be used to qualify the poses. Therefore, each individual pose envelope detector will generate an independent set of poses, each with a different pose time envelope. The feature generator logic of each classification system creates features for the poses created by its corresponding pose envelope detector logic. These features can be different for each classification system. The classifier models used by each parallel classifier can be the same or different, or some can be the same and others can be different. Since the pose time envelopes and features used to train each classifier model are different, the parallel classification systems will produce different classification outputs 1516, 1518, 1520.

[0379] The classification outputs 1516, 1518, 1520 of each classification system can be fed into a classifier combiner subsystem 1522. The classifier combiner subsystem 1522 can combine and weight the classification outputs 1516, 1518, 1520 of the individual classification systems 1510, 1512, 1514 to produce a single overall classification result, namely the combined classification output 1524. This weighting can be static or dynamic. For example, in terms of pose recognition, some classifiers may perform better when correctly predicting the poses of one group of people, while other classifiers may perform better when correctly predicting the poses of another group of people. The classifier combiner subsystem 1522 can use different weights for different users or different scenario conditions to improve the performance of the overall classifier ensemble. Then the trained system can be used to process the unlabeled data 1506.

[0380] Other examples of time issues include, but are not limited to, autonomous driving, driver warning systems (warning the driver when detecting dangerous traffic conditions), driver alertness detection, speech recognition, video classification (such as security camera monitoring), and weather pattern recognition.

[0381] Ignoring the temporal nature of the data input and any features associated with the temporal envelope of the data input can limit the performance of the classifier and render the classifier inapplicable to classification tasks where reliable detection depends on features intrinsic to segments of a variable temporal envelope or to the overall variable temporal envelope. If the appropriate time period cannot be reliably determined or if the time period varies depending on the pose, the person, etc., performance and usability can break down.

[0382] As described herein, the improved method frames the time problem with a variable temporal envelope such that information associated with the overall variable temporal envelope or its segments can be extracted and included in the feature set used to train the classifier. The proposed improved method improves performance and reduces the amount of training data required, since features can be defined relative to the temporal bounds of the variable temporal envelope, thereby reducing sensitivity to temporal and user variance.

[0383] In addition to looking for the temporal envelope of the pose, the system can also look for an event temporal envelope. In this approach, the system may determine the pose and pose envelope, but then do so for additional poses, and then define an event envelope, such as the start and end of an eating event.

[0384] Scenarios for Improving Overall Accuracy

[0385] Figure 16An example of a machine classification system including a cross - correlation analysis subsystem is shown. The classification output 1602 can be fed into the cross - correlation analysis subsystem 1604. The cross - correlation analysis subsystem 1604 can make adjustments based on one or more situational cues to improve accuracy. In an example of gesture recognition, an example of a situational cue can be the temporal proximity to other predicted gestures. For example, eating gestures tend to cluster together in time as part of an eating activity such as a meal or a snack. As an example, the cross - correlation analysis subsystem 1604 can increase the confidence level that a predicted gesture is an eating gesture based on the confidence level and proximity of nearby predictions.

[0386] In another embodiment, the cross - correlation analysis subsystem 1604 can take individual predicted gestures 1614 from the classification output 1602 as input and can cluster the individual predicted gestures into predicted activities 1608. For example, the cross - correlation analysis subsystem 1604 can map multiple biting gestures to an eating activity such as a snack or a meal. Similarly, the cross - correlation analysis subsystem 1604 can map multiple sucking gestures to a drinking activity. Other examples of activity prediction based on gesture clustering are possible. The cross - correlation analysis subsystem 1604 can modify the confidence level of the predicted gestures based on the time intervals and sequences of the predicted activities. As an example, if a predicted gesture is detected soon after or during a "brushing teeth" activity, the cross - correlation analysis subsystem 1604 can reduce the confidence level that the predicted gesture is an eating gesture. In another example, if a predicted gesture is detected during or soon after a brushing teeth activity, the cross - correlation analysis subsystem 1604 can reduce the confidence level that the predicted gesture is a drinking gesture. In this case, the cross - correlation analysis subsystem 1604 can decide to increase the confidence level that the gesture is a mouth - rinsing gesture.

[0387] The cross-correlation analysis subsystem 1604 can adjust the classification output of the predicted pose based on historical information 1612 or other non-pose metadata 1610 information such as location, date, and time, other biometric inputs, calendar, or phone call activity information. For example, if the GPS coordinates indicate that the person is located in a restaurant, the cross-correlation analysis subsystem 1604 can increase the confidence level that the predicted pose is an eating pose or the predicted activity is an eating activity. In another example, if the time at which it occurs is a time of day when past behavior indicates that the user typically eats at that time of day, the cross-correlation analysis subsystem 1604 can increase the confidence level that the predicted pose is an eating pose or the predicted activity is an eating activity. In yet another example of the present disclosure, if the predicted pose or predicted activity is before or after a calendar event or phone call session and if past behavior indicates that the user typically eats before or after a similar calendar event (e.g., having the same attendees, located at a certain location, having a certain meeting agenda, etc.) or phone call session (e.g., from a specific phone number), the cross-correlation analysis subsystem 1604 can increase the confidence level that the predicted pose is an eating pose or the predicted activity is an eating activity. Although the above examples mention eating, it will be apparent to those skilled in the art that this can also be applied to poses other than eating. In general, a machine classifier with a cross-correlation analysis subsystem uses situational cues, historical information, and insights from temporal proximity sensing to improve accuracy, where specific situational cues, historical information, and insights from temporal proximity sensing are determined and how to apply them by the methods disclosed or suggested herein.

[0388] In some embodiments of the present disclosure, the classification output 1602 can include additional features or pose time envelope information. The cross-correlation analysis subsystem 1604 can process such additional features or pose time envelope information to determine or extract additional characteristics of the pose or activity. As an example, in one embodiment of the present disclosure, the cross-correlation analysis subsystem 1604 derives an estimated duration of a drinking pose from the pose time envelope, and the cross-correlation analysis subsystem 1604 or one or more systems external to the machine classifier system can use this information to estimate the fluid intake associated with the drinking pose.

[0389] In another embodiment, the cross-correlation analysis subsystem 1604 may derive an estimated duration of an eating gesture from the gesture time envelope, and the cross-correlation analysis subsystem 1604 or one or more systems external to the machine classifier system may use this information to estimate a bite size associated with the eating gesture. The cross-correlation analysis subsystem 1604 may combine the predicted drinking gesture with other sensor data to more accurately predict whether a person is consuming an alcoholic beverage and estimate the amount of alcohol consumed. Examples of other sensor data may include, but are not limited to, measuring hand vibrations, heart rate, voice analysis, skin temperature, measuring blood, breath chemistry, or body chemistry.

[0390] The detector subsystem 1600 may predict a particular eating or drinking method, and the cross-correlation analysis subsystem 1604 may combine information obtained from the detector subsystem 1600 about the details of the eating or drinking method with additional metadata to estimate the composition, health, or caloric intake of the food. Examples of eating / drinking methods may include, but are not limited to, eating with a fork, eating with a knife, eating with a spoon, eating with fingers, drinking from a glass, drinking from a cup, drinking from a straw, etc. Examples of metadata may include, but are not limited to, time of day, location, environmental, or social factors.

[0391] Another Exemplary Embodiment

[0392] Figure 17 According to one embodiment, Figure 1 A similar high-level functional diagram of a modified monitoring system. Figure 17 As shown, the sensor unit 1700 interacts with an event detection subsystem 1701 , which in turn interacts with an object information retrieval subsystem 1702 and provides input to a processing and analysis system, the results of which may be stored in a data storage unit 1704 .

[0393] In some embodiments, Figure 17The components shown are implemented in electronic hardware, while in other embodiments, some components are implemented in software and executed by a processor. Some functions may share hardware and processor / memory resources, and some functions may be distributed. Functions may be implemented entirely within a sensor device, such as a wrist-worn wearable device, or functions may be implemented across a sensor device, a processing system with which the sensor device communicates, such as a smart phone, and / or a server system that processes some function remote from the sensor device. For example, a wearable sensor device may make measurements and communicate the measurements to a mobile device, which may process the data received from the wearable sensor device and use this information, possibly combined with other data inputs, to activate the object information retrieval subsystem 1702. The object information retrieval subsystem 1702 may be implemented on the mobile device, on the wearable sensor device, or on another electronic device. The object information retrieval subsystem 1702 may also be distributed across multiple devices, such as, for example, across a mobile device and a wearable sensor device. Data or other information may be stored in a suitable format, distributed across multiple locations, or stored centrally in a recorded form or after some degree of processing. Data may be stored temporarily or permanently. Data may be stored locally on the wearable sensor device, on the mobile device, or uploaded to a server via the Internet.

[0394] Figure 17 A first component of the system shown is the event detection subsystem 1701. One role of the event detection subsystem 1701 is to identify the actual, possible, or impending occurrence of an event. An event may be, for example, an event related to a particular activity or behavior. Particular activities or behaviors may include, but are not limited to, eating, drinking, smoking, taking medication, brushing teeth, flossing, washing hands, applying lipstick or mascara, shaving, making coffee, cooking, urinating, using the restroom, driving, exercising, or participating in a particular sport. Other examples of events that may be detected by the event detection subsystem 1701 may be an operator on a production line or elsewhere performing a particular task or executing a particular process. Another example may be a robot or robotic arm performing a particular task or executing a particular process in a production department or elsewhere.

[0395] The event detection subsystem can use inputs from one or more sensor units 1700, other user inputs 1705, or a combination of one or more sensor inputs from sensor units 1700 and one or more other user inputs 1705 to determine or infer the actual, possible, or impending occurrence of an event. The event detection subsystem 1701 can perform additional processing on the sensor and / or user inputs to determine the actual, possible, or impending occurrence of an event. Generally speaking, the event detection system records the inferred event and / or reacts to the inferred event when the event detection system has inputs and / or data from which it determines that the event may actually be starting, is likely to start, or is impending. In some cases, the event detection system may infer an event when the event has not actually occurred and treat it as an event, but this may not happen often.

[0396] The event detection subsystem 1701 can also perform additional processing on the sensor and / or user inputs to determine additional information about the event. Such information can include, but is not limited to, the duration of the event, the start time of the event, the end time of the event, a metric associated with the rate or speed at which the subject is participating in the event. Other event data elements are possible. For example, if the event is an eating event, the event data element could be the number of bites or the amount of food consumed. Similarly, if the event is a drinking event, the event data element could be the number of sips or the fluid intake. These event data elements may be stored in a database that holds data elements regarding the inferred event.

[0397] Using gesture sensing technology, the event detection system can trigger an external device to collect additional information.

[0398] In a particular embodiment, the electronic device housing the object information retrieval subsystem 1702 or a portion of the object information retrieval subsystem 1702 includes near field communication (NFC) technology, and the object information retrieval subsystem 1702 obtains information about the object or subject with which the subject may be interacting at least in part via transmission over a wireless NFC link.

[0399] In a particular embodiment, the external device is an NFC reader and detects various objects having NFC tags thereon. The NFC reader may be integrated with gesture sensing technology or with some components of gesture sensing technology.

[0400] In cases where the objects are food / drink related, the event detection system can determine what the gesture is related to. For example, a food / drink container may have an NFC tag embedded in the product packaging, and the food intake monitoring system may automatically determine that the gesture is related to an eating event and then signal the NFC reader to turn on and read nearby NFC tags, thereby reading the NFC tag on the consumed product so that the gesture and the event are associated with a specific product.

[0401] In one example, the monitoring system may have a wearable device that determines the gestures and, based on these gestures, identifies an eating event or a drinking event. Suppose a drinking event is detected and based on the detected sensor inputs and gestures, the monitoring system determines that the user has consumed three-quarters of a can of soda, such as by counting the sucks and estimating or calculating the size of each suck. Since the gesture is likely to be the same whether it is a diet soda or a regular soda. Using an NFC reader, the specific brand and type of the soda can also be detected.

[0402] The sensors can be located in the wearable device, where the gesture determination logic or processing occurs in an external device (such as a mobile phone) communicatively coupled to the wearable device, or the gesture determination can occur partially on the wearable device and partially on an external device communicatively coupled to the wearable device.

[0403] The NFC reader can be located in the wearable device that houses the sensors, in an external device communicatively coupled to the wearable device and performing at least some of the gesture determination, or in another external device communicatively coupled to the wearable device, communicatively coupled to an external device performing at least some of the gesture determination, or communicatively coupled to both.

[0404] In general, the detection of the occurrence of an event can be used to initiate a process / system / circuit / device to collect information about the object / item or other subject that the person interacting with the object that the event represents is performing an activity or behavior on. This information can be recorded in the form of data elements. The object data elements can be stored in a database. One or more object data elements and one or more event data elements of the same event can be recorded as a single entry in the database. The object data elements and the event data elements can also be recorded as separate entries in the database. It is also possible to use or alternatively use other data structures consistent with the teachings herein.

[0405] When the NFC reader system is activated, it initiates one or more NFC read commands and wirelessly obtains additional information from the relevant object via the NFC link. Additional processing can be applied, such as filtering the NFC data to simplify downstream processing.

[0406] In other variations, other wireless links are used instead of NFC or other wireless links are used in combination with NFC. Examples of other wireless technologies include, but are not limited to, Bluetooth, Bluetooth Low Energy, Wi-Fi, Wi-Fi derivatives, and proprietary wireless technologies.

[0407] The detection of the occurrence of an event signal can be used to filter information about specific relevant objects that are related to the activity / behavior of interest that is part of that event.

[0408] Although the object detection process may be automatic, it may also require user intervention to activate the object information retrieval system. For example, this can be as simple as asking the user to turn on the camera or make a decision on whether to continue the detection process and seeking the user's advice on that decision. As another example, the user may be prompted to move the NFC reader closer to the NFC tag of interest. In another example, the user may be prompted to activate the NFC reader circuit or a portion of the NFC reader circuit, or take one or more actions that allow the NFC reader circuit to issue a read command.

[0409] In addition to the NFC reader, a camera may also be activated, and additional information from the relevant object may be obtained by analyzing an image or video recording of the relevant object. The camera and the NFC reader may be combined into a single unit. In some embodiments, only the camera is used, or other auxiliary sensors are used to obtain additional information about the event.

[0410] The information about the activity or behavior that is part of the event obtained from the event detection subsystem can be combined with the information about the object or subject with which the user is interacting, and additional processing / analysis can be performed on the combined data set to obtain additional information or insights about the activity / behavior that cannot be obtained by examining only one of the data sources alone.

[0411] Although many of the examples in this disclosure involve the event detection system analyzing the pose to detect the actual, likely, or impending occurrence of an event, other sensor inputs are possible. For example, audio signals in or near the mouth, throat, or chest can be used to detect and obtain information about the user's consumption. To achieve this, the event detection subsystem 1701 can use the input 1706 from one or more sensor units 1700. The sensors can include, but are not limited to, accelerometers, gyroscopes, magnetometers, magnetoangular rate gravity (MARG) sensors, image sensors, cameras, optical sensors, proximity sensors, pressure sensors, odor sensors, gas sensors, glucose sensors, heart rate sensors, ECG sensors, thermometers, photometers, global positioning system (GPS), and microphones.

[0412] When an actual, possible, or impending occurrence of an event is detected, the event detection subsystem may activate the object information retrieval subsystem 1702. Alternatively, the event detection subsystem 1701 may perform additional processing to determine whether to activate the object information retrieval subsystem 1702. The event detection subsystem 1701 may immediately activate the object information retrieval subsystem 1702 or wait for a certain period of time before activating the object information retrieval subsystem 1702.

[0413] The role of the object information retrieval subsystem 1702 is to collect information about the objects or other subjects with which the subject interacts some time when or after the event detection subsystem 1701 has detected the actual, possible, or impending occurrence of an event.

[0414] In one such embodiment, upon receiving an activation input signal 1707 from the event detection subsystem 1701, or some time after receiving the activation input signal 1707, the object information retrieval subsystem 1702 initiates an NFC reading action to read NFC tags attached to, contained within, or otherwise associated with one or more objects that are within the NFC range of the device housing the object information retrieval subsystem 1702. The object information retrieval subsystem 1702 sends the data received from the one or more objects to the processing and analysis subsystem 1703. The object information retrieval subsystem 1702 may perform additional processing on the data before sending the data to the processing and analysis subsystem 1703. In other embodiments, additional auxiliary sensors or sensor systems, such as cameras or other electronic devices, may be used.

[0415] Processing may include, but is not limited to, filtering; extracting specific data elements; modifying data or data elements; combining data or data elements obtained from multiple objects; combining data or data elements with data collected via other sources that are not collected via NFC. Examples of filtering may include, but are not limited to, filtering based on the distance or estimated distance between one or more objects and the object information retrieval subsystem, based on the signal strength of the received NFC signal, based on the order in which data is received from the object, based on information in the data or in specific data elements. Other filtering mechanisms or criteria for filtering are also possible. The object information retrieval subsystem 1702 may stop reading NFC tags after a fixed time, after a configurable time, after reading a fixed or configurable number of tags. Other criteria may also be used.

[0416] It is also possible that the object information retrieval subsystem collects information about the objects with which the subject interacts or other subjects independently of the event detection subsystem. In such an implementation, the processing and analysis subsystem 1703 can use the signal 1708 received from the event detection subsystem 1701 together with the data signal 1709 received from the object information retrieval subsystem 1702 to infer relevant information about the objects with which the subject interacts or other subjects during an activity or when exhibiting a certain behavior.

[0417] In a particular implementation of the present disclosure, the object information retrieval subsystem 1702 reads NFC tags from objects continuously, periodically, or otherwise independently of the input from the event detection subsystem 1701, where the NFC tags are within the range of an electronic device that fully or partially houses the object information retrieval subsystem 1702. The detection of the occurrence of the activity / behavior signal can be used to filter information about specific relevant objects that are related to the activity / behavior of interest. In the case where an object is associated with an event, an indication or reference to the specific object may be recorded as part of the record of the event so that the information can be used later for filtering or other processing.

[0418] In a particular implementation, the object information retrieval subsystem 1702 collects data independently of the input it receives from the event detection subsystem, but only sends the data to the processing and analysis subsystem 1703 when it receives an activation input signal 1707 from the event detection subsystem 1701 or some time after that. The object information retrieval subsystem 1702 may send only a subset of the data that it has received from the object via the NFC link. For example, it may send only the data that it has received within a fixed or configurable time window related to the time when it received the activation input signal 1707. For example, it may send data only immediately before and / or immediately after the activation input signal 1707. Other time windows are possible. The object information retrieval subsystem 1702 may perform additional processing on the data before sending it to the processing and analysis subsystem 1703.

[0419] In one implementation of the present disclosure, the object information retrieval subsystem 1702 includes a camera and can derive information about one or more objects from the analysis of an image or video recording.

[0420] In a preferred embodiment of the present disclosure, the object information retrieval subsystem 1702 collects data from the object without any intervention or input from the subject. In another embodiment of the present disclosure, some user input or intervention is required. For example, the user may be prompted to move the NFC reader closer to the NFC tag of interest. In another example, the user may be prompted to activate the NFC reader circuit or a portion of the NFC reader circuit, or to take one or more actions that allow the NFC reader circuit to issue a read command.

[0421] Figure 18 FIG. shows a high-level functional diagram of a monitoring system that requires user intervention according to one embodiment. As Figure 18 shown, one or more sensor units 1800 interact with an event detection subsystem 1801 and send sensor data to the event detection subsystem 1801. When an actual, possible, or impending event is detected or inferred, the event detection subsystem sends one or more notifications to a user interaction unit 1802 to request activation of an NFC scan action. The notifications may be presented to the user as a displayed text message, as a displayed image, as an audio message, as an LED signal, as a vibration, etc. Combinations of user interfaces may also be used. Other user interfaces may also be used. The user may respond to the notifications using one or more user interfaces of the user interaction unit 1802. The user response may trigger the user interaction unit 1802 to send a scan command 1810 to an object information retrieval subsystem 1803. Upon receipt of the scan command 1810 or some time after receipt of the scan command 1810, the object information retrieval subsystem 1803 may activate an NFC reader 1806 and obtain information about an object 1804 via a wireless communication link (also referred to simply as a "wireless link") 1811 between the NFC reader 1806 and an NFC tag 1807. The information obtained via the wireless link 1811 may include information about the brand, type, composition, expiration date, lot number, etc. of the object 1804. Other information about the object may also be retrieved. Event data elements 1814 from the event detection subsystem 1801 and object data elements 1813 retrieved by the object information retrieval subsystem 1803 from one or more objects via the wireless link 1811 may be sent to a processing and analysis subsystem 1805. Additional processing may be performed and the event and object data elements may be stored in a database on one or more data storage units 1815.

[0422] In another example, the event detection system automatically scans the NFC tag, but when data is received from the object, the object information retrieval subsystem sends a message to the subject and the subject authorizes or does not authorize the transmission to the processor or analysis subsystem or the subject directly confirms the information. The message may be sent by the processing and analysis subsystem.

[0423] The processing performed by the processing and analysis subsystem 1805 may include, but is not limited to, filtering; extracting specific data elements; modifying data or data elements; combining data or data elements obtained from multiple objects; combining data or data elements with data obtained from other sources that is not collected via NFC. Examples of filtering may include, but are not limited to, filtering based on the distance or estimated distance between one or more objects and the object information retrieval subsystem, based on the signal strength of the received NFC signal, based on the order in which data is received from the object, based on information in the data or in a specific data element. Other filtering mechanisms or criteria for filtering are also possible. The object information retrieval subsystem 1803 may stop sending data after a fixed time, after a configurable time, after reading a fixed or configurable number of tags. Other criteria may also be used. The object information retrieval subsystem 1803 may send data only from a single object, from a subset of objects, or from all objects from which it receives data within a specified time window.

[0424] In various embodiments of the present disclosure, the object information retrieval subsystem 1803 reads NFC tags from objects (which are within the range of an electronic device that fully or partially houses the object information retrieval subsystem 1803) continuously, periodically, or otherwise independently of input from the event detection subsystem 1801 and sends such data to the processing and analysis subsystem 1805 independently of any signal from the event detection subsystem.

[0425] The processing and analysis subsystem 1805 receives data inputs from the event detection subsystem 1801 and the object information retrieval subsystem 1803. It is also possible that the processing and analysis subsystem 1805 receives input only from the object information retrieval subsystem 1803. The processing and analysis subsystem 1805 may perform additional processing on the data and may analyze the data to extract information about the object or subject from the data it receives. The processing may include, but is not limited to, filtering; extracting specific data elements; modifying data or data elements; combining data or data elements obtained from multiple objects. The analysis may also include comparing specific data elements with data stored in a lookup table or database; associating data elements with data elements obtained at an earlier time and / or from a different subject. Other processing and analysis steps are also possible. The processing and analysis subsystem 1805 may store the raw or processed data in the data storage unit 1815. The storage may be temporary or permanent.

[0426] In some embodiments, the output from the processing and analysis subsystem 1805 may be available in real time, during or shortly after an event. In other embodiments, the output may not be available until a later time.

[0427] The processing and analysis subsystem 1805 can be implemented on a mobile device, on a wearable sensor device, or on another electronic device. The processing and analysis subsystem 1805 can also be distributed across multiple devices, such as, for example, across a mobile device and a wearable sensor device. In another example, the processing and analysis subsystem 1805 can be distributed across a mobile device and a local or remote server. The processing and analysis subsystem 1805 can also be implemented entirely on a local or remote server. Information can be stored in a suitable format, distributed across multiple locations, or centrally stored in a recorded form or after some degree of processing. Data can be stored temporarily or permanently. Data can be stored locally on a wearable sensor device, on a mobile device, or uploaded to a server via the Internet.

[0428] The object information retrieval subsystem can be fully or partially housed within a battery-operated electronic device, and it may be desirable to minimize the power consumption of the object information retrieval subsystem. When an event is not detected, the radio circuit (e.g., the NFC reader circuit) can be placed in a low-power state. When an actual, possible, or impending occurrence of an event is detected or inferred, the object information retrieval subsystem can be placed in a higher-power state. One or more additional circuits within the object information retrieval subsystem can be powered on to activate the object information retrieval subsystem, improve the range or performance of the object information retrieval subsystem, etc. In a specific example, the NFC reader is disabled or placed in a low-power standby or sleep mode when an event is not detected. When an event is detected or inferred, the NFC reader is placed in a higher-power state, in which it can communicate with the NFC tags of neighboring objects. After reading a preconfigured number of NFC tags, after a preconfigured time, or when the end or completion of the event is detected, the NFC reader can be disabled again or placed back in a low-power standby or sleep mode.

[0429] Example of Drug Delivery System

[0430] Systems in accordance with the principles and details described herein can be used to detect the start of eating / drinking to initiate the administration of microdosed insulin as a form of insulin therapy and / or a meal-aware artificial pancreas. The insulin dose calculator can consider the glucose level at the start of eating, the slope of the glucose level at the start of eating to determine the dose and timing of insulin delivery. Insulin can be delivered either once or in multiple microdoses. If additional information about the food is obtained from the object information retrieval subsystem (e.g., drinking a regular soda instead of a diet soda), the insulin dose calculator can consider this information. For example, if a food item with a high sugar content is being consumed, the insulin dose calculator can increase the dose administered per microdose event or can increase the number of microdoses delivered within a given time period.

[0431] Figure 19Shows a high-level functional diagram of a drug delivery system covered by the present disclosure. The drug delivery system may partially include one or more of the following: a diet tracking and feedback system 1902, one or more sensor units 1900, a measurement sensor processing unit 1909, a drug administration calculation unit 1906, one or more measurement sensor units 1904, and a drug delivery unit 1908.

[0432] In one embodiment of the present disclosure, the drug to be administered is insulin, the measurement sensor unit 1904 is a continuous glucose monitor sensor that measures interstitial fluid glucose levels, the drug delivery unit 1908 is an insulin pump, and the drug administration calculation unit 1906 is an insulin dose calculation unit of an automated insulin delivery system (also known as an artificial pancreas).

[0433] Figure 19 Each of the elements shown may be implemented by a suitable structure. For example, these elements may be separate hardware elements, or may be implemented as software structures in a wearable device, an auxiliary device that communicates with the wearable device, or a server coupled to the auxiliary device and / or the wearable device via a network. Some elements may be entirely software and coupled to other elements capable of sending and / or receiving messages to or from the processor executing the software. For example, the drug delivery unit 1908 may be an embedded hardware system that injects a micro-dose of the drug in response to instructions given by a software system, and thus will only require minimal on-board processing.

[0434] The diet tracking and feedback system 1902 may be implemented as described elsewhere herein and may monitor the output of one or more sensor units to determine the actual, possible, or impending start of a food intake event. At or some time after detecting the actual, possible, or impending start of a food intake event, it may send a signal 1903 to the drug administration calculation unit 1906 to notify the drug administration calculation unit 1906 that the actual, possible, or impending start of a food intake event has been detected. The drug administration calculation unit 1906 may use this information to change its state to the "meal in progress" state.

[0435] At or some time after entering the meal in progress state, the drug administration calculation unit 1906 may calculate an initial meal drug dose to be administered and send one or more messages to the drug delivery unit 1908. Alternatively, the drug dose to be administered in combination with the start of a food intake event may have been pre-configured or calculated prior to the occurrence of the food intake event. Upon receiving these messages 1907, the drug delivery unit 1908 may initiate the delivery of the drug.

[0436] The drug delivery unit can deliver drugs either once or according to a delivery schedule. The delivery schedule can be determined by the drug administration calculation unit and communicated to the insulin delivery system. The delivery schedule can be determined when entering the in-meal state or some time after that. The delivery schedule can also be preconfigured before the occurrence of a food intake event.

[0437] The initial meal drug dose and / or delivery schedule can cover the entire expected drug dose for the food intake event. Alternatively, the initial meal drug dose and / or delivery schedule can only cover a portion of the entire expected drug dose for the food intake event, and additional drug doses are expected at a later time during or after the food intake event.

[0438] The drug administration calculation unit 1906 can consider additional inputs when calculating the initial drug dose and / or the initial delivery schedule. Some inputs can relate to current or recent measurements, current or recent user activities and behaviors, or other information corresponding to the user's current or recent state or condition. Other inputs can relate to historical measurements, historical user activities and behaviors, or other information corresponding to the user's past state or condition.

[0439] Examples of Additional Inputs

[0440] The drug administration calculation unit 1906 can consider one or more outputs 1910 from the measurement sensor processing unit 1909. The drug administration calculation unit 1906 can perform additional processing steps on the output 1910. For example, the measurement sensor unit 1904 can be a continuous glucose monitor (“CGM”), and the output 1910 of the measurement sensor processing unit 1909 can be an interstitial fluid glucose level reading. The output 1910 can be updated, for example, every few minutes. Other update frequencies are also possible. The output 1910 can also be updated continuously. The drug administration calculation unit 1906 can consider one or more interstitial fluid glucose level readings. For example, the drug administration calculation unit 1906 can consider the most recent reading. The drug administration calculation unit 1906 can calculate certain parameters indicating the change in the interstitial fluid glucose level reading. For example, the drug administration calculation unit 1906 can calculate the minimum value, average value, maximum value, standard deviation, slope, or second derivative of the interstitial fluid glucose level reading within one or more time windows. The time window can span a time period before the transition to the in-meal state, span a time period including the transition to the in-meal state, or span a time period some time after the transition to the in-meal state. Other or additional measurement sensor units (such as heart rate, blood pressure, body temperature, degree of hydration, degree of fatigue) are also possible. The insulin delivery system can also consider the user's current location.

[0441] The drug administration calculation unit 1906 may also consider other inputs, such as information related to the user's current or recent physical activity, sleep, stress, etc. The drug administration calculation unit 1906 may also consider personal information, such as gender, age, height, weight, etc.

[0442] The drug administration calculation unit 1906 may also consider information related to the user's drug administration needs, such as for example the user's insulin basal rate, the user's insulin / carbohydrate ratio, and the user's insulin correction factor. This information may be input or configured by the user, a caregiver, or a health record or healthcare maintenance system. Information related to the user's drug administration needs may also be derived from historical data collected and stored by the drug dispensing system. For example, the dose of a drug (such as insulin) delivered by the drug dispensing unit during a time period prior to the current food intake event. The drug administration calculation unit 1906 may consider the dose of the drug delivered in combination with one or more previous food intake events that occurred in the past at or around (e.g., within a specified time window) the same time of day and / or the same day of the week.

[0443] The drug administration calculation unit 1906 may also consider drugs that are still effective from previous drug dispensing events, such as for example residual insulin in the body.

[0444] The drug administration calculation unit 1906 may also include parameters related to food intake events that occurred in the past at or around (e.g., within a specified time window) the same time of day and / or the same day of the week and / or at the same location. The drug administration system may for example examine the duration of past food intake events, the estimated amount consumed during past food intake events, the average rate of eating during past food intake events, the method of eating for past food intake events, the type of utensils or containers used during past food intake events, or the amount of carbohydrates consumed during past food intake events. Other parameters are possible. The food intake tracking and feedback system may calculate some of these additional parameters (such as duration or rate) without any user intervention. In other cases, user intervention, input, or confirmation by the user may be necessary.

[0445] Drug Delivery Schedule

[0446] The drug administration calculation unit 1906 may instruct the drug dispensing unit to administer an initial drug dose all at once, or may specify a delivery schedule for administering the drug. In one embodiment of the present disclosure, the drug administration calculation unit 1906 calculates the drug dose and the schedule for delivering the drug. As an example, the drug administration calculation unit 1906 may determine that 5 units of insulin need to be delivered and may specify the delivery schedule as follows: administer 2 units immediately, 1 unit after 2 minutes, 1 unit after 5 minutes, and 1 unit after 7 minutes. This is only an example, and of course other time distribution structures are possible.

[0447] The drug administration calculation unit 1906 may communicate both the drug dose and the schedule to the drug dispensing unit 1908. Alternatively, the drug administration calculation unit 1906 may manage the schedule and send one or more messages along with the dose of the drug to be administered to the drug dispensing unit 1908 whenever the drug needs to be administered.

[0448] In a preferred embodiment of the present disclosure, the drug administration calculation unit 1906 may instruct the drug dispensing unit 1908 to initiate the delivery of the drug at 1907 when entering the in - meal state or some time after. It may, for example, instruct the drug dispensing unit 1908 to deliver the drug in one or more small micro - doses.

[0449] Additional Doses and Dose Adjustments

[0450] During and / or some time after a food intake event, the drug administration calculation unit 1906 may periodically (e.g., every few minutes) or continuously partially monitor one or more inputs 1905 from the measurement sensor processing unit 1909 and / or one or more inputs 1903 from the diet tracking and feedback system 1902 to determine whether additional drug should be administered and how much, or whether the predetermined drug dispensing should be adjusted. Other inputs may also be considered, such as those described in the previous parts of the present disclosure.

[0451] When calculating the drug dose or drug dose adjustment, the drug administration calculation unit 1906 may consider whether a food intake event is in progress. If the food intake event is not in progress or no longer in progress, the drug administration calculation unit 1906 may, for example, consider the time since the end of the past food intake event. If the food intake event is in progress, the drug administration calculation unit 1906 may, for example, consider the time elapsed since the start of the current food intake event, the average or median rate of eating since the start of the current food intake event, the estimated total amount consumed since the start of the current food intake event. Other examples are also possible.

[0452] The drug administration calculation unit 1906 may also consider one or more recent inputs from the measurement sensor processing unit 1909. For example, in the case where the drug administration calculation unit 1906 is an insulin administration unit in an automated insulin delivery system (also known as an artificial pancreas) and the measurement sensor processing unit 1909 is a CGM, the drug administration calculation unit 1906 may consider the value of the most recent interstitial fluid glucose level reading and / or the change in the interstitial fluid glucose level reading within a time window immediately preceding or adjacent to the current time. If the most recent interstitial fluid glucose level reading is below a specified threshold and / or the change in the interstitial fluid glucose level reading exceeds a specified negative threshold, the drug administration calculation unit 1906 may decide to decrease the insulin dose or pause insulin delivery until the interstitial fluid glucose level reading reaches a second specified threshold and / or the change in the interstitial fluid glucose level reading no longer exceeds a second specified negative threshold, has become positive, or exceeds a specified positive threshold.

[0453] In some embodiments of the present disclosure, upon detecting an output related to the measurement sensor processing unit, the drug administration calculation unit 1906 may send a warning to the user, one or more of his caregivers, a healthcare provider, a monitoring system, or an emergency response system, or a third party that may have a direct or indirect interest in being informed of the occurrence of such an event.

[0454] Similarly, if the most recent interstitial fluid glucose level reading exceeds a specific threshold and / or the change in the interstitial fluid glucose level reading exceeds a specified positive threshold, the drug administration calculation unit 1906 may determine that an additional drug dose should be administered, that a previously scheduled drug dose needs to be adjusted to a larger dose, or that the drug dose should be delivered at a time earlier than the currently scheduled time. The drug administration calculation unit 1906 may optionally consider additional inputs from the diet tracking and feedback system 1902 to calculate the additional drug dose or the drug dose adjustment. The drug administration calculation unit 1906 may send one or more messages 1907 to the drug dispensing unit 1908 to inform the drug dispensing unit 1908 of the additional or adjusted drug dispensing requirements.

[0455] At or after some time when the actual or near end of a food intake event is detected, the dietary tracking and feedback system 1902 may send a signal 1903 to the drug administration computing unit 1906 to notify the drug administration computing unit 1906 that the actual or near end of a food intake event has been detected. The drug administration computing unit 1906 may use this information to change its state to an "ongoing meal not in progress" state. In certain embodiments, when in the ongoing meal not in progress state, the drug administration computing unit may periodically or continuously partially monitor one or more inputs 1905 from the measurement sensor processing unit 1909 and / or one or more inputs 1903 from the dietary tracking and feedback system 1902 to determine whether and how much additional drug should be administered or whether a predetermined drug distribution should be adjusted. Other inputs may also be considered, such as those described in previous portions of the present disclosure. When in the ongoing meal not in progress state, the frequency of monitoring and / or updating / regulating drug administration may be different from the frequency of monitoring and / or updating / regulating drug administration when in the "meal in progress" state. The algorithm for determining the drug dose or drug dose adjustment may also be different between the "meal in progress" state and the "ongoing meal not in progress" state.

[0456] Description of Drug Administration Learning System

[0457] In some embodiments of the present disclosure, the drug administration computing unit 1906 may collect and store data and information regarding food intake events. In some embodiments, the drug administration computing unit 1906 may perform additional processing steps on the collected data and information before storage. The processing steps may be filtering, averaging, applying arithmetic operations, and applying statistical operations. Other processing steps are also possible.

[0458] Data and information regarding food intake events may be stored as data elements in a database that maintains a data record regarding food intake events.

[0459] The data elements may be event data elements and contain information or parameters characterizing the food intake event. Such information may include, but is not limited to, the event start time, the day of the week on which the event occurred, the date of the event, the duration of the event, the event end time, a metric associated with the rate or speed at which the subject ate or drank, and a metric associated with the amount of food or liquid consumed during the event.

[0460] The data element can also be a measurement data element and includes information or parameters characterizing one or more signals measured by one or more measurement sensor units 1904 and processed by one or more measurement sensor processing units 1909. Such information can include, but is not limited to, the sensor reading level corresponding to a specific time related to a food intake event, or the average, minimum, or maximum sensor reading level corresponding to a specific time window related to a food intake event. The specific time may be, for example, the start of a food intake event, a periodic or predefined time point during a food intake event, the end of a food intake event, or a periodic or predefined time after a food intake event. Other times are also possible. The specific time window may be, for example, the duration immediately before the start of a food intake event, the duration before the start of a food intake event, the duration of a food intake event, a specific duration within a food intake event, the duration immediately after the end of a food intake event, or the duration some time after the end of a food intake event.

[0461] In a particular embodiment, the drug administration calculation unit is an automated insulin delivery system, and the sensor reading level is the interstitial fluid glucose reading level obtained from a continuous glucose monitoring sensor.

[0462] The data element can also be a dose data element and includes information or parameters characterizing the drug dose and delivery schedule related to a food intake event.

[0463] Other data elements are also possible. One or more event data elements, one or more measurement data elements, and / or one or more dose data elements of the same food intake event can be recorded as a single record entry in a database. The event data element, the measurement data element, and / or the dose data element can also be recorded as separate records in the database. It is also possible to use or alternatively use other data structures consistent with the teachings herein.

[0464] The drug administration calculation unit 1906 can include a processing and analysis subsystem.

[0465] The processing and analysis subsystem can use statistical, machine learning, or artificial intelligence techniques on the entries in the database to build a model that recommends an adequate drug dose and / or delivery schedule. The processing and analysis subsystem can be used to recommend an initial drug dose and / or recommend additional drug doses or dose adjustments.

[0466] Figure 20 is an illustrative example of a machine learning system that can be used with other elements described in this disclosure. Figure 20 The machine learning system includes a dose training subsystem 2020 and a dose predictor subsystem 2021. In some embodiments of the present disclosure, the machine learning system can include additional subsystems orFigure 2 A modified version of the subsystem shown. The dose training subsystem 2020 may use the event data element 2022, the measurement data element 2023, and the dose data element 2024 as inputs. The dose training subsystem applies machine learning techniques to build a model that recommends an adequate drug dose and / or a drug administration schedule. It may use supervised learning techniques on one or more entries from a database to train the model. The event data element 2022 and / or the measurement data element 2023 may be used as features of the model. One or more dose data elements 2024 may be used as labels. Then the trained model 2025 and / or 2029 is used in the dose predictor subsystem 2021 to generate drug dose recommendations and / or drug administration recommendations corresponding to new unlabeled data inputs 2026.

[0467] The drug administration calculation unit 1906 may include a processing unit and perform additional processing on the data elements and analyze the data to extract information about the user's eating and drinking activities and behaviors, sensor measurements (e.g., blood glucose control), and / or drug regimens. The processing may include, but is not limited to, filtering; extracting specific data elements; modifying data or data elements; combining data or data elements. The analysis may also include comparing specific data elements with data stored in a lookup table or database; associating data elements with data elements obtained at an earlier time and / or from different subjects. The drug administration calculation unit 1906 may store the raw or processed data in one or more data storage units. The storage may be temporary or permanent.

[0468] In some variations, the records in the database may be subdivided into groups (e.g., based on meal type - breakfast, lunch, dinner, snack) and different models may be used for each subgroup. Alternatively, the same model may be used, but only data from one of the subgroups or a selective set of subgroups may be used to train the model. In other variations, instead of using supervised machine learning methods (i.e., where features are manually specified), unsupervised learning may be used. With unsupervised learning, the classifier will autonomously generate features from the provided raw data set.

[0469] The medication administration calculation unit 1906 can collect and store data and information regarding other user activities. For example, the medication administration calculation unit 1906 can collect information or data regarding the user's physical activity, sleep activity, sexual activity. The medication administration calculation unit 1906 can also collect and store information regarding the user's stress, heart rate, blood pressure, etc. In some embodiments, the medication administration calculation unit 1906 can perform additional processing steps on the collected data and information before storage. The processing steps can be filtering, averaging, applying arithmetic operations, and applying statistical operations. Other processing steps are possible. The medication administration calculation unit 1906 can associate the data and information with one or more food intake events. Data and information regarding food intake events may be stored as data elements in a database that maintains a data record regarding food intake events. These data elements can also be used as inputs to the processing and analysis subsystems of the medication administration calculation unit 1906. For example, these data elements can be used as additional or alternative event data inputs to the dose training subsystem 1920. These data elements can be features of the model, for example.

[0470] The medication administration system can also collect inputs such as information related to the user's physical activity, sleep, stress, etc. The medication administration system can, for example, compare the user's current or recent physical activity with past physical activity and use the output of the comparison for the calculation of a medication dose.

[0471] Although not shown in detail in Figure 20 The machine learning system, the dose training subsystem 2020, and the dose predictor subsystem 2021 can be implemented using various structural elements. For example, dose prediction can be implemented using computer hardware such as a processor, a program code memory, and program code stored in the program code memory. This can be a separate embedded unit, or it can be implemented on a processor with a memory for other functions and tasks as well as dose prediction. The processor and the memory can be built into a wearable device, a mobile device communicating with the wearable device, a server communicating directly or indirectly with the wearable device or the sensor, or some combination of the above. Other elements can be implemented similarly, such as a storage device for the event data element 2022; a storage device for the measurement data element 2023; and a storage device for the dose data element 2024; program code implementing machine learning techniques to build a model that recommends an adequate medication dose and / or a medication dispensing schedule; a storage device for the model; a storage device for a database for training the model; and the hardware circuitry required to convey messages such as sending a medication dose recommendation message, a medication dispensing recommendation message, or a signal to a hardware device that dispenses medication.

[0472] In certain embodiments, Figure 19The drug dispensing system can operate without any manual intervention or at least without much manual intervention. In other embodiments, Figure 19 the drug dispensing system may require some manual intervention. In one example, the drug administration calculation unit 1906 can calculate the drug dose and / or the drug delivery schedule, but it does not instruct the drug dispensing unit 1908 to initiate or schedule the delivery of the drug. Instead, it sends a message to the patient, one or more caregivers of the patient, healthcare professionals, monitoring systems, etc. to confirm the proposed drug dose and / or drug delivery schedule. The message can be a text message, a push notification, a voice message, etc., but other message formats are also possible.

[0473] In some embodiments, the patient, caregiver, etc. may have the option to change the proposed drug dose and / or drug delivery schedule. Upon receiving confirmation from the patient, caregiver, etc., the drug administration calculation unit 1906 can send one or more instructions to the drug dispensing unit 1908 to initiate or schedule the delivery of the drug. The instructions can also be sent by a device or unit other than the drug administration calculation unit 1906. As an example, the instructions can be sent directly from a device to the drug dispensing unit 1908, on which the patient, caregiver, etc. receive the message to confirm the drug dose and / or drug delivery schedule. Other user interventions are also possible, such as allowing a "nap" function to move the message to a pre-determined time in the future.

[0474] Examples

[0475] Example 1: An automated drug administration and dispensing system, the automated drug administration and dispensing system comprising: a sensor that detects motion and other physical inputs related to a user of the automated drug administration and dispensing system; a computer-readable storage medium that includes program code instructions; and a processor, wherein the program code instructions are configurable to cause the processor to execute a method including the steps of: determining the occurrence of a posture-based body behavior event of the user from sensor readings obtained from the sensor; and adjusting a drug dose, drug dispensing parameters, or both the drug dose and drug dispensing parameters in response to the determination.

[0476] Example 2: The system according to Example 1, wherein at least one of the sensor readings measures the motion of a body part of the user.

[0477] Example 3: The system according to Example 1, the system further comprising an event detection module that determines the posture of the user from the sensor readings.

[0478] Example 4: The system according to Example 1, wherein the method further comprises the step of sending a message to the user, wherein the message is related to the adjustment.

[0479] Example 5: The system according to Example 1, wherein the posture-based body behavior event corresponds to a user activity unrelated to a food intake event.

[0480] Example 6: The system according to Example 5, wherein the user activity unrelated to the food intake event includes a smoking event, a personal hygiene event, and / or a drug-related event.

[0481] Example 7: The system according to Example 1, wherein the posture-based body behavior event corresponds to a food intake event.

[0482] Example 8: The system according to Example 1, wherein the adjustment is performed when an actual, possible, or impending start of a posture-based body behavior event is detected.

[0483] Example 9: The system according to Example 1, wherein the adjustment is based on characteristics of the posture-based body behavior event.

[0484] Example 10: The system according to Example 9, wherein the posture-based body behavior event corresponds to a food intake event; and the adjustment is based on at least one of the following characteristics of the food intake event: duration; speed; start time; end time; number of bites; number of sucks; eating method; type of tableware used; type of container used; amount of chewing before swallowing; chewing speed; amount of food consumed; amount of carbohydrates consumed; time between bites; time between sucks; composition of the food consumed.

[0485] Example 11: The system according to Example 1, wherein: the drug managed by the system is insulin; and the adjustment step calculates a dose of insulin to be administered and a delivery schedule for the calculated dose of insulin.

[0486] Example 12: The system according to Example 1, wherein the sensor includes an accelerometer that measures the movement of the user's arm and a gyroscope that measures the rotation of the user's arm.

[0487] Example 13: A method of operating an automated drug administration and dispensing system having sensors that detect user-related movements and other physical inputs, the method comprising the steps of: obtaining, using a processor of the automated drug administration and dispensing system, a set of sensor readings, wherein at least one of the sensor readings in the set of sensor readings measures the movement of a body part of the user; determining, from the set of sensor readings, the occurrence of a posture-based body behavior event of the user; and adjusting a drug dose, drug dispensing parameters, or both the drug dose and the drug dispensing parameters in response to the determination.

[0488] Example 14: The method according to Example 13, the method further comprising the step of performing a computer-based action in response to the determination, wherein the computer-based action is one or more of the following: obtaining other information to be stored in a memory associated with data representing a posture-based body behavior event; interacting with the user to provide information or a reminder; interacting with the user to prompt the user for input; sending a message to a remote computer system; sending a message to another person; sending a message to the user.

[0489] Example 15: The method according to Example 13, wherein the posture-based body behavior event corresponds to a user activity unrelated to a food intake event.

[0490] Example 16: The method according to Example 15, wherein the user activity unrelated to a food intake event includes a smoking event, a personal hygiene event, and / or a drug-related event.

[0491] Example 17: The method according to Example 13, wherein the posture-based body behavior event corresponds to a food intake event.

[0492] Example 18: The method according to Example 13, wherein the adjustment is performed upon detection of the actual, possible, or imminent start of the posture-based body behavior event.

[0493] Example 19: The method according to Example 13, wherein the adjustment is based on the characteristics of the posture-based body behavior event.

[0494] Example 20: The method according to Example 19, wherein: the posture-based body behavior event corresponds to a food intake event; and the adjustment is based on at least one of the following characteristics of the food intake event: duration; speed; start time; end time; number of bites; number of sucks; eating method; type of eating utensil used; type of container used; amount of chewing before swallowing; chewing speed; amount of food consumed; time between bites; time between sucks; composition of the food consumed.

[0495] Example 21: An automated drug administration and dispensing system, the automated drug administration and dispensing system comprising: a sensor that detects movement associated with a user of the automated drug administration and dispensing system; a computer-readable storage medium that includes program code instructions; and a processor, wherein the program code instructions are configurable to cause the processor to execute a method comprising the steps of: determining, from sensor readings obtained from the sensor, the start or expected start of a current food intake event of the user; reviewing historical data collected for a previously recorded food intake event of the user; identifying a correlation between the current food intake event and a plurality of previously recorded food intake events; and adjusting a drug dose, drug dispensing parameters, or both the drug dose and drug dispensing parameters based on the identified correlation.

[0496] Example 22: The system according to Example 21, wherein at least one of the sensor readings measures movement of a body part of the user.

[0497] Example 23: The system according to Example 21, the system further comprising an event detection module that determines a body behavior event of the user from the sensor readings.

[0498] Example 24: The system according to Example 23, wherein the event detection module determines a posture of the user characterizing the current food intake event.

[0499] Example 25: The system according to Example 21, wherein the adjustment is based on at least one of the following characteristics of the food intake event: duration; speed; start time; end time; number of bites; number of sucks; eating method; type of tableware used; type of container used; amount of chewing before swallowing; chewing speed; amount of food consumed; time between bites; time between sucks; composition of the food consumed.

[0500] Example 26: The system according to Example 21, wherein: the drug managed by the system is insulin; and the adjustment step calculates a dose of insulin to be administered and a delivery schedule for the calculated dose of insulin.

[0501] Example 27: The system according to Example 21, wherein the sensor includes an accelerometer that measures movement of the user's arm and a gyroscope that measures rotation of the user's arm.

[0502] Example 28: The system according to Example 21, wherein the historical data includes parameters not directly related to the food intake event.

[0503] Example 29: The system according to Example 28, wherein the parameter includes at least one of the following: location information; the time of day the user wakes up; stress level; sleep behavior pattern; calendar event details; phone call information; email metadata.

[0504] Example 30: A method of operating an automated drug administration and dispensing system, the automated drug administration and dispensing system having a sensor that detects movement related to a user, the method comprising the steps of: determining the start or expected start of a current food intake event of the user from sensor readings obtained from the sensor; reviewing historical data collected for previously recorded food intake events of the user; identifying a correlation between the current food intake event and a plurality of previously recorded food intake events; and adjusting a drug dose, drug dispensing parameter, or both the drug dose and the drug dispensing parameter based on the identified correlation.

[0505] Example 31: The method according to Example 30, wherein at least one of the sensor readings measures movement of a body part of the user.

[0506] Example 32: The method according to Example 30, the method further comprising the step of determining a body behavior event of the user from the sensor readings.

[0507] Example 33: The method according to Example 32, wherein the body behavior event determined from the sensor readings includes the posture of the user characterizing the current food intake event.

[0508] Example 34: The method according to Example 30, wherein the adjustment is based on at least one of the following characteristics of the food intake event: duration; speed; start time; end time; number of bites; number of sucks; eating method; type of tableware used; type of container used; amount of chewing before swallowing; chewing speed; amount of food consumed; time between bites; time between sucks; composition of the food consumed.

[0509] Example 35: The method according to Example 30, wherein: the drug managed by the system is insulin; and the adjustment step calculates the dose of insulin to be administered and the delivery schedule of the calculated dose of insulin.

[0510] Example 36: The method according to Example 30, wherein the sensor includes an accelerometer that measures movement of the user's arm and a gyroscope that measures rotation of the user's arm.

[0511] Example 37: The method according to Example 30, wherein the historical data includes parameters not directly related to the food intake event.

[0512] Example 38: The method according to Example 37, wherein the parameter comprises at least one of the following: location information; the time of day when the user wakes up; stress level; sleep behavior pattern; calendar event details; phone call information; email metadata.

[0513] Example 39: The method according to Example 30, wherein the adjustment is performed when an actual or imminent start of a current food intake event is detected.

[0514] Example 40: The method according to Example 30, wherein the adjustment is based on characteristics of a current posture-based body behavior event.

[0515] Conclusion

[0516] As described above, various methods and devices can be used as part of the drug delivery regimens and alternatives provided herein. Unless otherwise specifically stated or clearly contradicted by the context, connective phrases such as the phrase "at least one of A, B, and C" shall be understood otherwise in accordance with the context which is generally used to present items, terms, etc. as either being A or B or C, or as being any non-empty subset of the set A and B and C. For example, in an illustrative example of a set having three members, the connective phrase "at least one of A, B, and C" refers to any of the following sets: {A}, {B}, {C}, {A, B}, {A, C}, {B, C}, {A, B, C}. Thus, such connective phrases are not generally intended to imply that certain embodiments require the presence of at least one of A, at least one of B, and at least one of C.

[0517] Unless otherwise specified herein or clearly contradicted by the context, the operations of the processes described herein can be performed in any suitable order. The processes described herein (or variations and / or combinations thereof) can be performed under the control of one or more computer systems configured with executable instructions and can be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that collectively execute on one or more processors, and can be implemented by hardware or combinations thereof. The code can be stored, for example, in the form of a computer program comprising multiple instructions executable by one or more processors on a computer-readable storage medium. The computer-readable storage medium can be non-transitory.

[0518] Unless otherwise stated, the use of any and all examples or exemplary language (e.g., "such as") provided herein is for the purpose of better illustrating the embodiments of the present invention and does not limit the scope of the present invention. No language in the specification should be construed as indicating that any non-claimed element is essential for the practice of the present invention.

[0519] After reading this disclosure, those of ordinary skill in the art can conceive of further embodiments. In other embodiments, combinations or sub-combinations of the inventions disclosed above may be advantageously made. Exemplary arrangements of components are shown for purposes of illustration, and it should be understood that combinations, additions, rearrangements, etc. may be envisioned in alternative embodiments of the present invention. Thus, while the present invention has been described with respect to exemplary embodiments, those skilled in the art will recognize that many modifications are possible.

[0520] For example, the processes described herein may be implemented using hardware components, software components, and / or any combination thereof. Accordingly, the specification and drawings are to be regarded as illustrative rather than restrictive. However, it is apparent that various modifications and changes can be made without departing from the broader spirit and scope of the invention as set forth in the claims, and the invention is intended to cover all such modifications and equivalents within the scope of the following claims.

[0521] All references cited herein (including publications, patent applications, and patents) are hereby incorporated by reference to the extent as if each reference was individually and specifically indicated to be incorporated by reference and set forth in full herein.

Claims

1. An automated drug administration and dispensing system, the automated drug administration and dispensing system comprises: a sensor that detects movement and other physical inputs related to a user of the automated drug administration and dispensing system; an object information retrieval subsystem that can be activated to automatically obtain information about an object with which the user interacts, the object information retrieval subsystem including a wireless tag reader; a computer-readable storage medium that includes program code instructions; and a processor, wherein the program code instructions can be configured to cause the processor to execute a method including the following steps: determine the occurrence of a posture-based body behavior event of the user from sensor readings obtained from the sensor; in response to determining the occurrence of the posture-based body behavior event, activate and operate the wireless tag reader of the object information retrieval subsystem; obtain information about the at least one item via transmission on a wireless link between the wireless tag reader and at least one wireless tag of at least one item associated with a recent user interaction; and in response to the determination and based on the information of the at least one item obtained, adjust the drug dose, drug dispensing parameters, or both the drug dose and drug dispensing parameters of the automated drug administration and dispensing system.

2. The automated drug administration and dispensing system according to claim 1, wherein at least one of the sensor readings measures movement of a body part of the user.

3. The automated drug administration and dispensing system according to claim 1, the automated drug administration and dispensing system further comprising an event detection module that determines the posture of the user from the sensor readings.

4. The automated drug administration and dispensing system according to claim 1, wherein the method further comprises the step of sending a message to the user, wherein the message is related to the adjustment.

5. The automated drug administration and dispensing system according to claim 1, wherein the posture-based body behavior event corresponds to a user activity unrelated to a food intake event.

6. The automated drug administration and dispensing system according to claim 5, wherein the user activity unrelated to the food intake event includes a smoking event, a personal hygiene event, and / or a drug-related event.

7. The automated drug administration and dispensing system according to claim 1, wherein the posture-based body behavior event corresponds to a food intake event.

8. The automated drug administration and dispensing system according to claim 1, wherein the adjustment is performed when the actual, possible, or imminent start of the posture-based body behavior event is detected.

9. The automated drug administration and dispensing system according to claim 1, wherein the adjustment is based on the characteristics of the posture-based body behavior event.

10. The automated drug administration and dispensing system according to claim 9, wherein: the posture-based body behavior event corresponds to a food intake event; and The adjustment is based on at least one of the following characteristics of the food intake event: duration; speed; start time; end time; number of bites; number of sucks; eating method; type of tableware used; type of container used; amount of chewing before swallowing; chewing speed; amount of food consumed; amount of carbohydrates consumed; time between bites; time between sucks; composition of the food consumed.

11. The automated drug administration and dispensing system according to claim 1, wherein: the drug managed by the automated drug administration and dispensing system is insulin; and the adjustment step calculates the dose of insulin to be administered and the delivery schedule of the calculated dose of insulin.

12. The automated drug administration and dispensing system according to claim 1, wherein the sensor includes an accelerometer that measures the movement of the user's arm and a gyroscope that measures the rotation of the user's arm.

13. A method of operating an automated drug administration and dispensing system, the automated drug administration and dispensing system having a sensor for detecting movement and other physical inputs related to a user, a drug dispensing unit, and an object information retrieval subsystem capable of being activated to automatically obtain information about an object with which the user is interacting, the object information retrieval subsystem including a wireless tag reader, the method comprising the steps of: using a processor of the automated drug administration and dispensing system to obtain a set of sensor readings, wherein at least one of the sensor readings in the set of sensor readings measures the movement of a body part of the user; determining the occurrence of a posture-based body behavior event of the user from the set of sensor readings; in response to determining the occurrence of the posture-based body behavior event, activating and operating the wireless tag reader of the object information retrieval subsystem; obtaining information about the at least one item via transmission over a wireless link between the wireless tag reader and at least one wireless tag of at least one item associated with a recent user interaction; and in response to the determination and based on the information obtained about the at least one item, adjusting the drug dose, drug dispensing parameters, or both the drug dose and drug dispensing parameters of the automated drug administration and dispensing system.

14. The method according to claim 13, the method further comprising the step of performing a computer-based action in response to the determination, wherein the computer-based action is one or more of the following: obtaining other information to be stored in a memory associated with data representing the posture-based body behavior event; interacting with the user to provide information or a reminder; interacting with the user to prompt the user for input; sending a message to a remote computer system; sending a message to another person; sending a message to the user.

15. The method according to claim 13, wherein the posture-based body behavior event corresponds to a user activity that is not related to a food intake event.

16. The method according to claim 15, wherein the user activities unrelated to the food intake event include smoking events, personal hygiene events, and / or drug-related events.

17. The method according to claim 13, wherein the posture-based physical behavior event corresponds to a food intake event.

18. The method according to claim 13, wherein the adjustment is performed when the actual, possible, or imminent start of the posture-based physical behavior event is detected.

19. The method according to claim 13, wherein the adjustment is based on the characteristics of the posture-based physical behavior event.

20. The method according to claim 19, wherein: the posture-based physical behavior event corresponds to a food intake event; and the adjustment is based on at least one of the following characteristics of the food intake event: duration; speed; start time; end time; number of bites; number of sucks; eating method; type of tableware used; type of container used; amount of chewing before swallowing; chewing speed; amount of food consumed; time between bites; time between sucks; composition of the food consumed.

21. An automated drug administration and dispensing system, the automated drug administration and dispensing system comprising: a sensor that detects movements related to a user of the automated drug administration and dispensing system; a drug dispensing unit that is capable of being controlled to dispense a drug to the user; an object information retrieval subsystem that is capable of being activated to automatically obtain information about an object with which the user is interacting, the object information retrieval subsystem including a wireless tag reader; at least one computer-readable storage medium that includes program code instructions; and at least one processor, wherein the program code instructions are configurable to cause the at least one processor to perform a method including the following steps: determine the start or expected start of the user's current food intake event from sensor readings obtained from the sensor; in response to determining the start or expected start of the current food intake event, activate and operate the wireless tag reader of the object information retrieval subsystem; obtain information about the at least one consumable item via transmission on a wireless link between the wireless tag reader and at least one wireless tag of at least one consumable item associated with a recent user interaction; view historical data collected for the user's previously recorded food intake events, wherein the viewing takes into account the information obtained about the consumable item; identify a correlation between the current food intake event and the plurality of previously recorded food intake events; adjust at least one of a drug dose for the drug, a drug dispensing parameter for the drug, or both a drug dose for the drug and a drug dispensing parameter for the drug based on the identified correlation; and dispense the drug by the drug dispensing unit in response to the adjustment.

22. The automated drug administration and dispensing system according to claim 21, wherein at least one of the sensor readings measures movement of a body part of the user.

23. The automated drug administration and dispensing system according to claim 21, the automated drug administration and dispensing system further comprising an event detection module that determines a body behavior event of the user from the sensor readings.

24. The automated drug administration and dispensing system according to claim 23, wherein the event detection module determines a posture of the user that characterizes the current food intake event.

25. The automated drug administration and dispensing system according to claim 21, wherein the adjustment is based on at least one of the following characteristics of the food intake event: duration; speed; start time; end time; number of bites; number of sucks; eating method; type of tableware used; type of container used; amount of chewing before swallowing; chewing speed; amount of food consumed; time between bites; time between sucks; composition of the food consumed.

26. The automated drug administration and dispensing system according to claim 21, wherein: the drug managed by the automated drug administration and dispensing system is insulin; and the adjustment step calculates a dose of insulin to be administered and a delivery schedule of the calculated dose of insulin.

27. The automated drug administration and dispensing system according to claim 21, wherein the sensor includes an accelerometer that measures movement of the user's arm and a gyroscope that measures rotation of the user's arm.

28. The automated drug administration and dispensing system according to claim 21, wherein the historical data includes parameters not directly related to the food intake event.

29. The automated drug administration and dispensing system according to claim 28, wherein the parameters include at least one of the following: location information; time of day the user wakes up; stress level; sleep behavior pattern; calendar event details; phone call information; email metadata.

30. A method of operating an automated drug administration and dispensing system having a sensor that detects movement related to a user, a drug dispensing unit, and an object information retrieval subsystem that can be activated to automatically obtain information about an object with which the user interacts, the object information retrieval subsystem including a wireless tag reader, the method comprising the steps of: determining a start or expected start of a current food intake event of the user from sensor readings obtained from the sensor; in response to determining a start or expected start of a current food intake event, activating and operating the wireless tag reader of the object information retrieval subsystem; obtaining information about the at least one consumable item via transmission over a wireless link between the wireless tag reader and at least one wireless tag of at least one consumable item associated with a recent user interaction; View historical data collected for previous recorded food intake events of the user, where the viewing takes into account the obtained information about the consumable item; Identify the correlation between the current food intake event and multiple of the previously recorded food intake events; Adjust, based on the identified correlation, either the drug dose for the drug, the drug dispensing parameter for the drug, or both the drug dose for the drug and the drug dispensing parameter for the drug; and Dispense the drug by the drug dispensing unit in response to the adjustment.

31. The method according to claim 30, wherein at least one of the sensor readings measures the movement of a body part of the user.

32. The method according to claim 30, the method further comprising the step of determining a body behavior event of the user from the sensor readings.

33. The method according to claim 32, wherein the body behavior event determined from the sensor readings includes the posture of the user characterizing the current food intake event.

34. The method according to claim 30, wherein the adjustment is based on at least one of the following characteristics of the food intake event: duration; speed; start time; end time; number of bites; number of sucks; eating method; type of tableware used; type of container used; amount of chewing before swallowing; chewing speed; amount of food consumed; time between bites; time between sucks; composition of the food consumed.

35. The method according to claim 30, wherein: the drug managed by the automated drug administration and dispensing system is insulin; and the adjustment step calculates the dose of insulin to be administered and the delivery schedule of the calculated dose of insulin.

36. The method according to claim 30, wherein the sensor includes an accelerometer that measures the movement of the user's arm and a gyroscope that measures the rotation of the user's arm.

37. The method according to claim 30, wherein the historical data includes parameters not directly related to the food intake event.

38. The method according to claim 37, wherein the parameters include at least one of the following: location information; time of day the user wakes up; stress level; sleep behavior pattern; calendar event details; phone call information; email metadata.

39. The method according to claim 30, wherein the adjustment is performed when the actual or imminent start of the current food intake event is detected.

40. The method according to claim 30, wherein the adjustment is based on the characteristics of the current posture-based body behavior event.

Images