Methods and apparatus for tracking food intake and other behaviors and providing relevant feedback

Abstract

The present application provides a method for sensing the wearer's activities. A sensing device monitors and tracks food intake events and details. A suitably programmed processor controls various aspects of the sensing device to capture data, store data, analyze data, and provide appropriate feedback related to food intake. More generally, the method may include detection, identification, analysis, quantification, tracking, processing, and / or influence related to food intake, eating habits, eating patterns, and / or food intake events, triggering matters of eating habits or eating patterns. The goal of feedback may be to influence food intake, eating habits or eating patterns, and / or triggering matters of these aspects. The sensing device can also be used to track and provide feedback beyond food-related behavior, and more generally, track behavioral events, detect behavioral event triggers and behavioral event patterns, and provide appropriate feedback.

Description

[0001] This application is a divisional application of an application with an application date of January 30, 2017, an international application number of PCT / US2017 / 015682, a Chinese application number of "201780013019.1", and an invention name of "Method and device for tracking food intake and other behaviors and providing related feedback". Technical Field

[0002] The present invention relates generally to electronic devices related to health technology, and more specifically to methods and apparatus for tracking an individual's food intake using sensors, processors for analyzing the food intake process, and electronic circuits for providing feedback to the individual. The methods and apparatus may relate to more than just food intake.

[0003] Cross-reference to priority and related applications

[0004] This application claims priority to, and is a non-provisional to, U.S. Provisional Patent Application No. 62 / 288,408, entitled “Method and Apparatus for Food Intake Tracking and Feedback,” filed on January 28, 2016. The entire disclosure of the above application is incorporated herein by reference for all purposes as if fully set forth in this document. Background Art

[0005] Diet-related health problems have become one of the most prominent global public health issues. Over the past few decades, obesity and other diet-related health problems have emerged dramatically. According to the Centers for Disease Control (CDC), in 2011-2012, 69% of all American adults aged 20 and over were overweight, and more than one-third of American adults were obese. Obesity can lead to many health problems, such as cardiovascular disease, type 2 diabetes, high blood pressure, cancer, respiratory problems, gallbladder disease, and reproductive complications. Although there may be multiple factors that cause or contribute to obesity, a key factor is personal behavior because personal behavior is related to food intake.

[0006] Over the years, several attempts have been made to track food and nutritional intake. One common way for individuals to track their food intake is to keep a diary. There are several problems with this approach. First, the accuracy of the information a person enters is often limited. Second, keeping a diary is cumbersome and time consuming, causing many users to give up after a short period of time. Third, there is no real-time feedback mechanism. Fourth, they do not provide any insight into important aspects of eating behavior, such as eating speed.

[0007] Recently, software that is typically installed on or accessed from a tablet, mobile phone, laptop, or computer can be used to facilitate the recording and tracking of an individual's food intake. Such software applications typically utilize databases containing nutritional and calorie information for a large number of food items. Unfortunately, the devices and software used to facilitate food log recording are often cumbersome to use and require a lot of human intervention, such as manual data entry or lookup. In addition, they mainly focus on food intake content and portion tracking, and do not provide insights into other eating behavior aspects such as the number of bites or eating speed. They also cannot provide real-time feedback on eating habits or behaviors.

[0008] There are devices and methods that attempt to reduce the burden of manual data entry or data lookup, and they provide another way to obtain log data about food consumption. For example, tableware and utensils with built-in sensors have been proposed to track food intake more automatically. For example, a plate with integrated sensors and circuits can automatically quantify and track the food content placed on the plate. Similarly, an integrated sensor in a drinking container can identify, quantify and track the content of the liquid in the cup. In another example, an eating vessel includes a sensor that counts the number of bites of an individual using the eating vessel. These methods may not have the ability to automatically identify and quantify the food content consumed, and may also only be applicable to a limited set of dining situations and dining occasions, and are not very suitable for properly covering a wide range of different dining situations and dining occasions that an average person may encounter in a day.

[0009] For seamless and comprehensive food intake tracking, it is important to be able to handle a wide variety of dining situations and occasions. Eating utensils-based approaches may not be able to properly track the intake of drinks, snacks, or small meals, and such approaches may also interfere with an individual's normal social behavior. For example, it may not be socially acceptable for a user to bring their own eating utensils to a restaurant or a friend's house for a meal.

[0010] Devices and methods have been described for quantifying and tracking food intake based on analysis of food images acquired by a portable device with imaging capabilities, such as an application running on a mobile phone or tablet with a camera. Some devices may use spectroscopy to identify food items based on their molecular composition. Such devices may use crowd sourcing and / or computer vision techniques, sometimes supplemented with other image processing techniques, to identify food items, estimate their nutritional content, and / or estimate their portion size. However, many of these devices and methods have been found to be not readily usable and available in certain social settings.

[0011] Although today's spectroscopy technology has been miniaturized enough to be included in portable devices, spectroscopy-based devices do have several significant problems. First, such devices require a lot of human intervention and cannot be easily used in an unobtrusive manner. In order to produce accurate spectrogram measurements, a person eating needs to keep the spectrometer close to or in contact with each food item that needs to be identified for a few seconds. Since the light produced by such portable spectrometers can only penetrate the food up to a few centimeters deep, multiple measurements are required for food items that do not have a uniform composition, and therefore portable spectrometers do not work well for sandwiches, layered cakes, mixed salads, etc. Such human intervention hinders the dining experience and may be unacceptable in many dining occasions.

[0012] There is a need for improved methods and apparatus for food intake monitoring and analysis. Summary of the invention

[0013] A sensing device monitors and tracks food intake events and details. A suitably programmed processor controls various aspects of the sensing device to capture data, store data, analyze data, and provide appropriate feedback related to food intake. More generally, the method may include detection, identification, analysis, quantification, tracking, processing, and / or influence related to food intake, eating habits, eating patterns, and / or food intake events, triggering matters of eating habits or eating patterns. The goal of feedback may be to influence food intake, eating habits or eating patterns, and / or triggering matters of these aspects. The sensing device can also be used to track and provide feedback beyond food-related behavior, and more generally, track behavioral events, detect behavioral event triggers and behavioral event patterns, and provide appropriate feedback.

[0014] The following detailed description together with the attached Figure 1 Together they will provide a better understanding of the nature and advantages of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Various embodiments according to the present disclosure will be described with reference to the accompanying drawings, in which:

[0016] Figure 1 is an illustrative example of an environment in accordance with at least one embodiment.

[0017] Figure 2 are illustrative examples of block diagrams in which various embodiments may be implemented.

[0018] Figure 3 is an illustrative example of an environment in accordance with at least one embodiment.

[0019] Figure 4 is an illustrative example of an environment that includes communicating with at least one additional device over the Internet in accordance with at least one embodiment.

[0020] Figure 5 is an illustrative example of an environment in which a food intake monitoring and tracking device communicates directly with a base station or access point in accordance with at least one embodiment.

[0021] Figure 6 is an illustrative example of a high-level block diagram of a monitoring and tracking device in accordance with at least one embodiment.

[0022] Figure 7 is an illustrative example of a block diagram of a monitoring and tracking device in accordance with at least one embodiment.

[0023] Figure 8 An example of a machine classification system in accordance with at least one embodiment of the present disclosure is shown.

[0024] Fig. 9 An example of a machine classification training subsystem in accordance with at least one embodiment of the present disclosure is shown.

[0025] Fig.10 An example of a machine classification detector subsystem is shown in accordance with at least one embodiment of the present disclosure.

[0026] Fig.11 An example of a machine classification training subsystem using non-temporal data as well as other data is shown.

[0027] Fig.12 An example of a machine classification detector subsystem using non-temporal data as well as other data is shown.

[0028] Fig.13 An example of a training subsystem for an unsupervised classification system according to at least one embodiment of the present disclosure is shown.

[0029] Fig.14 An example of a detector subsystem for an unsupervised classification system is shown in accordance with at least one embodiment of the present disclosure.

[0030] Fig.15 An example of a classifier ensemble system is shown.

[0031] Fig.16 An example of a machine classification system including a cross-correlation analysis subsystem is shown. DETAILED DESCRIPTION

[0032] In the following description, various embodiments will be described. For the purpose of explanation, specific configurations and details are set forth in order to provide a thorough understanding of the embodiments. However, it should also be clear to those skilled in the art that the embodiments may be practiced without the specific details. In addition, well-known features may be omitted or simplified in order not to confuse the described embodiments.

[0033] Various examples of devices are provided herein, and for various purposes, such as providing dietary information and feedback, individuals will use the device to monitor, track, analyze and provide feedback on food intake, intake process and timing, and other related aspects of personal eating, drinking and other consumption. Data related to the food intake process may include: timing of the eating process, eating speed, time since the last food intake event, what to eat, estimated values ​​of the food ingredients eaten, etc. Although a large number of examples described herein relate to food intake events, the methods and devices described herein are also applicable to other behavioral events, such as brushing teeth, smoking, biting nails, etc. Data can be obtained from some fixed devices with sensors and electronic parts, some mobile devices with sensors and electronic parts that individuals are easy to move and carry around, and / or from wearable devices with sensors and electronic parts attached to themselves or clothing or as part of personal clothing. In general, such devices are referred to as sensing devices in this article. Herein, individuals with such devices and whose consumption is monitored are referred to as users, but it should be understood that the device can be used unchanged in situations where personal consumption, personal monitoring and personal assessment feedback do not have to be all the same person. Herein, what is consumed is referred to as food intake, but it should be clear that these devices can be used to track consumption and consumption patterns more generally. The behavior tracking / feedback system as described herein may include one or more wearable devices, and may also include one or more additional devices that are not worn. These additional devices may be carried by the wearer or kept nearby so that the additional devices can communicate with the wearable device. The behavior tracking / feedback system may also include remote elements, such as remote cloud computing elements and / or remote storage devices for user information.

[0034] The wearable device can be worn at different locations on the wearer (i.e., the individual whose behavior is being monitored), and the wearable device can be programmed or configured to account for those differences and differences from wearer to wearer. For example, a right-handed person can wear the device on his or her right wrist, while a left-handed person can wear the device on his or her left wrist. Users may also have different orientation preferences. For example, some users may wish to control buttons on one side, while other users may prefer to control buttons on the opposite side. In one embodiment, the user can manually input wrist preferences and / or device orientation.

[0035] In another embodiment, wrist preference and / or device orientation may be determined by asking the user to perform one or more predefined gestures and monitoring sensor data from the wearable device corresponding to the user performing the predefined gestures or gesture sets. For example, the user may be asked to move his hand toward his mouth. Changes in accelerometer sensor readings across one or more axes may then be used to determine wrist and device orientation. In yet another example, a behavior tracking / feedback system may process sensor readings from a wearable device while the user wears the device for a certain duration. Optionally, the behavior tracking / feedback system may further combine sensor readings with other data or metadata about the wearer to infer wrist and device orientation. For example, a behavior tracking / feedback system may monitor a user for a day and record accelerometer sensor readings across one or more axes.

[0036] Because the movement of the lower arm is constrained by the elbow and upper arm, some accelerometer readings will be more frequent than others based on the wrist and device orientation. The accelerometer information can then be used to determine the wrist and / or device orientation. For example, the average, minimum, maximum, and / or standard deviation of the accelerometer readings can be used to determine the wrist and / or device orientation.

[0037] In some embodiments, the sensing device may sense the start / end of a food intake event, eating speed, drinking speed, number of bites, number of sips, fluid intake estimates, and / or portion size estimates without user interaction. Operation with little human intervention, no human intervention, or only intervention that is not obvious to other operations will allow the device to adjust to different dining situations and different social situations. Sensing may include capturing details of food before it is consumed, as well as known user actions that accompany eating, such as repeated rotation of the upper arm or other hand-to-mouth movements. Sensors may include accelerometers, gyroscopes, cameras, and other sensors.

[0038] The use of the device can provide individuals with low usage conflict to detect, quantify, track and provide feedback related to personal food intake content and personal food intake behavior. Such methods are likely to prevent, treat and even cure diet-related diseases in some cases. Such devices can improve efficacy, accuracy and compliance, and reduce the burden of use and increase social acceptance. The device can operate autonomously without human intervention or minimal human intervention, and does not interfere with personal normal activities or social interactions or infringe personal privacy in an invasive or other significant negative way. The device can cope with a wide range of dining situations and dining occasions in an unobtrusive and socially acceptable manner, and can estimate and track food intake content and amount and other aspects of eating behavior. The device can provide individuals with real-time and non-real-time feedback on their eating behavior, habits and patterns.

[0039] It is generally known and understood that certain eating behaviors may be related to, triggered by, or otherwise influenced by physiological, psychological, or environmental conditions, such as hunger, stress, sleep, addiction, disease, physical location, social pressure, and exercise. These characteristics may form inputs to processes performed by or for the device.

[0040] The device may be applicable to individuals who are concerned about their diet. For example, people with type 1 diabetes usually undergo insulin therapy, where they administer appropriate insulin doses based on their food intake and other factors. Although the cause of type 1 diabetes may not be directly related to individual feeding behavior, people with type 1 diabetes need to carefully track their food intake in order to manage their insulin therapy. Such patients will also benefit from easier to use and less conspicuous food intake tracking methods. In some embodiments of the sensing device, the sensing device is part of a feedback-driven automated insulin delivery treatment system. Such systems may include continuous monitoring of patient glucose levels, precise insulin delivery systems, and the use of insulin with a faster absorption rate, which will further benefit from information that can be extracted from automated and seamless food intake tracking (e.g., tracking of carbohydrate and sugar intake). The device may also be applicable to health programs, etc.

[0041] Food intake events are generally related to situations, circumstances or actions by which an individual eats, drinks or otherwise takes in an edible substance. Edible substances may include, but are not limited to, solid foods, liquids, soups, drinks, desserts, medicaments, vitamins, medicines, herbal supplements, snacks, pre-prepared foods, raw foods, meal, appetizers, main dishes, desserts, candies, breakfast, sports or energy drinks. Edible substances include, but are not limited to substances that may contain toxins, allergens, viruses, bacteria or other components that may be harmful to an individual or harmful to a group or a subset of a group. For ease of reading herein, food is used as an example of an edible substance, but it should be understood that, unless otherwise indicated, other edible substances may be used to replace food.

[0042] Eating habits and patterns generally relate to how people consume food. Eating habits and patterns may include, but are not limited to: eating or drinking speed, the size of each mouthful, the amount of chewing before swallowing, the speed of chewing, the frequency of food intake events, the amount of food consumed during a food intake event, the position of the body during a food intake event, possible movement of the body or a particular body part during a food intake event, the mental or physical state during a food intake event, and the utensils or other devices used to present, handle or consume food. The eating or drinking speed may be reflected in the time between consecutive bites or sips.

[0043] Triggers generally relate to the reasons behind a food intake event, the amount consumed, and how it is consumed. Triggers for food intake events and eating habits or patterns may include, but are not limited to, hunger, stress, social pressure, fatigue, addiction, discomfort, medical need, physical location, social situation or circumstances, smell, memory, or physical activity. Triggers may occur simultaneously with the food intake event that they trigger. Alternatively, triggers may occur outside of a food intake event window and may occur before or after a food intake event at a time that may or may not be directly related to the time of the food intake event.

[0044] In some embodiments of the sensing device or system, not all features and functions presented in the present disclosure are implemented. For example, some embodiments may only focus on detecting and / or processing and tracking food intake, and are not intended to guide users to modify their food intake or do not track, process or guide eating habits or patterns.

[0045] In many examples herein, the setting context is that an electronic device is provided to a user, either alone or when it communicates with a nearby supporting device that may or may not be worn, and the user wears the electronic device, the supporting device being, for example, a smart phone used to perform operations offloaded by the worn electronic device. In such examples, there is an individual wearing the electronic device, and this person is referred to as the "wearer" in the example, and the system includes the worn device and may include other components that are not worn and are nearby, as well as remote components that are preferably capable of communicating with the worn device. Thus, the wearer wears the electronic device, and the electronic device includes a sensor that senses an environment related to the wearer. The sensing may be about environmental characteristics, physical characteristics, movement, and other sensed signals, as described elsewhere herein.

[0046] In many instances, the functionality of an electronic device may be implemented by hardware circuits or by program instructions or a combination executed by a processor in the electronic device. When a processor is instructed to do something, it may be that the processor does this because it executes instructions read from an instruction memory, wherein the instructions are used to perform this. Although other people may be involved, a common example here is that the wearer of the electronic device uses the electronic device to monitor his or her own actions, such as gestures, behavioral events including a series of gestures, activities, the start of activities or behavioral events, the stop of activities or behavioral events, etc. When describing a processor executing a specific process, it may refer to the part of the process that is completed independently of the worn electronic device in a distributed processing manner. Therefore, the description of the process executed by the processor of the electronic device is not necessarily limited to the processor within the worn electronic device, but may be a processor in a supporting device that communicates with the worn electronic device.

[0047] Figure 1A high-level functional diagram of a dietary tracking and feedback system according to an embodiment of the present invention is shown. The system for dietary tracking and feedback may include, in part, one or more of the following: a food intake event detection subsystem 101, one or more sensors 102, a tracking and processing subsystem 103, a feedback subsystem 106, one or more data storage units 104, and a learning subsystem that can perform non-real-time analysis. In some embodiments, Figure 1 The elements shown are implemented in electronic hardware, while in other embodiments, some elements are implemented in software and executed by a processor. Some functions can share hardware and processor / memory resources, and some functions can be distributed. Functions can be implemented entirely in a sensor device, or functions can be implemented across a sensor device, a processing system (such as a smart phone) with which the sensor device communicates, and / or a server system that processes some functions away from the sensor device. For example, a wearable sensor device can generate measurements and communicate them to a mobile device, which then uploads the measurements to a server via the Internet, which further processes the data. Data or other information can be stored in a suitable format, distributed in multiple locations or centrally stored in the form of records or after a certain level of processing. Data can be stored temporarily or permanently.

[0048] Figure 1 The first component of the system shown in is a food intake event detection subsystem 101. The role of this subsystem is to identify the start and / or end of a food intake event and to communicate the actual, possible or imminent occurrence of the start and / or end of a food intake event to other components in the system.

[0049] In general, the device detects when a food intake event begins or when a food intake event may begin, but as long as the device properly determines such beginning / possible beginning, the device will function adequately to achieve its purpose. For clarity, the detection is referred to as the "perceived beginning" of a food intake event, and when various processes, operations, and elements are to perform some actions or behaviors in conjunction with the beginning of a food intake event, it will be acceptable for those various processes, operations, and elements to treat the perceived beginning as the beginning, even if sometimes the perceived beginning is not actually the beginning of a food intake event.

[0050] In one embodiment, the detection and / or signaling of the occurrence of the perceived start of a food intake event is simultaneous with the perceived start of the food intake event. In another embodiment, this may sometimes occur after the perceived start of a food intake event. In yet another embodiment, this may sometimes occur before the perceived start of a food intake event. It is generally desired that the signal be transmitted proximate to the perceived start of the food intake event. In some embodiments of the present disclosure, it may be beneficial that the detection and / or signaling of the perceived start of a food intake event occurs before the food intake event begins. For example, this may be useful in situations where a message or signal is to be sent to a user, healthcare provider, or caregiver prior to the start of a food intake event as a guidance mechanism to help guide the user's food intake decisions or eating habits.

[0051] In a preferred embodiment of the present disclosure, the detection of the start and / or end of a food intake event by the food intake event detection subsystem 101 occurs autonomously and without any special user intervention. To achieve this, the food intake event detection subsystem may use input 107 from one or more sensors 102. The sensors may include, but are not limited to, accelerometers, gyroscopes, magnetometers, magnetic angular rate and gravity (MARG) sensors, image sensors, cameras, optical sensors, proximity sensors, pressure sensors, odor sensors, gas sensors, glucose sensors, global positioning systems (GPS), and microphones.

[0052] The method of autonomous detection may include but is not limited to detection based on monitoring the movement or position of the body or a specific part of the body, monitoring arm movement, position or posture, monitoring hand movement, position or posture, monitoring finger movement, position or posture, monitoring swallowing pattern, monitoring lip movement, monitoring saliva, monitoring cheek or jaw movement, monitoring biting or grinding, monitoring signals from the mouth, throat and digestive system. The method of detection may include visual, audio or any other type of sensory monitoring of the individual and / or its surroundings. The monitored signal may be generated by a dietary tracking and feedback system. Alternatively, the signal may be generated by a separate system but may be accessed by a dietary tracking and feedback system through an interface. Machine learning and other data analysis techniques may be applied to detect the start or possible start of a food intake event from the monitored input signal.

[0053] In one example, the food intake event detection subsystem 101 may monitor the output of an accelerometer and / or gyroscope sensor to detect a possible biting gesture or a possible sipping gesture. Such gestures may be determined by a gesture processor that uses machine learning to extract gestures from sensor readings. The gesture processor may be part of a processor in the worn device or in another part of the system.

[0054] Gesture detection machine learning techniques as described elsewhere herein may be used to detect a biting gesture or a sipping gesture, but other techniques are possible. The food intake event detection subsystem 101 may further assign a confidence level to a detected biting gesture or a sipping gesture. The confidence level corresponds to the likelihood that the detected gesture is indeed a biting or sipping gesture. The food intake detection system may determine that the start of a food intake event has occurred based on the detection of the gesture and its confidence level without any additional input. For example, the food intake event detection subsystem 101 may decide that the start of a food intake event has occurred when the confidence level of the biting or sipping gesture exceeds a preconfigured threshold.

[0055] Alternatively, when a possible biting or sipping gesture is detected, the food intake event detection subsystem 101 may use additional input to determine that the start or possible start of a food intake event has occurred. In one example, the food intake event detection subsystem 101 may monitor other gestures that are close in time to determine whether the start of a food intake event has occurred. For example, after detecting a possible biting gesture, the food intake event detection subsystem 101 may wait to detect another biting gesture within a certain time window after the detection of the first gesture and / or with a certain confidence level before determining that the start of a food intake event has occurred.

[0056] After such detection, the food intake detection subsystem 101 may place one or more circuits or components in a higher performance mode to further improve the accuracy of gesture detection. In another example, the food intake event detection subsystem 101 may take into account the time of day or the location of the user to determine whether the start or possible start of a food intake event has occurred. The food intake event detection system may use machine learning or other data analysis techniques to improve the accuracy and reliability of its detection capabilities. For example, training data obtained from the user and / or from other users at an earlier time may be used to train a classifier. Training data may be obtained by requesting user confirmation when a possible biting or sipping gesture has been detected. The labeled data record may then be formed and stored in a memory readable by a gesture processor, which includes features related to the gesture and other background features, such as time of day or location. The classifier may then be trained for the labeled data set, which is composed of labeled data records of multiple labeled data record sets, and the trained classifier model may then be used in the food intake event detection system to more accurately detect the start of a food intake event.

[0057] In another embodiment, the food intake detection subsystem may use triggers to autonomously predict the possible onset of a food intake event. Methods for autonomously detecting the possible onset of a food intake event based on triggers may include, but are not limited to: monitoring an individual's sleep patterns, monitoring an individual's level of tension, monitoring an individual's activity level, monitoring an individual's location, monitoring people around an individual, monitoring an individual's vital signs, monitoring an individual's level of hydration, and monitoring an individual's level of fatigue. In some cases, the food intake detection subsystem may monitor one or more specific trigger signals or trigger events over a longer period of time, and in conjunction with the non-real-time analysis and learning subsystem 105, apply machine learning or other data analysis techniques to predict the onset of a possible food intake event.

[0058] For example, without any additional information, it may be extremely difficult to predict when a user will eat breakfast. However, if the system records the time and day of the week that the user wakes up for several days, the system can use this historical pattern to determine the time when the user is likely to eat breakfast. Those records can be determined by the system, possibly with feedback from the user on their accuracy, or those records can be determined by the user and input via the system user interface. The user interface can be the worn device itself or, for example, a smartphone application. Therefore, the system can process the correlations in the historical data to predict the time or time window when the user is most likely to eat breakfast based on the current day of the week and when the user wakes up. Other trigger signals or trigger events can also be used by the non-real-time analysis and learning subsystem 105 to predict the time when the user will eat breakfast.

[0059] In another example, the non-real-time analysis and learning subsystem 105 may record the user's stress level during a certain time period. The stress level may be determined, for example, by monitoring and analyzing the user's heart rate or certain parameters related to the user's heart rate. The stress level may also be determined by analyzing the user's voice. The stress level may also be determined by analyzing the user's messages or electronic communication content. There may also be other methods of determining the stress level. The non-real-time analysis and learning subsystem 105 may additionally record the occurrence of food intake events and certain characteristics of food intake events during the same time period, such as eating speed, amount of food consumed, time intervals between food intake events, etc. It is possible to then predict the probability that the user will start a food intake event in a future time window based on the current stress level by analyzing the historical data of stress level, occurrence of food intake events, and characteristics of food intake events, and by looking at the correlations in the historical data of stress level, occurrence of food intake events, and characteristics of food intake events, or predict what time window in the future the user will most likely start a food intake event. It is possible to also predict the characteristics of the food intake event, such as eating speed or consumption.

[0060] In certain embodiments, the non-real-time analysis and learning subsystem may use historical data from different users or a combination of data from other users and from the wearer, and use similarities between one or more of the different users and the wearer, such as age, gender, medical condition, etc., to predict the likely start of a food intake event for the wearer.

[0061] In other examples, the non-real-time analysis and learning subsystem 105 may use methods similar to those described herein to predict when a user is most likely to relapse into binge eating or most likely to begin snacking.

[0062] Various sensors can be used for such monitoring. The monitored signal can be generated by the dietary tracking and feedback system. Alternatively, the signal can be generated by a separate system but can be accessed by the dietary tracking and feedback system to be processed and / or used as a trigger signal. Machine learning and other data analysis techniques can also be applied to predict some other characteristics of possible ingestion events, such as the type and / or amount of food that is likely to be consumed, the eating speed that the individual will likely have, the satisfaction that the individual will obtain from consuming the food, etc.

[0063] The machine learning process performed as part of gesture recognition can use external data to further refine its decisions. This can be done through non-real-time analysis and learning subsystem processes. The data analysis process can, for example, consider food intake events detected by a food intake detection system based on gesture sensing and a tracking and processing system based on gesture sensing, thereby forming a second layer of machine learning. For example, over a period of time, food intake events and characteristics related to those food intake events, such as eating speed, food consumption, food content, etc., are recorded, while other parameters that are not directly or obviously related to food intake events are also tracked. For example, this can be location information, the time of day when an individual wakes up, the level of tension, certain patterns of personal sleep behavior, calendar event details including time, event location and a list of participants, phone call information including time, duration, phone number, etc., such as email metadata such as time, duration, sender, etc. The data analysis process then identifies patterns and correlations. For example, it can determine the correlation between the number of calendar events during the day and the characteristics of food intake events at night. This can be attributed to the user being more likely to start snacking when returning home, or having a larger and / or more numerous dinner when the number of calendar events for the day exceeds a certain threshold. Through the non-real-time analysis and learning subsystem 105, it becomes possible to predict food intake events and characteristics based on other signals and events that have no obvious relationship to food intake.

[0064] Processing and analysis of one or more sensor inputs and / or one or more images over longer time periods - optionally using machine learning or other data analysis techniques - can also be used to estimate the duration of a food intake event, or can be used to predict that a food intake event is likely or will soon end.

[0065] In another embodiment, some user inputs 108 may be necessary or desirable to properly or more accurately detect the start and / or end of a food intake event. Such user inputs may be provided in addition to external inputs and inputs received from sensor 102. Alternatively, one or more user inputs may be used in place of any sensor input. User inputs may include, but are not limited to, enabling a device, pressing a button, touching or moving a device or a specific portion of a device, taking a picture, issuing a voice command, making a selection on a screen, or entering information using hardware and / or software, which may include, but are not limited to, a keyboard, a touch screen, or voice recognition technology. If one or more user inputs are required, it is important to conceive and implement the user interaction in a manner that minimizes the negative impact on an individual's normal activities or social interactions.

[0066] The food intake event detection subsystem may combine multiple methods to autonomously detect the actual, probable, or imminent start and / or end of a predicted food intake event.

[0067] Another component of the system is a tracking and processing subsystem 103. In a preferred embodiment of the present disclosure, this subsystem is interfaced 109 with the food intake event detection subsystem 101 and is enabled when it receives a signal from the food intake event detection subsystem that an actual, possible or imminent start of an event has been detected, and is disabled when it receives a signal from the food intake event detection subsystem that an actual, possible or imminent end of an event has been detected or after a certain time. Upon detecting the start of a food intake event, the device may trigger the enabling of other sensors or components of the food intake tracking system, and may also trigger the disabling of said other sensors or components upon detecting the end of a food intake event.

[0068] In another embodiment of the present disclosure, the tracking and processing subsystem may be enabled and / or disabled independently of any signal from the food intake detection subsystem. It is also possible to track and / or process certain parameters independently of any signal from the food intake detection subsystem, while tracking and / or processing of other parameters may only begin after receiving a signal from the food intake event detection subsystem.

[0069] The tracking and processing subsystem generally involves collecting data from one or more sensors 102 through interface 110 and processing the data to extract relevant information.

[0070] The sensor input may be the same or similar to the input sent to the food intake event detection subsystem. Alternatively, different and / or additional sensor inputs may be collected. The sensor may include, but is not limited to, an accelerometer, a gyroscope, a magnetometer, an image sensor, a camera, an optical sensor, a proximity sensor, a pressure sensor, an odor sensor, a gas sensor, a global positioning system (GPS) circuit, a microphone, a galvanic skin response sensor, a thermometer, an ambient light sensor, a UV sensor, an electromyographic ("EMG") potential detection electrode, a bioimpedance sensor, a spectrometer, a glucose sensor, a touch screen, or a capacitive sensor. Examples of sensor data include motion data, temperature, heart rate, pulse, galvanic skin response, blood or body chemistry, audio or video recording, and other sensor data depending on the sensor type. The sensor input may be communicated to the processor wirelessly or by wire in analog or digital form through gating and / or clock line interfacing or directly provided.

[0071] The processing methods used by the tracking and processing subsystem may include, but are not limited to, data manipulation, algebraic calculations, geo-tagging, statistical calculations, machine learning, computer vision, speech recognition, pattern recognition, compression, and filtering.

[0072] The collected data may optionally be temporarily or permanently stored in the data storage unit 104. The tracking and processing subsystem 103 may place data or other information in and retrieve data or other information from the data storage unit 104 using its interface 114 to the data storage unit 104.

[0073] In a preferred embodiment of the present disclosure, the collection, processing and tracking of data occurs autonomously and without any special user intervention. The parameters tracked may include, but are not limited to, the following: location, ambient temperature, ambient light, ambient sound, biometric information, activity level, food image capture, food name and description, portion size, fluid intake, calories and nutritional information, mouthful count, bite count, sip count, duration between consecutive bites or sips, and length of food intake events. For each bite or sip, the parameters tracked may also include the duration that the user's hand, arm and / or utensil is near the user's mouth, the duration that the bite or drink contents are in the user's mouth before swallowing. The method may vary based on what sensor data is available.

[0074] In other embodiments of the present disclosure, some user intervention 111 is required or may be desired to achieve, for example, greater accuracy or to enter additional details. User intervention 111 may include, but is not limited to: enabling the device or a specific function of the device, holding the device in place, taking a picture, adding a voice annotation, recording a video, making corrections or adjustments, providing feedback, making data inputs, taking measurements of food or food samples. Measurements may include, but are not limited to, for example, non-destructive techniques to obtain one or more spectra of a food item, or chemical methods that may require taking a sample from the food.

[0075] The processing of sensor data and user input by the tracking and processing subsystem 103 typically occurs in real time or near real time. There may be some delays, such as to save power or comply with certain hardware limitations, but in some embodiments, the processing occurs during a food intake event, or in the case of tracking outside of a food intake event, the processing occurs around the time that the sensor or user input has been received.

[0076] In some embodiments or in some cases, there may not be real-time or near real-time access to the processing unit required to perform some or all of the processing. This may be due to power consumption or connection constraints, for example. There may also be other motivations or reasons. In this case, the input and / or partially processed data may be stored locally until access to the processing unit is available at a later time.

[0077] In a particular embodiment of the present disclosure, sensor signals that track the movement of an individual's arm, hand, or wrist may be sent to a tracking and processing subsystem. The tracking and processing subsystem may process and analyze such signals to identify that a mouthful of food or a sip of liquid has been consumed or is likely to have been consumed by the individual. The tracking and processing subsystem may additionally process and analyze such signals to identify and / or quantify other aspects of eating behavior, such as the time interval between bites or sips, the speed of hand-to-mouth movement, etc. The tracking and processing subsystem may additionally process and analyze such signals to identify certain aspects of eating method, such as whether the individual eats with a fork or a spoon, drinks from a glass or a can, or consumes food without using any utensils.

[0078] In a particular example, it may be that the wearer rotates their wrist in one direction when bringing an eating vessel or hand to their mouth when biting, but rotates in another direction when sipping liquid. The amount of wrist rotation of the wearer as they move their wrist to or away from their mouth, and the length of time the wrist remains at a higher rotation angle, may also be different for drinking postures and eating postures. Other metrics may be used to distinguish eating postures from drinking postures, or to distinguish between different eating methods. Combinations of multiple metrics may also be used. Examples of other metrics that can be used to distinguish eating postures from drinking postures or to distinguish different eating methods include, but are not limited to: the change in lateral angle from the start or approximate start of the posture until the time or approximate time the hand reaches the mouth; the change in lateral angle from the time or approximate time the hand is close to the mouth until the end or approximate end of the posture; the variance of accelerometer or gyroscope readings across one or more axes in the time period when the hand is close to the mouth, or in the time period before and after the hand is close to the mouth, or in the time period that is not before and after the hand is close to the mouth but includes the time when the hand is closest to the mouth; the variance of the magnitude of the accelerometer readings in the time period when the hand is close to the mouth, or in the time period before and after the hand is closest to the mouth, or in the time period that is not before and after the hand is closest to the mouth but includes the time when the hand is closest to the mouth; the maximum magnitude of the accelerometer readings in the time period when the hand is close to the mouth, or in the time period before and after the hand is closest to the mouth, or in the time period that is not before and after the hand is closest to the mouth but includes the time when the hand is closest to the mouth. The magnitude of an accelerometer reading may be defined as the square root of the acceleration in each orthogonal direction (eg, sensing acceleration in the x, y, and z directions and calculating SQRT(ax2+ay2+az2)).

[0079] The position of the hand relative to the mouth can be determined, for example, by monitoring the pitch or pitch of the worn device and the wearer's arm starting therefrom. The time corresponding to the peak of the pitch can be used as the moment when the hand is closest to the mouth. The time when the pitch starts to rise can be used, for example, as the start time of the gesture. The time when the pitch stops falling can be used, for example, as the end time of the gesture.

[0080] Other definitions of the closest mouth position, the start of movement, and the end of movement are also possible. For example, the time when the lateral rotation changes direction can be used instead to determine when the arm or hand is closest to the mouth. The time when the lateral rotation stops changing in a certain direction or at a certain speed can be used instead to determine the start time of movement toward the mouth.

[0081] The tracking and processing subsystem may additionally process and analyze such signals to determine an appropriate or preferred time to enable other sensors. In one specific instance, the tracking and processing subsystem may process and analyze such signals to determine an appropriate or preferred time to enable one or more cameras to acquire one or more static or moving images of the food. By utilizing sensors that track arm, hand, finger, or wrist movement and / or the orientation and position of the camera to enable the camera and / or automate the image capture process, the complexity, capabilities, and power consumption of the image capture and image analysis system may be greatly reduced, and in some cases better accuracy may be achieved. This also significantly reduces any privacy violations, as it is now possible to more accurately control the timing of image capture and to coincide with the camera focusing on the food.

[0082] For example, the processor may analyze motion sensor inputs from accelerometers, gyroscopes, magnetometers, etc. to identify the optimal time to enable the camera and capture a picture, and may trigger the camera at the optimal time based on the processor determining when the camera's field of view includes the food to be photographed. In one example, the processor determines the start of an eating event and signals the wearer to capture an image of the food being eaten, and also determines the end of an eating event and also signals the wearer to capture an image of the remaining food or plate, etc. Such images may be processed to determine consumption and / or confirm consumption that has been determined by the processor. In some embodiments, image processing may be used as part of feedback to train machine learning used by the processor.

[0083] In some embodiments, the system may use a sensor that tracks the movement of the wearer's arm or hand, and only enable the camera when the system determines from the movement sensing that the arm or hand is near the mouth. In another example, the system may enable the camera sometime between the start of movement toward the mouth and the time when the arm or hand is closest to the mouth. In yet another example, the system may enable the camera sometime between the time when the arm or hand is closest to the mouth and the end of movement away from the mouth.

[0084] As mentioned above, the position of the hand relative to the mouth can be determined by monitoring the pitch and the rising pitch indicating the start time of the movement towards the mouth and the falling pitch indicating the end time. Other definitions of the closest mouth position, the start of the movement and the end of the movement are also possible.

[0085] The position of the hand relative to the mouth can be determined, for example, by monitoring the pitch or pitch of the worn device and the wearer's arm starting therefrom. The time corresponding to the peak of the pitch can be used as the moment when the hand is closest to the mouth. The time when the pitch starts to rise can be used, for example, as the start time of the gesture. The time when the pitch stops falling can be used, for example, as the end time of the gesture.

[0086] The processing and analysis of sensor signals that track the user's arm, hand, or wrist movements can be combined with other methods such as the proposed food image capture as the food enters the mouth to build in redundancy and improve the robustness of the meal tracking and feedback system. For example, by processing and analyzing the user's arm, hand, or wrist movements, information related to bite counts and bite patterns will still be retained even if the camera is obscured or tampered with.

[0087] One or more of the sensor inputs may be static or streaming images obtained from one or more camera modules. Such images may require some degree of processing and analysis. The processing and analysis methods may include one or more of the following methods, among other methods: compression, deletion, resizing, filtering, image editing, and computer vision techniques to identify objects such as specific foods or dishes or features such as portion sizes.

[0088] In addition to measuring bite counts and sip counts, the processor can analyze details such as pace and duration to determine the size of a bite or sip. Measuring the time the wearer's hand, utensil, or fluid container is close to their mouth can be used to derive "near mouth" time, which in turn is used as input to generate an estimate of the size of a bite or sip. The amount of wrist rotation while sipping can be useful for hydration tracking.

[0089] Measuring the amount of wrist rotation in one or more time periods within the start and end of a gesture can also be used to estimate the amount of eating or drinking a mouthful. For example, the system can measure the amount of wrist rotation from a certain time after the start of the gesture to the time when the arm or hand is closest to the mouth. The time corresponding to the peak of the pitch can be used as the moment when the hand is closest to the mouth. The time when the pitch begins to rise can be used as the start time of the movement toward the mouth, for example. The time when the pitch stops falling can be used as the end time of moving away from the mouth, for example. Other definitions of the closest mouth position, the start of movement, and the end of movement are also possible. For example, the time when the lateral change of direction can be used as the time when the arm or hand is closest to the mouth. The time when the lateral stops changing in a certain direction or at a certain speed can be used as the start time of the movement toward the mouth. One or more static or streaming media images can be analyzed and / or compared by the tracking and processing subsystem for one or more purposes, including but not limited to identifying food items, identifying food contents, identifying or deriving nutritional information, estimating portion sizes, and inferring certain eating behaviors and eating patterns.

[0090] As an example, computer vision techniques, optionally in combination with other image manipulation techniques, can be used to identify food types, specific food items, and / or estimate portion sizes. Alternatively, images can be manually analyzed using Mechanical Turk processing or other crowdsourcing methods. Once food types and / or specific food items have been identified, this information can be used to retrieve nutritional information from one or more food / nutrition databases.

[0091] As another example, information about the speed at which a user eats or drinks may be inferred from analyzing and comparing multiple images captured at different times during the course of a food intake event. As yet another example, images, optionally combined with other sensor inputs, may be used to distinguish between a sit-down meal and a snack or dessert. As yet another example, analysis of one image acquired at the beginning of a food intake event and another image acquired at the end of the food intake event may provide information about the amount of food actually consumed.

[0092] In general, a processor receives sensor data, which is analyzed by the processor, possibly together with previously recorded data and / or metadata about the individual involved in the sensor data sensing. The processor performs calculations such as those described herein to derive a series of sensed gestures. The sensed gesture can be one of the gestures described elsewhere herein, as well as relevant data about the sensed gesture, such as the time of occurrence of the sensed gesture. The processor analyzes the series of sensed gestures to determine the start of a behavioral event, such as the start of an eating event.

[0093] The determination of the start of a feeding event may be based on a series of sensed gestures, but may also be based on the detection of a single event (possibly with a context that is not based on gestures). For example, if the system detects a biting gesture with a fairly high confidence level, the processor may consider this detection of this individual gesture as the start of a feeding event. The processor may also analyze the series of sensed gestures to determine the end of a behavioral event. The determination of the end of a feeding event may also be based on the absence of a detected event. For example, if no biting gestures are detected within a given time period, the processor may consider the feeding event to be over.

[0094] Knowing the start and end of a behavioral event allows the processor to more accurately determine a gesture because the gesture is acquired in the background, and / or the processor can activate additional sensors or place one or more sensors or other components in a higher performance state, such as in the examples described elsewhere herein. Knowing the start and end of a behavioral event also allows power saving, because in some cases, it is possible to place the worn device in a lower power mode outside of certain behavioral events. In addition, aggregating individual gestures into events—possibly combined with previously recorded data about similar behavioral events from the same user or from other users in the past—allows the processor to derive meaningful characteristics about behavioral events. For example, the speed of eating during breakfast, lunch, and dinner can be determined in this way. As another example, if the processor has a state for the current behavior and the current behavior is brushing teeth, a gesture that may look like an eating or drinking gesture will not be interpreted as an eating or drinking gesture, and therefore sipping while brushing teeth will not be interpreted as liquid consumption. Behavioral events can be general events (eating, walking, brushing teeth, etc.) or more specific (eating with a spoon, drinking with a glass, drinking with a can, etc.).

[0095] Although it is possible to decode indirect gestures, such as detecting a pointing gesture and then determining the object to which the sensed individual is pointing, of interest are gestures that are themselves directly part of the detected event. Some gestures are incidental gestures, such as gestures associated with operating a device, in which case consideration of the incidental gestures may be excluded.

[0096] In a specific example, the system uses a certain set of sensors to determine the start of an eating event with a certain confidence level, and if the confidence level is above a threshold, the system enables additional sensors. Thus, the accelerometer sensor can be used to determine the start of an eating event with a high confidence level, but the gyroscope is placed in a low power mode to save battery life. The accelerometer alone can detect gestures that indicate a possible bite or sip (e.g., an arm up or hand moving, or a hand or arm moving generally in the direction of the mouth) or gestures that generally indicate the start of an eating event. After detecting a first gesture that generally indicates the possible start of an eating event, additional sensors (e.g., a gyroscope, etc.) can then be activated. If a subsequent bite or sip gesture is detected, the processor determines that the start of an eating event has occurred with a higher confidence level.

[0097] Knowing the start / end of a behavioral event allows the processor to place one or more sensors or other components in a higher performance state during the behavioral event. For example, when the start of a behavioral event has been determined, the processor can increase the sampling rate of an accelerometer and / or gyroscope sensor used to detect gestures. As another example, when the start of a behavioral event has been determined, the processor can increase the update rate at which sensor data is sent to the electronic device 19 for further processing to reduce latency.

[0098] Reference again Figure 1 , in addition to the tracking and processing subsystems, Figure 1 The system may also include a non-real-time analysis and learning subsystem 105. The non-real-time analysis and learning subsystem may perform analysis on larger data sets collected over a longer period of time, such as historical data spanning multiple food intake events and / or data from a larger population. The methods used by the non-real-time analysis and learning subsystem may include, but are not limited to, data manipulation, algebraic calculations, geo-tagging, statistical calculations, machine learning and data analysis, computer vision, speech recognition, pattern recognition, compression, and filtering.

[0099] The methods used by the non-real-time analysis and learning subsystem 105 may include, among other things, data analysis of larger data sets collected over longer time periods. For example, one or more data inputs may be captured over longer time periods and across multiple food intake events to train a machine learning model. Such data inputs are hereinafter referred to as training data sets. It is generally desirable that the time period, hereinafter referred to as the training period, over which the training data set is collected is sufficiently long so that the collected data represents a typical food intake of an individual.

[0100] The training data set may include, among other things, one or more of the following food intake related information: number of bites per food intake event, total bite count, duration of food intake event, food intake rate or time between consecutive counts, classification of food intake content, such as distinguishing solid foods from liquids, or distinguishing table meals from snacks or canapes. This information may be derived from one or more sensor inputs.

[0101] The training data set may additionally include images of each or most items consumed during each food intake event within the training period. The images may be processed using computer vision and / or other methods to identify food types, specific food items, and estimated portion sizes. This information may then be used to quantify the number of calories and / or macronutrient content of the food items, such as the amount of carbohydrates, fat, protein, etc.

[0102] In the case where the food is not completely consumed, it may be necessary to take a photo of the food item at the beginning of the food intake event and a photo at the end of the food intake event to derive the actual amount of food consumed. Other methods including but not limited to manual user input can be used to add portion size information to the data in the training data set.

[0103] The training data set may additionally include metadata that does not directly quantify food intake and / or eating behavior and patterns, but that may indirectly provide information, may be associated with food intake events and / or eating behavior, and / or may be a trigger for the occurrence of food intake events or may influence eating habits, patterns and behaviors. Such metadata may include, among other things, one or more of: gender, age, weight, socioeconomic status, timing information about food intake events such as date, time of day, day of the week, information about the location of food intake events, vital sign information, hydration level information, and other physiological, psychological or environmental conditions such as hunger, stress, sleep, fatigue level, addiction, disease, social pressure and exercise.

[0104] One or more training data sets may be used to train one or more machine learning models, which may then be used by one or more components of a dietary tracking and feedback system to predict food intake events and certain aspects of eating patterns and behavior.

[0105] In one example, a model may be trained to predict the occurrence of a food intake event based on tracking of one or more metadata that may influence the occurrence of a food intake event. Other characteristics associated with a possible food intake event may also be predicted, such as the type and / or amount of food that is likely to be consumed, the eating speed that an individual will likely have, the duration of the food intake event, and / or the level of satisfaction that an individual will obtain from consuming food, etc. The metadata may include, among other things, one or more of: gender, age, weight, socioeconomic status, timing information about the food intake event such as date, time of day, day of the week, information about the location of the food intake event, vital sign information, hydration level information, and other physiological, psychological, or environmental conditions such as hunger, stress, sleep, fatigue level, addiction, disease, social pressure, and exercise.

[0106] In another example, machine learning and data analysis may be applied to derive metrics that can be used outside of the training period, thereby estimating calorie or other macronutrient intake even if only limited or no food intake sensor input or images are available. Metadata may be used to further adapt the value of such metrics based on additional contextual information. Metadata may include, among other things, one or more of: gender, age, weight, socioeconomic status, timing information about food intake events such as date, time of day, day of week, information about the location of food intake events, information about food categories, vital sign information, hydration level information, calendar event information, phone call logs, email logs, and other physiological, psychological, or environmental conditions such as hunger, stress, sleep, fatigue level, addiction, disease, social pressure, and exercise.

[0107] An example of such a metric may be "calories per bite". A "calories per bite" metric may be established from one or more training data sets by combining bite counts with calorie information obtained from image processing and analysis. This metric may then be used outside of the training period to estimate calorie intake based solely on bite counts, even when no images or only limited images are available.

[0108] Another metric may be "typical bite size". A "typical bite size" metric may be established from one or more training data sets by combining bite counts with portion size information obtained from image processing and analysis. This metric may then be used outside of the training period to estimate portion sizes based solely on bite counts, even when no images or only limited images are available. It may also be used to identify inconsistencies between reported and measured food intake based on bite counts and typical bite sizes. An inconsistency may indicate that a user did not report all of the food items they consumed. Or alternatively, it may indicate that a user did not consume all of the food they reported.

[0109] The bite action can be determined by the processor reading the accelerometer and gyroscope sensors, or more generally by reading the motion sensor that senses the movement of the wearer's body parts. Then, by counting the number of bites, the total number of bites can be inferred. In addition, the time sequence of bites can be used by the processor to deduce the eating pattern.

[0110] The non-real-time analysis and learning subsystem 105 can also be used to track, analyze and help visualize larger historical data sets, track progress toward specific established or configured goals, and help establish such goals. In addition, it can be used to identify and track records, streaks, and compare behavioral performance with that of friends or larger, optionally anonymous groups.

[0111] In addition, in certain embodiments, in addition to other data manipulation and processing techniques, non-real-time analysis and learning subsystem 105 can apply machine learning and data analysis techniques to predict the urgency or possibility of certain health problems, diseases and other medical conditions. In this case, training usually requires historical food intake and / or eating behavior data captured over a long period of time and in a larger group. In addition, it is desirable that the training data set includes additional metadata, such as age, weight, gender geographic information, socioeconomic status, vital signs, medical record information, calendar information, phone call logs, email logs and / or other information. Prediction can then be used to help guide health outcomes and / or prevent or delay the onset of certain diseases, such as diabetes.

[0112] The non-real-time analysis and learning subsystem 105 may also be used to learn and extract more information about other aspects, including but not limited to one or more of the following: user meal and food preferences, user dining preferences, user restaurant preferences, and user food consumption. Such information may be used by the food intake tracking and feedback system to make specific recommendations to the user. The food intake tracking and feedback system described herein may also be interfaced to or integrated with other systems such as restaurant reservation systems, online food or meal ordering systems, etc., to facilitate, simplify, or automate the process of food or meal ordering or reservations.

[0113] The non-real-time analysis and learning subsystem 105 can also be used to monitor food intake over a longer period of time and detect any unusually long events without food intake activity. Such events may indicate, among other things, that the user has stopped using the device, intentionally or unintentionally tampered with the device, a functional defect of the device, or a medical situation such as a user falling or dying or losing consciousness. Detection of unusually long events without food intake activity can be used to send notifications or alerts to the user, one or more of their caregivers, a monitoring system, an emergency response system, or to a third party that may be directly or indirectly interested in receiving notifications of the occurrence of such events.

[0114] Figure 1 Another component of the illustrated system is a feedback subsystem 106. The feedback subsystem 106 provides one or more feedback signals to the user or to any other individual to whom such feedback information may be relevant. The feedback subsystem 106 may provide real-time or near real-time feedback related to a particular food intake event. Real-time or near real-time feedback generally refers to feedback given around the time of the food intake event. This may include feedback given during the food intake event, feedback given before the food intake event begins, and feedback given sometime after the food intake event ends. Alternatively or in addition, the feedback subsystem may provide feedback to a user that is not directly related to a particular food intake event.

[0115] The feedback methods used by the feedback subsystem may include, but are not limited to: tactile feedback, whereby a tactile interface that applies force, vibration, and / or motion to the user is used; audio feedback, whereby a speaker or any other audio interface may be used; or visual feedback, whereby a display, one or more LEDs, and / or projected light patterns may be used. The feedback subsystem may use only one or a combination of more than one feedback methods.

[0116] The feedback subsystem may be implemented in hardware, software, or a combination of hardware and software. The feedback subsystem may be implemented on the same device as the food intake event detection subsystem 101 and / or the tracking and processing subsystem 103. Alternatively, the feedback subsystem may be implemented in a device separate from the food intake event detection subsystem 101 and / or the tracking and processing subsystem 103. The feedback subsystem may also be distributed across multiple devices, some of which may optionally house Figure 1 Some of the other subsystems shown in .

[0117] In one embodiment, the feedback subsystem may provide feedback to the user to signal the start of an actual, possible, or impending food intake event. The feedback subsystem may also provide feedback to the user during the food intake event to remind the user of the fact that the food intake event is occurring, thereby improving present awareness and / or promoting mindful eating. The feedback subsystem may also provide guidance on suggested portion sizes and / or food content or provide alternative suggestions for eating. The alternative suggestions may be default suggestions, or they may be custom suggestions that the user may program or configure at different times.

[0118] The feedback signal may include, but is not limited to, a periodic tactile feedback signal on the wearable device, an audible alarm, a display message, or one or more notifications pushed to the display of its mobile phone.

[0119] After receiving the signal indicating the start of a food intake event or sometime thereafter, the user can confirm that a food intake event did occur. The confirmation can be used, for example, to trigger the recording of the event or can cause the system to prompt the user for additional information.

[0120] In another embodiment of the present disclosure, the feedback subsystem initiates feedback during a food intake event only when a certain threshold of one or more of the tracked parameters is reached. For example, if the time between consecutive bites or sips is tracked, feedback to the user may be initiated if the time (which may be an average of multiple bites or sips) is shorter than a fixed or programmed value to encourage the user to slow down. Similarly, feedback may be initiated if a fixed or programmed bite or sip count is exceeded.

[0121] In a feedback subsystem that provides feedback during a food intake event, the feedback provided by the feedback subsystem is generally related to the details of this particular food intake event. However, other information including but not limited to information about previous food intake events, biometric information, mental health information, activity or health level information, and environmental information may also be provided by the feedback subsystem.

[0122] In yet another embodiment of the present disclosure, the feedback subsystem may send one or more feedback signals external to a specific food intake event. In one example of such an embodiment, ambient temperature and / or other parameters that may affect hydration requirements or otherwise directly or indirectly measure hydration levels may be tracked. Such tracking may occur continuously or periodically, or otherwise be independent of a specific food intake event. If one or more such parameters exceed a fixed or programmed threshold, a feedback signal may be sent, for example, to prompt an individual to take steps to improve hydration. The feedback subsystem may estimate its inputs and determine that the preferred time to send feedback is not during a food intake event, but after the food intake event ends. Some of the inputs to the feedback subsystem may come from food intake events, but some may come from other monitoring that is not directly measured as a result of a food intake event.

[0123] For example, in the embodiment described in the previous paragraph, the decision to send a feedback signal may be independent of any food intake tracking. Alternatively, such a decision may be related to food intake tracking across one or more food intake events. For example, in one embodiment of the present disclosure, the system described above may be modified to also track an individual's fluid intake directly or indirectly. For different ambient temperature ranges, the embodiment may have pre-programmed fluid intake requirement thresholds. If, for a measured ambient temperature, the individual's fluid intake that may be tracked and accumulated over a period of time does not meet the threshold of the ambient temperature, then the system may issue a feedback signal to advise the individual to increase their fluid intake level.

[0124] Similarly, feedback signals or suggestions related to food intake may be related to tracking of activity levels, sleep levels, social situations or circumstances, health or disease diagnosis, and health or disease monitoring, among other parameters.

[0125] In yet another embodiment of the present disclosure, the feedback subsystem may initiate a feedback signal when it has detected that a food intake event has begun or is about to or is likely to occur. In such embodiments, the feedback may be used, for example, as a prompt to remind the user to record food intake events or certain aspects of food intake events that cannot be automatically tracked, or to influence or guide personal food intake behavior and / or the amount or content of food consumed.

[0126] The information provided by the feedback subsystem may include, but is not limited to: information regarding eating patterns or habits; information regarding specific edible substances, such as the name, description, nutrient content, comments, ratings, and / or images of food items or dishes; information regarding triggers for food intake; information regarding triggers for eating patterns or habits; biometric or environmental information; or other information that may be directly or indirectly related to an individual's general food intake behavior, health status, and / or health.

[0127] The feedback subsystem may include a display of images of food items or dishes that have been consumed or may be consumed. In addition, the feedback subsystem may include additional information about the food items or dishes, such as an indication of their healthiness, nutrient content, basic information or preparation details, ratings, personalized feedback, or other personalized information.

[0128] In certain embodiments of the present disclosure, the information provided by the feedback subsystem may include non-real-time information. The feedback subsystem may, for example, include feedback based on the processing and analysis of historical data and / or the processing and analysis of data accumulated in a larger user population. The feedback subsystem may further provide feedback that is not related to the tracking of any specific parameter. For example, the feedback subsystem may provide general food, nutrition, or health information or guidance.

[0129] In certain embodiments of the present disclosure, a user may interact with the feedback subsystem and provide input 116. For example, a user may suppress or customize some or all feedback signals.

[0130] Non-real-time feedback may include, among other things: historical data, tendency overviews, personal records, temperament, behavioral performance toward goals or compared to friends or other people or groups of people, alert tendency notifications, feedback from friends, social networks and social media, caregivers, nutritionists, doctors, etc., guidance and instruction.

[0131] Data or other information may be stored in the data storage unit 104. It may be stored in raw format. Alternatively, the data or other information may be stored after having undergone some degree of processing. Data may be stored temporarily or permanently. Data or other information may be stored for a wide variety of reasons, including but not limited to: temporary storage while waiting for a processor or other system resource to become available; temporary storage to be combined with other data that may not be available until a later time; storage to be fed back to the user via the feedback subsystem in raw or processed format; storage for later consultation or review; storage for analysis for dietary and / or health coaching purposes; storage for statistical analysis across larger groups or on larger data sets; storage to perform pattern recognition methods or machine learning techniques on larger data sets.

[0132] The stored data and information, or portions thereof, may be used by users of the system. It is also possible that the stored data and information, or portions thereof, may be shared with or accessed by third parties. Third parties may include, but are not limited to, friends, family members, caregivers, healthcare providers, nutritionists, health coaches, other users, companies that develop and / or sell systems for dietary tracking and guidance, companies that develop and / or sell components or subsystems of systems for dietary tracking and guidance, and insurance companies. In some cases, it may be necessary to anonymize the data before it is available to third parties.

[0133] Figure 2 Some of the components disposed in an electronic system for meal tracking and coaching according to one embodiment of the present disclosure are shown. The electronic system includes a first electronic device (hereinafter may also be referred to as "electronic device") 218, a second electronic device (hereinafter may also be referred to as "electronic device") 219, and a central processing and storage unit 220. A typical system may have a calibration function to allow sensor and processor calibration.

[0134] Figure 2 Variations of the system shown in are also possible and are within the scope of the present disclosure. For example, in one variation, electronic device 218 and electronic device 219 may be combined into a single electronic device. In another variation, the functionality of electronic device 218 may be distributed across multiple devices. In some variations, the electronic device 218 may be a part of the electronic device 219. Figure 2 A portion of the functionality shown in FIG. 2 may instead be included in the electronic device 219. In some other variations, the electronic device 219 may be included in the electronic device 219. Figure 2 A portion of the functionality shown in 200 may instead be included in electronic device 218 and / or central processing and storage unit 220. In yet another variation, a central processing and storage unit may not be present, and all processing and storage may be done locally on electronic device 218 and / or electronic device 219. Other variations are also possible.

[0135] Figure 2 Examples of electronic systems in Figure 3 . For example, the electronic device 218 can be a wearable device 321 worn on a wrist, arm or finger. The electronic device 218 can also be implemented as a wearable piece that can be attached to the human body or embedded in clothing. The electronic device 218 can also be a module or an additional device that can be attached to other wearable devices, jewelry or clothing, for example. The electronic device 219 can be, for example, a mobile device 322, such as a mobile phone, a tablet computer or a smart watch. There may also be other embodiments of the electronic device 219 and the electronic device 218. The central processing and storage unit 220 typically includes one or more computer systems or servers and one or more storage systems. The central processing and storage unit 220 can be, for example, a remote data center 324 that can be accessed via the Internet using an Internet connection 325. The central processing and storage unit 220 is often shared and / or accessed by multiple users.

[0136] The wearable device 321 can communicate with the mobile device 322 via a wireless network. The wireless protocols used for communication between the wearable device 321 and the mobile device 322 via a wireless network may include, but are not limited to, Bluetooth, Bluetooth Smart (also known as Bluetooth Low Energy), Bluetooth Mesh, ZigBee, Wi-Fi, Wi-Fi Direct, NFC, Cellular, and Thread. Dedicated or wireless protocols, modified versions of standardized wireless protocols, or other standardized wireless protocols may also be used. In another embodiment of the present disclosure, the wearable device 321 and the mobile device 322 may communicate via a wired network.

[0137] The mobile device 322 can communicate wirelessly with a base station or access point ("AP") 323 connected to the Internet via an Internet connection 325. Through the Internet connection 325, the mobile device 322 can transmit data and information from the wearable device 321 to one or more central processing and storage units 220 residing at a remote location, such as a remote data center. Through the Internet connection 325, the mobile device 322 can also transmit data and information from one or more central processing and storage units 220 residing at a remote location to the wearable device 321. Other examples are also possible. In some embodiments, the central processing and storage unit 220 may not be at a remote location, but may reside in the same location as the wearable device 321 and / or the mobile device 322 or close to it. The wireless protocol used for communication between the mobile device 322 and the base station or access point 323 may be the same as the wireless protocol between the mobile device and the wearable device. Dedicated or wireless protocols, modified versions of standardized wireless protocols, or other standardized wireless protocols may also be used.

[0138] Figure 2 The electronic system of the can also send data, information, notifications and / or instructions to and / or receive data, information, notifications and / or instructions from additional devices connected to the Internet. Such devices can be, for example, tablets, mobile phones, laptops or computers of one or more caregivers, members of a doctor's office, coaches, family members, friends, people with whom the user has connected on social media, or other people with whom the user has authorized information to be shared. An example of such a system is Figure 4 In Figure 4In the example shown in , the electronic device 441 is wirelessly connected to a base station or access point 440 connected to the Internet via an Internet connection 442. Examples of the electronic device 441 may include, but are not limited to, a tablet computer, a mobile phone, a laptop computer, a computer, or a smart watch. Through the Internet connection 442, the electronic device 441 may receive data, instructions, notifications, or other information from one or more central processing and storage units, which may reside locally or reside at a remote location such as a remote data center. The communication capability may include an Internet connection 442 or other communication channels. The electronic device 441 may also send information, instructions, or notifications to one or more computer servers or storage units 439. The central processing and storage unit 439 may forward this information, instruction, or notification to the mobile device 436 via the Internet 438 and a base station or access point ("AP") 437.

[0139] Other examples are also possible. In some embodiments, the central processing and storage unit 439 may not be at a remote location, but may reside at the same location as the wearable device 435 and / or the mobile device 436 or in close proximity thereto. Figure 4 An electronic device 441 is shown wirelessly connected to a base station or access point. There may also be a wired connection between the electronic device 441 and a router connected to the internet via an internet connection 442.

[0140] Figure 5 Another embodiment of the present disclosure is shown. Figure 5 In the embodiment of the present invention, the wearable device 543 can exchange data or other information directly with the central processing and storage system 546 through the base station or access point 544 and the Internet without having to go through the mobile device 545. The mobile device 545 can exchange data or other information with the wearable device 543 through the central processing and storage system 546 or through a local wireless or wired network. The central processing and storage system 546 can exchange information with one or more additional electronic devices 550.

[0141] Figure 6 Some of the components disposed in the electronic device 218 are shown according to one embodiment. The electronic device 218 typically includes, in part, one or more sensor units 627, a processing unit 628, a memory 629, a clock or crystal 630, a radio circuit 634, and a power management unit ("PMU") 631. The electronic device 218 may also include one or more camera modules 626, one or more stimulation units 633, and one or more user interfaces 632. Although not shown, other components like capacitors, resistors, inductors, etc. may also be included in the electronic device 218. The power management unit 631 may include one or more of the following, among others: a battery, a charging circuit, a regulator, hardware to disable power to one or more components, a power plug.

[0142] In many embodiments, the electronic device 218 is a size-constrained, power-sensitive, battery-powered device with a simple and limited user interface. Where power is limited, the electronic device 218 may be programmed to conserve power outside of behavioral events. For example, a processor in the electronic device 218 may be programmed to determine the start of a behavioral event, such as an eating event, and then power on additional sensors, place certain sensors in a higher performance mode, and / or perform additional calculations until the processor determines the end of the behavioral event, at which point the processor may shut down the additional sensors, place certain sensors back in a lower performance mode, and omit the additional calculations.

[0143] For example, the processor may be programmed to disable all motion detection related circuits except the accelerometer. The processor may then monitor the accelerometer sensor data, and if those data indicate actual or apparent food intake activity such as a biting or sipping gesture, the processor may enable additional circuits, such as a data logging mechanism. The processor may use the accelerometer sensor data to monitor the pitch of the wearer's arm.

[0144] For example, the processor may measure the pitch of the wearer's arm until the pitch exceeds a certain threshold, perhaps a value indicating that the hand or arm is moving toward the wearer's mouth. Once this value is detected, the processor may change state (e.g., by changing a memory location dedicated to this state from "not in action" or "out of event" to "in action" or "in event") and enable additional circuits or enable a higher performance mode of a particular circuit or component. In another embodiment, other accelerometer sensor data characteristics such as a first acceleration integral (velocity) or a second acceleration integral (distance traveled) as determined from one or more accelerometer axes may be used, or characteristics related to or derived from the first and / or second acceleration integrals. A machine learning process may be used to detect specific movements and convert the movements into gestures.

[0145] The end of a food intake event can be detected by the processor by considering whether a certain time has expired since the last bite or sip movement or when other data (metadata about the wearer, motion detection sensor data and / or historical data of the wearer, or a combination of these data) Based on that data, the processor determines that a food intake event is unlikely and then changes the state of the electronic device to an inactive monitoring state, possibly a lower power mode.

[0146] The lower power mode may be implemented by the processor reducing the sampling rate of the accelerometer and / or gyroscope, powering off the gyroscope, reducing the update rate at which sensor data is transmitted from an electronic device (e.g., electronic device 218) to a supporting device (e.g., electronic device 219), and compressing data before transmitting the data from the sensing electronic device to the supporting electronic device.

[0147] In some embodiments of the present disclosure, Figure 5 Some of the components shown as separate components in the can be combined. For example, the processing unit, memory, radio circuitry, and PMU functionality can be combined in whole or in part in a single wireless microcontroller unit ("MCU"). Other combinations are also possible. Similarly, in Figure 5 Components shown as single components in the example may be implemented as multiple components. For example, processing functions may be distributed across multiple processors. Similarly, data storage functions may be distributed across multiple memory components. Other examples of distributed implementations are also possible.

[0148] In another embodiment of the present disclosure, radio circuitry may not be present, and instead a different interface (eg, a USB interface and cable) may be used to transfer data or information to and / or from the electronic device 218 .

[0149] The stimulation unit 633 may provide feedback to the user of the electronic device. The stimulation unit 633 may include, but is not limited to, a tactile interface that applies force, vibration, or motion to the user, a speaker or headphone interface that provides sound to the user, and a display that provides visual feedback to the user.

[0150] In certain embodiments, processing and analysis of signals from embedded electronic device 218 can detect when the electronic device has been disabled, tampered with, removed from the body, or not being used. This can be used to save power, or to send notifications to the user, friends, or others who may be directly or indirectly interested in being informed if the electronic device 218 is not being used properly.

[0151] Descriptive detection / prediction of food intake event start / end

[0152] In a preferred embodiment, the electronic device 218 is worn on a wrist, arm, or finger and has one or more sensors that generate the data necessary to detect the start and / or end of a food intake event. The electronic device 218 may also be integrated into a small piece that can be attached to an individual's arm or wrist. The electronic device 218 may also be a module or add-on device that can be attached to another device worn on a wrist, arm, or finger. The sensors used to detect the start and / or end of a food intake event may include one or more of the sensors described herein as well as other sensors.

[0153] The raw sensor outputs may be stored locally in the memory 629 and processed locally on the processing unit 628 to detect whether the start or end of a food intake event has occurred. Alternatively, one or more sensor outputs may be sent to the electronic device 219 and / or the central processing and storage unit 220 in a raw or processed format for further processing and detecting whether the start or end of a food intake event has occurred. Regardless of where the processing of food intake detection occurs, the sensor outputs in a raw or processed format may be stored within the electronic device 218, within the electronic device 219, and / or within the central processing and storage unit 220.

[0154] The sensor or sensors that generate the data necessary to detect the start and / or end of a food intake event may be internal to the electronic device 218. Alternatively, one or more of the sensors responsible for detecting the start of a food intake event may be external to the electronic device 218, but capable of relaying relevant information to the electronic device 218, either directly through direct wireless or wired communication with the electronic device 218 or indirectly through another device. It is also possible that the electronic device 218 and the external sensor or sensors are capable of relaying information to the electronic device 219, but are not capable of directly relaying information to each other.

[0155] In the case of indirect communication through another device, such as a mobile phone or another portable or fixed device, such a third device is capable of receiving data or information from one or external sensor units, optionally processing such data or information, and forwarding raw or processed data or information to the electronic device 218. Communications to and from the electronic device 218 may be wired or wireless, or a combination of both.

[0156] Examples of sensors that may be external to the electronic device 218 may be one or more sensors embedded in a necklace or pendant worn around the neck, one or more sensors embedded in a small patch attached to different locations on the body, one or more sensors embedded in an auxiliary second wearable device worn on the other arm or wrist or on the fingers of the other hand, or one or more sensors integrated into teeth. In some embodiments, the electronic device is worn on one hand or arm but detects movement of the other hand or arm. In some embodiments, the electronic device is worn on each hand.

[0157] Information obtained from the non-real-time analysis and learning subsystem 105 may also be used, optionally in conjunction with information from one or more sensors 627, to predict or facilitate detection of the start / end of a possible, impending, or actual food intake event.

[0158] It is generally desirable that the detection and / or prediction of the start and / or end of a food intake event occurs autonomously without user intervention. For example, if the actual, possible or imminent start of a food intake event is predicted or detected autonomously, this information can be used as a trigger for enabling or powering on specific components or circuits that are only needed during the food intake event. This can help save power and extend the battery life of the electronic device 218. The prediction or detection of the actual, possible or imminent start of a food intake event can also be used to issue prompts or reminders to the user. For example, a prompt can be sent to the user to remind him / her to take further measures, including but not limited to recording the food intake event or taking a photo of the food. After the start of the food intake event is detected, one or more prompts may be propagated during the food intake event to remind the user that the food intake event is occurring, thereby improving the perception of the present and / or promoting mindful eating. Prompts or reminders can be sent, for example, using one or more stimulation units 633 through discrete tactile feedback. Other methods using one or more user interfaces 632, such as one or more LEDs, display messages, or audio signals, are also possible. Alternatively, the mobile device may be used to communicate prompts, reminders, or other information to the user, such as portion size suggestions or meal substitution suggestions.

[0159] If the actual, possible, or imminent end of a food intake event is predicted or detected autonomously, this information can be used as a trigger to power down or at least place one or more circuits or components of the electronic device 218 that are only needed during the food intake event. This can help conserve power and extend the battery life of the electronic device 218. Detection of the actual, possible, or imminent end of a food intake event can also be used to modify or pause feedback provided to the user through one or more stimulation units 633, through one or more user interfaces 632, and / or through a mobile device.

[0160] In some embodiments of the present disclosure, the detection or prediction of the actual, possible or imminent start and / or end of a food intake event may not be completely autonomous. For example, a user may be required to make a specific arm, wrist, hand or finger gesture to signal the actual, possible or imminent start and / or end of a food intake event to the electronic device 218. The arm, wrist, hand or finger gesture is then detected by one or more sensors inside the electronic device 218. It is generally desirable that the arm, wrist, hand or finger gesture or gestures required to indicate the start and / or end of a food intake event can be performed in a small and discontinuous manner. Other methods can also be used. For example, a user may be required to push a button on the electronic device 218 to indicate the start and / or end of a food intake event. Voice activated commands using a microphone built into the electronic device 18 can also be used. Other methods are also possible.

[0161] Description of tracking eating behaviors and patterns

[0162] In certain embodiments, the electronic device 218 is worn on a wrist, arm, or finger and has one or more sensors that generate data that facilitates measurement and analysis of eating behaviors, patterns, and habits. Sensors for measuring and analyzing certain eating behaviors and patterns may include one or more of the sensors described herein.

[0163] Relevant metrics that can be used to quantify and track eating behavior and eating patterns may include, but are not limited to: the time between consecutive bites or sips; the distance between the plate and the user's mouth; the speed at which the arm moves toward and / or away from the user's mouth; and the number of bites or sips during a single food intake event, which is derived from the arm movements corresponding to bites or sips, specific chewing behaviors and characteristics, the time between a bite and swallowing, and the total count of the amount chewed before swallowing.

[0164] The raw sensor outputs may be stored locally in the memory 29 and processed locally on the processing unit 28. Alternatively, one or more sensor outputs may be sent to the electronic device 19 and / or the central processing and storage unit 20 in a raw or processed format for further processing and analysis. Regardless of where the processing and analysis of eating behaviors and patterns occurs, the sensor outputs in a raw or processed format may be stored within the electronic device 18, within the electronic device 19, and / or within the central processing and storage unit 20.

[0165] In some embodiments, the generation, collection and / or processing of data that facilitates the measurement and analysis of eating behaviors, patterns and habits may be continuous, periodic or otherwise independent of the start and / or end of a food intake event. Alternatively, the generation, collection and / or processing of data that facilitates the measurement and analysis of eating behaviors and patterns may occur only during a food intake event or otherwise be related to a specific food intake event. It is also possible to generate, collect and / or process some sensor data continuously, periodically or otherwise independently of the start and / or end of a food intake event, while other sensor data is acquired during a food intake event or otherwise related to a food intake event.

[0166] The sensor or sensors that generate the data necessary to measure and analyze eating behavior and eating patterns may be internal to the electronic device 18. Alternatively, one or more of the sensors that generate the data necessary to measure and analyze eating behavior and eating patterns may be external to the electronic device 18 but capable of relaying relevant information to the electronic device 18 directly through direct wireless or wired communication with the electronic device 18 or indirectly through another device.

[0167] In the case of indirect communication through another device, such as a mobile phone or another portable or fixed device, such a third device is capable of receiving data or information from the external sensor unit, optionally processing such data or information, and forwarding the raw or processed data or information to the tracking device. Communications to and from the electronic device 18 may be wired or wireless, or a combination of both.

[0168] Examples of sensors that may be external to the electronic device 18 may be one or more sensors embedded in a necklace or pendant worn around the neck, one or more sensors embedded in small patches attached to different locations on the body, one or more sensors embedded in an auxiliary second wearable device worn on the other arm or wrist or on the fingers of the other hand, or one or more sensors integrated into the teeth.

[0169] Description of camera module usage and image capture

[0170] Although the use of cameras to capture images of food has been proposed in the prior art, this typically relies on the user taking a picture with their mobile phone or tablet. Unfortunately, image capture using a mobile phone or tablet imposes significant conflicts of use, may be socially unacceptable in certain dining situations, or may interfere with the authenticity of the dining experience. It is often undesirable or inappropriate for the user to take out their mobile phone, unlock the screen, open a mobile application, and take a picture using the camera built into the mobile phone.

[0171] If user intervention is required, it is generally desirable that it can be performed in a subtle and discrete manner and with as little friction as possible. To minimize usage friction, it is often desirable that image capture can be initiated directly from the electronic device 18.

[0172] Although the examples provided herein use image capture of food and dining situations as examples, it should be clear after reading this disclosure that the methods and devices described herein can be applied to image capture of objects and scenes other than food and dining situations. For example, a viewfinder-free camera can be applied outside the food event capture domain.

[0173] In some embodiments, the electronic device 18 is worn on a wrist, arm, or finger and includes one or more camera modules 26. The one or more camera modules 26 may be used for the capture of static images according to one embodiment of the present disclosure and for the capture of one or more video streams according to another embodiment of the present disclosure. In yet another embodiment of the present disclosure, a combination of static and streaming images is also possible.

[0174] One or more camera modules may also be included in a device worn at various locations on the body, such as a necklace or pendant worn around the neck, or a device attached to or integrated with the user's clothing, where the camera or camera module is preferably aimed forward so that its field of view encompasses the food being consumed.

[0175] In some embodiments, the camera module and / or the enabling of image capture by the camera module may require some degree of user intervention. User intervention may include, among other things: pressing a button; issuing a voice command to a microphone built into the electronic device 18 or mobile device; making selections using a display integrated into the electronic device 18 or mobile device; issuing specific arm, wrist, hand, or finger gestures; directing the camera so that the object of interest is within the camera's field of view; removing obstacles that may be in the line of sight between the camera and the object of interest; and / or adjusting the position of the object of interest so that it is within the camera's field of view. Other user intervention methods or combinations of multiple user intervention methods are also possible.

[0176] In one embodiment of the present disclosure, a camera module is built into an electronic device, such as a wearable device, which may not have a viewfinder or may not have a display that can give feedback to the user about the area within the camera's field of view. In this case, the electronic device may include a light source that projects a pattern of visible light onto a surface or object to indicate to the user the area within the camera's field of view. One or more light emitting diodes (LEDs) may be used as the light source. Other light sources including, but not limited to, lasers, halogen lamps, or incandescent light sources are also possible. The pattern of visible light may be used by the user, in particular, to adjust the position of the camera, adjust the position of the object of interest, and / or remove any objects that block the line of sight between the object of interest and the camera.

[0177] Light sources may also be used to communicate other information to the user. For example, the electronic device may use inputs from one or more proximity sensors, process those inputs to determine whether the camera is within an appropriate distance range to an object of interest, and use one or more light sources to communicate to the user that the camera is within an appropriate distance range, that the user needs to increase the distance between the camera and the object of interest, or that the user needs to decrease the distance between the camera and the object of interest.

[0178] The light source may also be used in conjunction with an ambient light sensor to communicate to the user whether the ambient light is insufficient or too intense for good quality image capture.

[0179] Light sources may also be used to communicate information including, but not limited to, low battery conditions or functional defects.

[0180] The light source can also be used to convey dietary guidance information. For example, the light source can indicate, among other things, whether the time elapsed since the last food intake event is insufficient or excessive, or can convey to the user how well his / her specific dietary goals are being achieved.

[0181] The signaling mechanism for conveying a specific message using one or more light sources may include, but is not limited to, one or more of the following: a specific light intensity or light intensity pattern, a specific light color or light color pattern, a specific spatial or temporal light pattern. Multiple mechanisms may also be combined to signal a specific message.

[0182] In another embodiment of the present disclosure, the camera module may be built into an electronic device 18, such as a wearable device, that does not have a viewfinder or a display that can give feedback to the user about the area within the camera's field of view. Instead of or in addition to using a light source, one or more images captured by the camera module, possibly combined with input from other sensors embedded in the electronic device 18, may be sent to a processing unit within the electronic device 18, a processing unit within the electronic device 19, and / or a central processing and storage unit 20 for analysis and determination of whether

[0183] Whether the object of interest is within the appropriate field of view and / or appropriate focus range of the camera. The results of the analysis can be communicated to the user using one of the feedback mechanisms available in the electronic device 18, including but not limited to tactile feedback, visual feedback using one or more LEDs or displays, and / or audio feedback.

[0184] In some other embodiments of the present disclosure, the electronic device 18 may capture one or more images without any user intervention. The electronic device 18 may capture static or streaming images continuously, periodically, or otherwise independently of any food intake event. Alternatively, the electronic device 18 may only enable one or more of its camera modules before, during, or around the time of a food intake event. For example, the electronic device may only enable one or more of its camera modules and capture one or more images after the start of a food intake event has been detected and before the end of the food intake event has been detected. It may use one or more of its camera modules to capture one of more images of an entire food item or dish, or an image of a portion of one or more food items or dishes.

[0185] In some embodiments, one camera may be used to capture one or more images of a food item on a plate, table, or other fixed surface, and a second camera may be used to capture one or more images of a food item (e.g., a snack or drink) held by a user. The use of more than one camera may be desirable where user intervention is not required and the position, viewing area, or focus range of a single camera is not suitable for capturing all possible dining scenarios.

[0186] In one example embodiment, the position, orientation, and viewing angle of the camera make it possible to capture images or videos without any user intervention. In such embodiments, the wearable device may use various techniques to determine the appropriate timing of image or video stream capture so that it can capture the food or portion of the food being consumed. It may also choose to capture multiple images or video streams for this purpose. The techniques used to determine the appropriate timing may include, but are not limited to, proximity sensing, sensing of acceleration or motion (or its absence), location information. Such sensor information may be used alone or in conjunction with pattern recognition or data analysis techniques (or a combination of both) to predict the optimal timing of image or video capture. Techniques may include, but are not limited to, model training based on machine learning.

[0187] The captured still and / or streaming images typically require some degree of processing. Processing may include, but is not limited to, compression, deletion, resizing, filtering, image editing, and computer vision techniques to identify objects, such as specific foods or dishes, or features such as portion size. Regardless of whether the camera module or multiple camera modules are internal to the electronic device 18, the processing units that can be used to process still or streaming images from the camera module or multiple camera modules include, but are not limited to: a processing unit internal to the electronic device 18, a processing unit internal to the electronic device 19, and / or a central processing and storage unit 20, which may reside in the same location as where the electronic device is used, or may reside in a remote location (such as in a cloud server), in which case the central processing and storage unit can be accessed via the Internet. Image processing may also be distributed among a combination of the processing units mentioned above.

[0188] Examples of local processing may include, but are not limited to, selecting one or more still images from a plurality of images or one or more video streams, compressing images or video streams, and applying a computer vision algorithm to one or more images or video streams.

[0189] Local processing may include compression. In the case of compression, the compressed image may be sent as part of a time-critical transaction, while its uncompressed version may be saved for later transmission.

[0190] One or more static or streaming images may be analyzed and / or compared for one or more purposes including, but not limited to, detecting the start and / or end of a food intake event, identifying food items, identifying food contents, identifying or deriving nutritional information, estimating portion sizes, and inferring certain eating behaviors and eating patterns.

[0191] As an example, computer vision techniques, optionally in combination with other image manipulation techniques, can be used to identify food types, specific food items, and / or estimate portion sizes. Alternatively, images can be manually analyzed using Mechanical Turk processing or other crowdsourcing methods. Once food types and / or specific food items have been identified, this information can be used to retrieve nutritional information from one or more food / nutrition databases.

[0192] As another example, information about the speed at which a user eats or drinks may be inferred from analyzing and comparing multiple images captured at different times during the course of a food intake event. As yet another example, images, optionally combined with other sensor information, may be used to distinguish between table meals and snacks or desserts. As yet another example, analysis of one image acquired at the beginning of a food intake event and another image acquired at the end of the food intake event may provide information about the amount of food actually consumed.

[0193] Description of user feedback

[0194] In a preferred embodiment of the present disclosure, the electronic device 18 is worn on a wrist, arm or finger and has one or more stimulation units and / or a user interface that allow feedback to be generated to the user or wearer of the electronic device. In different embodiments of the present disclosure, the electronic device 18 can be implemented as a wearable patch that can be attached to the human body or can be embedded in clothing.

[0195] Feedback typically includes feedback about food or food intake. Feedback methods may include, but are not limited to, tactile feedback, visual feedback using LEDs or displays, or audio feedback. In one such embodiment, the electronic device 18 may have a tactile interface that vibrates once or more when the start and / or end of a food intake event is detected. In another embodiment, the electronic device 18 may have a tactile interface that vibrates once or more when the tracking and processing subsystem identifies that the wearer of the device is consuming food and presents eating behaviors that exceed certain programming thresholds such as eating too fast, too slow, or too much. Alternatively, the tactile interface may vibrate once or more during a food intake event without being associated with any particular eating behavior, to, for example, remind the wearer of the fact that a food intake event is occurring and / or to improve perception and promote mindfulness eating at the moment. Other feedback methods are also possible, and different metrics or benchmarks may be used to trigger the activation of such feedback methods.

[0196] In various embodiments of the present disclosure, feedback is provided to the user by a device separate from the electronic device 18. One or more stimulation units and / or user interfaces required to provide feedback to the user may be external to the electronic device 18. For example, one or more stimulation units and / or user interfaces may be internal to the electronic device 19, and one or more of the stimulation units and / or user interfaces internal to the electronic device 19 may be used to provide feedback to replace or supplement the feedback provided by the electronic device 18. Examples may include, but are not limited to, messages shown on a display of the electronic device 19 or audible alerts issued by an audio subsystem embedded within the electronic device 19.

[0197] Alternatively, the feedback may be provided by a device separate from both electronic device 18 and electronic device 19 but capable of receiving data from at least one of those devices, either directly or indirectly.

[0198] In addition to or instead of feedback provided before, during or after the food intake event, Figure 2 or Figure 3 The system may also provide feedback that may span multiple food intake events or may not be associated with a specific food intake event or set of food intake events. Examples of such feedback may include, but are not limited to, food content and nutritional information, historical data overview, overview of one or more tracked parameters over a longer period of time, progress of one or more tracked parameters, personalized dietary guidance and recommendations, benchmarking of one or more tracked parameters against peers or other users with similar profiles.

[0199] Detailed Description of Certain Embodiments

[0200] In a specific embodiment of the present disclosure, electronic device 218 is a wearable device in the form of a bracelet or wristband worn on the wrist or arm of the user's dominant hand. Electronic device 219 is a mobile phone, and central processing and storage unit 220 is one or more computing servers and data storage devices located at a remote location.

[0201] One possible embodiment of a wearable bracelet or wristband according to aspects of the present invention is Figure 7 Wearable device 770 may optionally be implemented using a modular design, where individual modules contain one or more subsets of components and overall functionality. A user may choose to add specific modules based on personal preferences and needs.

[0202] The wearable device 770 may include a processor, a program code memory and program code (software) stored therein, and / or internal electronics 219 to optionally allow a user to customize a subset of the functionality of the wearable device 770 .

[0203] The wearable device 770 relies on a battery 769 and a power management unit ("PMU") 760 to deliver power to all electronic circuits and components at the appropriate supply voltage level. The power management unit 760 may also include battery recharging circuitry. The power management unit 760 may also include hardware such as switches that allow power to specific electronic circuits and components to be cut off when not in use.

[0204] When there is no ongoing behavioral event, most of the circuits and components in the wearable device 770 are disconnected to save power. Only the circuits and components required to detect or help predict the start of a behavioral event can be kept enabled. For example, if no motion is detected, all sensor circuits except the accelerometer can be disconnected, and the accelerometer can be placed in a low-power motion wake-up mode, or in another lower power mode that consumes less power than its high-performance active mode. The processing unit can also be placed in a low-power mode to save power. When motion or a certain motion pattern is detected, the accelerometer and / or the processing unit can be switched to a higher power mode, and additional sensors such as a gyroscope and / or a proximity sensor can also be enabled. When a possible event start is detected, memory variables for storing event-specific parameters, such as gesture type, gesture duration, etc., can be initialized.

[0205] In another example, after motion is detected, the accelerometer is switched to a higher power mode, but other sensors remain disconnected until data from the accelerometer indicates that the start of a behavioral event is likely to have occurred. At that moment, additional sensors such as gyroscopes and proximity sensors may be activated.

[0206] In another example, when there is no ongoing behavioral event, both the accelerometer and the gyroscope are activated, but at least one of the accelerometer or the gyroscope is placed in a power mode lower than its normal power mode. For example, the sampling rate can be reduced to save power. Similarly, the circuit required to transmit data from the electronic device 218 to the electronic device 219 can be placed in a lower power mode. For example, the radio circuit 764 can be completely disabled. Similarly, the circuit required to transmit data from the electronic device 218 to the electronic device 219 can be placed in a lower power mode. For example, it can be completely disabled until it is determined that a possible or probable behavioral event has begun. Alternatively, it can remain activated but in a low power state to maintain the connection between the electronic device 218 and the electronic device 219, but without transmitting sensor data.

[0207] In yet another example, in the event that a particular behavioral event, such as a food intake event, is determined to be unlikely based on certain metadata, all motion detection related circuits, including an accelerometer, may be disconnected. This may be desirable, for example, to further conserve power. The metadata used to make this determination may include, among other things, one or more of the following: time of day, location, ambient light level, proximity sensing, and detection that the wearable device 770 has been removed from the wrist or hand, detection that the wearable device 770 is being charged. The metadata may be generated and collected internally in the wearable device 770. Alternatively, the metadata may be collected internally in the mobile phone or in another device that is external to the wearable device 770 and external to the mobile phone, and may exchange information directly or indirectly with the mobile phone and / or the wearable device 770. It is also possible that some of the metadata is generated and collected internally in the wearable device 770, while other metadata is generated and collected in a device external to the wearable device 770. In the event that some or all of the metadata is generated and collected externally to the wearable device 770, the wearable device 770 may periodically or from time to time power up its radio circuit 764 to retrieve metadata-related information from a mobile phone or other external device.

[0208] In yet another embodiment of the present invention, where certain metadata indicates that a particular behavioral event, such as a food intake event, is likely to occur, some or all of the sensors may be turned on or placed in a higher power mode. The metadata used to make this determination may include, among other things, one or more of: time of day, location, ambient light level, and proximity sensing. Some or all of the metadata may be collected internally to the mobile phone or internally to another device that is external to the wearable device 770 and external to the mobile phone and that may directly or indirectly exchange information with the mobile phone and / or wearable device 770. Where some or all of the metadata is generated and collected externally to the wearable device 770, the wearable device 770 may periodically or from time to time power up its radio circuit 764 to retrieve metadata related information from the mobile phone or other external device.

[0209] The detection of the onset of a behavioral event, such as a food intake event, can be signaled to the user through one of the available user interfaces on the wearable device 770 or on a mobile phone to which the wearable device 770 is connected. As an example, a tactile interface 761 inside the wearable device 770 can be used for this purpose. Other signaling methods are also possible.

[0210] The detection of the start of a behavioral event, such as a food intake event, may trigger some or all of the sensors to be placed in or maintained in a high power mode or active mode to track certain aspects of the user's eating behavior for part or all of the food intake event. One or more sensors may be powered off or placed in a lower power mode at or after the actual or possible end of the behavioral event (the perceived end of the behavioral event) has been detected. Alternatively, one or more sensors may be powered off or placed in a lower power mode after a fixed or programmable time period.

[0211] Sensor data used to track certain aspects of user behavior, such as the user's eating behavior, can be stored locally within the memory 766 of the wearable device 770 and processed locally using the processing unit 767 within the wearable device 770. The sensor data can also be transmitted to a mobile phone or a remote computing server for further processing and analysis using the radio circuit 764. It is also possible that some processing and analysis is completed locally within the wearable device 770, and other processing and analysis is completed on the mobile phone or on a remote computing server.

[0212] Detection of the start of a behavioral event, such as the start of a food intake event, may trigger the power-up and / or activation of additional sensors and circuits, such as the camera module 751. The power-up and / or activation of the additional sensors and circuits may occur simultaneously with the detection of the start of a food intake event or at some later time. Specific sensors and circuits may be turned on only when needed, at specific times during a food intake event, and may otherwise be turned off to save power.

[0213] It is also possible that the camera module is powered on or enabled only after explicit user intervention, such as pushing and holding the button 759. Releasing the button may switch the camera module off again to save power.

[0214] When the camera module 751 is powered on, the projected light source 752 may also be activated to provide visual feedback to the user about the area within the camera's field of view. Alternatively, the projected light source 752 may be enabled only sometime after the camera module has been enabled. In some cases, additional conditions may need to be met before enabling the projected light source 752. Such conditions may include, among other things, determining that the projected light source 752 is likely to be aimed in the direction of the object of interest, or determining that the wearable device has not moved excessively.

[0215] In one particular embodiment, partially pressing the button 759 on the wearable device 770 can power on the camera module 751 and the projection light source 752. Further pressing the button 759 can trigger the camera module 751 to acquire one or more static images or one or more streaming images. In some cases, further pressing the button 759 can trigger the deactivation, brightness modification, color modification, or mode modification of the projection light source 752 before or simultaneously with image capture. Releasing the button 759 can trigger the deactivation and / or power down of the projection light source 752 and / or the camera module 751.

[0216] The images may be tagged with additional information or metadata, such as camera focus information, proximity information from proximity sensor 756, ambient light level information from ambient light sensor 757, timing information, etc. Such additional information or metadata may be used during processing and analysis of food intake data.

[0217] Various light patterns are possible and can be formed in various ways. For example, they may include a mirror or mechanism to reflect the projection light source 752 so that the projection light source 752 produces one or more rays that roughly show the center or boundaries of a particular area, such as a cross, L-shape, circle, rectangle, multiple points or lines that frame the field of view or otherwise give the user visual feedback about the field of view.

[0218] One or more light emitting diodes (LEDs) may be used as the projected light source 752. The pattern of visible light may be used by a user to, among other things, adjust the position of the camera, adjust the position of an object of interest, and / or remove any objects that block the line of sight between the object of interest and the camera.

[0219] The projected light source 752 may also be used to communicate other information to the user. For example, the electronic device may use inputs from one or more proximity sensors, process those inputs to determine whether the camera is within an appropriate distance range from an object of interest, and use one or more light sources to communicate to the user that the camera is within an appropriate distance range, that the user needs to increase the distance between the camera and the object of interest, or that the user needs to decrease the distance between the camera and the object of interest.

[0220] The light source may also be used in conjunction with an ambient light sensor to communicate to the user whether the ambient light is insufficient or too intense for good quality image capture.

[0221] Light sources may also be used to communicate information including, but not limited to, low battery conditions or functional defects.

[0222] The light source can also be used to convey dietary guidance information. For example, the light source can indicate, among other things, whether the time elapsed since the last food intake event is insufficient or excessive, or can convey to the user how well his / her specific dietary goals are being achieved.

[0223] The signaling mechanism for conveying a specific message using one or more projected light sources may include, but is not limited to, one or more of the following: a specific light intensity or light intensity pattern, a specific light color or light color pattern, a specific spatial or temporal light pattern. Multiple mechanisms may also be combined to signal a specific message.

[0224] Microphone 758 can be used by the user to add specific or custom tags or messages to food intake events and / or images. The audio clips can be processed by a speech recognition engine.

[0225] In some embodiments, in addition to tracking at least one parameter directly related to food intake and / or eating behavior, an accelerometer, possibly in combination with other sensors, may also be used to track one or more parameters not directly related to food intake. Such parameters may include activity, sleep, or stress, among others.

[0226] Specific embodiments without built-in camera

[0227] In various embodiments, the electronic device 218 may not have any built-in image capture capabilities. The electronic device 218 may be a wearable device, such as a bracelet or wristband worn on an arm or wrist, or a ring worn on a finger. The electronic device 219 may be a mobile phone, and the central processing and storage unit 220 may be one or more computing servers and data storage devices located at a remote location.

[0228] In such embodiments, the food intake tracking and feedback system may not use images to extract information about food intake and / or eating behavior. Alternatively, the food intake tracking and feedback system may utilize image capture capabilities available within other devices, such as electronic device 219 or an electronic device external to electronic device 218.

[0229] Upon detecting or predicting the onset of a food intake event, the electronic device 218 may send a signal to the electronic device 219, or to an electronic device that otherwise accommodates image capture capabilities, to indicate that an actual, possible, or imminent onset of a food intake event has occurred. This may trigger the electronic device 219, or an electronic device that otherwise accommodates image capture capabilities, to enter a mode that will allow the user to capture images in at least one fewer user step than a default mode or alternate mode of the device.

[0230] For example, if the image capture capability is housed within the electronic device 219 and the electronic device 219 is a mobile phone, tablet, or similar mobile device, the electronic device 218 may send one or more signals to the software installed on the electronic device 219 to indicate the actual, possible, or imminent start of a food intake event. Upon receiving such a signal or signals, the software on the electronic device 219 may take one or more of the following actions, among others: unlock the screen of the electronic device 219; open a mobile application associated with the food intake and feedback subsystem; enable the camera mode of the electronic device 219; push notifications to the display of the electronic device 219 to assist the user in image capture; send a message to the electronic device 218 to alert, remind, and / or assist the user in image capture.

[0231] After image capture by the electronic device 219 or an electronic device that otherwise accommodates image capture capabilities, the electronic device 219 or an electronic device that otherwise accommodates image capture capabilities may provide visual feedback to the user. Examples of visual feedback may include patterns, shapes, or overlays that show suggested portion sizes, or patterns, shapes, or overlays that are colored in one or more colors and / or have one or more brightness levels to indicate the healthiness of the food. Other examples are also possible.

[0232] Integration with insulin therapy systems

[0233] One or more components of the food intake tracking and feedback system presented in the present disclosure may be interfaced to or integrated with an insulin therapy system. In a specific example, after the start of a food intake event is detected, feedback may be sent to the wearer to remind him or her to take a glucose level measurement and / or administer an appropriate insulin dose. One or more additional reminders may be sent during a food intake event.

[0234] The food intake tracking and feedback system described in the present disclosure or its components can also be used by patients diagnosed with type I or type II diabetes. For example, the components described in the present disclosure can be used to automatically detect when an individual starts eating or drinking. The detection of the start of a food intake event can be used to send a message to the wearer to remind him or her to measure glucose levels and / or administer appropriate insulin doses when the food intake event starts or approaches the start. Message transmission can be automatic and independent. Alternatively, the system can be integrated with a health system or a health care maintenance and reminder system. After being informed that the start of a food intake event has been detected, the health system or the health care maintenance and reminder system can send a message to the wearer. The health system or the health care maintenance and reminder system can receive additional information about the food intake event, such as the number of bites or sips, the estimated amount of food consumed, the length of meal, the speed of eating, etc. The health system or the health care maintenance and reminder system can send additional messages to the wearer during or after the food intake event based on the additional information.

[0235] In another example, specific information about the content of food intake can be used as an input, possibly combined with one or more other inputs to calculate an appropriate insulin dose to be administered. Information about the content of food intake can include, among other things, one or more of: amount of carbohydrates, amount of sugars, amount of fat, portion size, and molecular food category, such as solid or liquid. Real-time, near real-time, and historical information related to food intake and eating patterns and behaviors can be included as inputs or parameters for calculating insulin doses.

[0236] Other inputs that may be used as inputs or parameters for the algorithm for calculating insulin dosage may include, among others, one or more of: age, gender, weight, historical and real-time blood glucose levels, historical and real-time activity, sleep and stress levels, vital sign information, or other information indicative of the individual's physical or emotional health.

[0237] The calculation of the insulin dose can be done completely manually by the user, completely autonomously by a closed-loop insulin therapy system, or semi-autonomously, where some or all of the calculations are done by the insulin therapy system, but still require some user intervention. User intervention can include, among other things, enabling the insulin therapy calculation unit, confirming the dose, intervening or suspending insulin delivery if an anomaly is detected or identified by the user.

[0238] In a particular embodiment, upon detecting the actual, possible, or imminent onset of a food intake event, in addition to or instead of sending a notification to the user, the food intake tracking and feedback system described herein may also send one or more notifications to one or more caregivers of the user.

[0239] The user can capture one or more images of the food or meal to one or more caregivers after the food intake event begins (optionally prompted by a notification or signal from the system or one of his caregivers). The caregiver can analyze the image and send information about the content of the food back to the user. The information may include, among other things, estimates of certain major nutrient content such as carbohydrates, sugars or fats, estimates of calorie values, suggestions for portion sizes.

[0240] In the case where the user receives insulin therapy, additional information such as blood sugar readings may also be sent to the caregiver, who may also provide information back to the user including recommendations for the insulin dose to be administered and timing for when this insulin dose should be administered. In certain embodiments, the caregiver may not be a person but an artificial intelligence system.

[0241] Gesture Recognition

[0242] In the various systems described herein, it may be important to accurately determine gesture information. For example, it would be useful to distinguish between gestures associated with speaking and gestures indicating the start of a period of eating events. Some gestures may be easy to detect, such as the gesture of swinging an arm while walking, and thus measuring speed and steps, but other gestures may be more difficult to measure, such as determining when a user takes a bite of food, drinks a sip of a drink, bites his nails, etc. The latter may be applicable to assessing premonitory behavior. For example, it is envisioned that a health maintenance and reminder system detects a pattern of gestures associated with nervous eating five to ten minutes after a nail biting gesture. A user may program his health maintenance and reminder system to signal it two minutes after biting his nails, so that the user begins to notice and is more in tune with his originally unnoticed behavior. To achieve this, gesture detection should be accurate and reliable. This may be a problem in the case where there is no simple correlation between the movement of an accelerometer in a wearable bracelet and nervous eating. Part of the reason for this is that the attention gestures of the health maintenance and reminder system are not easy to derive from simple sensor readings.

[0243] Being able to determine whether a user is taking a bite of food or a sip of a drink and being able to distinguish between biting and sipping can be used to provide appropriate weight management guidance. For example, a weight management monitoring and reminder system can monitor the user's food intake events based on posture. A weight management monitoring and reminder system can additionally monitor the user's fluid intake events based on posture. Studies have shown that drinking enough water at the beginning or near the beginning of a meal and drinking enough water during the entire meal will reduce food consumption and help lose weight. The user, the user's instructor, the user's health care provider, or the provider of the weight management monitoring and reminder system can program the system so that it sends a reminder when the user starts eating without drinking water or when it detects that the user does not drink enough water during the entire meal. The system can also monitor the user's fluid intake throughout the day and is programmed to send a reminder when the fluid intake does not meet the preconfigured amount for a specific time of the day. To achieve this, posture detection should be reliable and accurate. This may be a problem in the case where it is necessary to distinguish postures with many similarities (e.g., distinguishing between eating postures and drinking postures).

[0244] In various embodiments described herein, a processing system (including program code, logic, hardware and / or software, etc.) receives sensor data generated by an electronic device or other element based on user activity. The sensor data may represent a snapshot of readings at a specific time or may represent readings over a period of time. The sensor may be an accelerometer, gyroscope, magnetometer, thermometer, illuminometer, etc. Based on the sensor data, the processing system uses stored rules and internal data (such as information about which sensor is used and past usage history) to identify behavioral events, where a behavioral event is a series of gestures, and the gestures are determined based on the sensor data with a start time, sensor readings, and end time, and a logical arrangement of external data. The behavioral event may be a high-level event, such as a meal, etc.

[0245] Determination of gesture boundaries (i.e., its start and end times) can be determined using the methods described herein. The data of the start time, sensor readings, and end time are collectively referred to herein as a gesture envelope. A gesture envelope may also include an anchor time, which is a data element that defines a single time associated with this gesture envelope. The anchor time may be a midpoint between the start time and the end time, but may be based on the sensor data of the gesture envelope according to some benchmark. The anchor time may be outside the time span from the start time to the end time. Multiple anchor times per gesture are also possible.

[0246] A machine classifier that is part of the processing system (but can also be a separate computer system and possibly separated by some network) determines what class of gesture it is based on the gesture envelope, the sensor data that produced this gesture envelope, and the details of the gesture. For example, the machine classifier may output: the sensor data indicates or indicates that the individual wearing the wristband containing the sensor is walking, eating, or pointing at something.

[0247] With this system, if the posture can be accurately identified, the health maintenance and reminder system (or other systems using posture information) can accurately respond to the posture. In the following example, there is a group of sensors connected to the machine classification system or at least input from a group of sensors, and the machine classification system outputs posture data according to sensor readings, taking into account rules and storage data derived from training the machine classification system. The training subsystem can be used to train the machine classification system and thus form storage data derived from the training. Each of these components can use different hardware or shared hardware, and can be localized and / or remote. In general, when a posture is detected, the system can analyze this posture, determine the activities that are likely to be actual, possible or about to occur, and provide feedback about those activities to the user. For example, the vibration as a feedback signal indicates that the user previously set the system to alert the user when the user has been drinking for a semi-continuous period greater than 45 minutes, or indicates that the user has reached the walking volume goal that should be completed in a period.

[0248] Figure 8 is an illustrative example of a typical machine classification system. Figure 8 The machine classification system includes a training subsystem 801 and a detector subsystem 802. In some embodiments of the present disclosure, the machine classification system may include additional subsystems or Figure 8 801 uses training data input 803 and labels 804 to train a trained classifier model 805. Labels 804 may be manually assigned by a person or may be generated in an automatic or semi-automatic manner. The trained classifier model 805 is then used in the detector subsystem 802 to generate a classification output 806 corresponding to a new unlabeled data input.

[0249] The stored sensor data contains a time component. The raw sensor readings are tagged with the time at which they were read. The raw sensor data can be obtained from accelerometers, gyroscopes, magnetometers, thermometers, barometers, humidity sensors, ECG sensors, etc., and the time data can come from other sources. Other examples of time sources can be audio, voice, or video recordings.

[0250] Respectively Fig. 9 and Fig.10An illustrative example of a training subsystem 801 and a detector subsystem 802 according to at least one embodiment of the present disclosure is shown in FIG. The temporal training data 907 and the labels 912 are fed to Figure 8 The classifier training subsystem.

[0251] As explained in the examples herein, the raw sensor data is processed to identify macrosignature events. A macrosignature event may define a gesture that includes sensor data over a period of time. A detector subsystem or other system may generate a gesture envelope data set that includes a start time, an end time, one or more anchor times, metadata, and sensor data that occurs within a temporal envelope of the gesture from the start time to the end time.

[0252] For example, with respect to the problem of gesture recognition, a gesture envelope detector may identify specific time segments in the raw temporal data that indicate possible gestures. The gesture envelope detector also generates a temporal envelope that specifies the relevant times or time segments within the gesture. The information contained in the temporal envelope may include, among other things, the start time of the gesture, the end time of the gesture, the time within the gesture that specifies the relevant gesture sub-segment, the time within the gesture that specifies the relevant gesture anchor time (point) and possibly other metadata, as well as the raw sensor data within the temporal envelope of the gesture.

[0253] As an example of other metadata, imagine that a historical pattern indicates that the wearer has a period of eating after a phone call to a particular phone number. The electronic device can signal this to the wearer to be aware of this pattern, which can help change behavior if the wearer so decides.

[0254] The temporal training data 907 is fed to a gesture envelope detector 908. The gesture envelope detector 908 processes the temporal training data 907 and identifies possible moments of a gesture 909 and a corresponding gesture temporal envelope from the temporal training data 907. The temporal training data 907 may include motion sensor data, and the gesture envelope detector 908 may process the motion sensor data and identify the gesture 909 based on a change in pitch angle. In one embodiment, the gesture envelope detector 908 may detect the start of a gesture based on detecting a rise in pitch angle above a specified value, and detect the end of an event based on a fall in pitch angle below a specified value. Other start and end references are also possible. An example of an anchor point that may be detected by the gesture envelope detector 908 and specified by the gesture temporal envelope may be the time within the gesture segment when the pitch angle reaches a maximum value. Other anchor point examples are also possible.

[0255] The gesture envelope detector 908 may add additional benchmarks to further identify the segment as a valid gesture. For example, a threshold may be specified for the peak pitch angle or average pitch angle within a segment. In another example, minimum and / or maximum limits may be specified for the duration of the entire segment or for the duration of a sub-segment within the entire segment. Other benchmarks are also possible. Hysteresis may be used to reduce sensitivity to noise jitter.

[0256] In other embodiments of the present disclosure, the gesture envelope detector 908 may monitor other metrics derived from the input providing the temporal training data 907 and use those metrics to detect gestures. Examples of other metrics include, but are not limited to, roll angle, side turn, first or higher order derivatives of motion sensor data, or first or higher order integrals. The temporal data may be or may include data other than motion sensor data. In some embodiments of the present disclosure, the gesture envelope detector 908 may monitor and use multiple metrics to detect a gesture or specify a gesture temporal envelope.

[0257] The posture 909, together with the posture temporal envelope information, is combined with the temporal training data 907 and fed to the feature generator module 910. The feature generator module 910 calculates one or more posture features using information from the temporal training data 907, the posture temporal envelope, or a combination of information from the temporal training data 907 and the posture temporal envelope. In some embodiments of the present disclosure, the feature generator module 910 calculates one or more posture features based on the temporal training data 907 in a certain time segment or in the entire time segment, and the time segment is in the posture temporal envelope. It is also possible that the feature generator module 910 calculates one or more posture features based on the temporal training data 907 in a certain time segment or in the entire time segment, and the time segment is not in or only partially in the posture temporal envelope but is still related to the posture temporal envelope. An example may be a posture feature calculated based on the temporal training data 907 in the time period immediately before the start of the posture temporal envelope or in the time period immediately after the end of the posture temporal envelope.

[0258] In some embodiments, feature generator module 910 may generate one or more features directly based on gesture temporal envelope information without using temporal training data 907. Examples of such features may include, but are not limited to, the total duration of a gesture temporal envelope, the time elapsed since the last gesture, the time until the next gesture occurs, or the duration of a particular sub-segment within the entire gesture temporal envelope or event temporal envelope.

[0259] In one embodiment, the temporal training data 907 may be motion sensor data, and features may include readings of pitch, roll, and / or yaw angles calculated within one or more sub-segments or throughout a gesture temporal envelope. Features may also include the minimum, maximum, mean, variance, first or higher order derivatives, first or higher order integrals of various motion sensor data inputs calculated within one or more sub-segments or throughout a gesture temporal envelope. Features may also include distance traveled along a particular sensor axis or in a particular direction calculated within one or more sub-segments or throughout a gesture temporal envelope. Other features are also possible.

[0260] Temporal training data 907 may be or may include data other than motion sensor data, such as sensor signals from one or more of the sensors described herein. A sub-segment may be selected based on a time point or time segment specified by a gesture temporal envelope, within which the feature generator module 910 computes features or throughout the sub-segment. A sub-segment may also be selected based on a time point or time segment from multiple gesture envelopes, such as adjacent gestures or gestures that may not be adjacent but are in close proximity.

[0261] Some embodiments may use multiple posture envelope detectors in parallel or otherwise. Parallel posture envelope detectors may operate on different subsets of sensor data, may use different thresholds or benchmarks to identify postures, etc. For example, in the case of posture recognition based on motion sensor data input, one posture envelope detector may use pitch angle, while a second parallel posture envelope detector may use roll angle. One of the posture envelope detectors may be a primary posture envelope detector, while one or more additional posture envelope detectors may serve as auxiliary posture envelope detectors. Feature generation logic may process postures generated by the primary posture envelope detector, but may implicitly display features derived using information from the posture temporal envelope of nearby postures (in time) obtained from one or more auxiliary parallel envelope detectors.

[0262] The training data may include a plurality of posture envelope data sets, each having an associated label (e.g., selected from a list of posture labels) representing a posture provided manually or in some other manner in a test environment. This training data in combination with the associated labels may be used to train a machine classifier so that it may later process posture envelopes of unknown postures and determine the posture label that most appropriately matches the posture envelope. Depending on the classification method used, the training set may be purged, but the original data (unsupervised classification) or a set of features derived therefrom may be purged in addition, while the original data (supervised classification) may be purged in addition.

[0263] Regardless of the classification method, defining appropriate data boundaries for each tag is important to the performance of the classifier. Defining appropriate data boundaries may be difficult in terms of time issues (i.e., whereby at least one of the data inputs has a time dimension associated with it). This is particularly true in cases where the time dimension is variable or dynamic and where features related to specific segments of the variable time envelope or to the entire variable time envelope significantly contribute to the classifier performance.

[0264] One example of such a temporal problem is gesture recognition, such as detecting eating or drinking gestures from raw motion sensor data. The duration of a bite or sip may vary from person to person and may depend on the dining context or details of the food consumed. Other examples of gesture recognition problems are recognition of hand gestures related to smoking, dental hygiene, nail biting, nose picking, hair pulling, sign language, etc. In some variations, the system is used in a production environment to improve productivity.

[0265] The feature generation logic may also generate features derived from combining outputs from multiple gesture envelope detectors. Examples include, but are not limited to, the time elapsed from a primary gesture to the closest gesture from a parallel secondary gesture envelope detector.

[0266] The output of the feature generator module 910 is a set of gestures 911 with corresponding temporal envelopes and features. Before the gestures 911 can be fed to the classifier training module 915, the labels 912 from the training dataset need to be mapped to their corresponding gestures. This mapping operation is performed by the label mapper module 913.

[0267] In some embodiments, the timestamp associated with a tag 912 is always within the temporal envelope of its corresponding gesture. In this case, the logic of the tag mapper module 913 can be a lookup method, where the timestamp of each tag is compared with the start and end times of each gesture temporal envelope, and each tag is mapped to a gesture where the timestamp of the tag is greater than the start time of the corresponding gesture temporal envelope and less than the end time of the corresponding gesture temporal envelope. A gesture for which there is no corresponding tag can be marked as "no sign", indicating that it does not correspond to any tag of interest.

[0268] However, in other embodiments of the present disclosure, the timestamps of the markers 912 may not always be within a certain gesture temporal envelope. This may be due to the details of the steps followed during the marking process, the timing uncertainty associated with the marking process, the unpredictability or variability of the actual raw data input, or artifacts of the gesture envelope detector logic. In such cases, the marker mapper may be modified to adjust the boundaries of the gesture envelope.

[0269] The gestures 914 characterized as features and tags may then be fed to a classifier training module 915 to generate a trained statistical model that may be used by the detector subsystem. The classifier training module 915 may use a statistical model such as a decision tree model, a K-nearest neighbor algorithm model, a support vector machine model, a neural network model, a logistic regression model, or other models suitable for machine classification. In other variations, such as in Fig. 9 The table structure and data format of the data used in may vary and may differ from Fig. 9 as shown in .

[0270] Fig.10 An illustrative example of a detector subsystem is shown. As shown here, the untagged time data 1017 is fed to Fig.10 The detector subsystem includes a posture envelope detector logic 1018 and a feature generator logic 1020. Functionally, the posture envelope detector logic 1018 used by the detector subsystem is similar to the posture envelope detector logic used by its corresponding training subsystem. Likewise, the feature generator logic 1020 of the detector subsystem is functionally similar to the feature generator module 910 of its corresponding training subsystem. In some embodiments, the posture envelope detector logic 1018 can monitor and use multiple metrics to detect a posture or specify a posture time envelope.

[0271] However, the implementation of the posture envelope detector logic 1018 and the feature generator logic 1020 may differ in the training subsystem and its corresponding detector subsystem. For example, the detector subsystem may be implemented on more power-constrained hardware, in which case the posture envelope detector logic 1018 may need to be optimized for lower power operation than its counterpart used in the corresponding training subsystem. The detector subsystem may also have more stringent latency requirements than the training system. In this case, the posture envelope detector logic 1018 used in the detector subsystem may need to be designed and implemented for lower latency than its counterpart used in the corresponding training subsystem.

[0272] The output of the feature generator logic 1020 is fed to the feature generator logic 1020, which classifies the gesture based on the trained classifier module from its corresponding training subsystem. The classification output may include one or more tags. Optionally, the detector 1022 may also assign a confidence level to each tag.

[0273] Classification of combinations of temporal and non-temporal data inputs

[0274] In another embodiment, the input into the classification system may include a combination of temporal and non-temporal data. Fig.11is an illustrative example of a training subsystem in accordance with at least one embodiment of the present disclosure, wherein at least some of the data inputs are temporal and at least some of the data inputs are non-temporal. Other implementations are also possible.

[0275] The non-temporal training data 1129 need not be processed by the gesture envelope detector 1125 and feature generator logic 1127. The non-temporal training data 1129 may be fed directly into the tag mapper logic 1132 along with the tags 1131. In some embodiments, the non-temporal training data may be processed by a separate feature generator module, the non-temporal feature generator module 1130, to extract specific non-temporal features of interest, which are then fed into the tag mapper logic 1132. The tag mapper logic 1132 may assign tags 1131 to gestures along with the non-temporal features 1136 attached to the tags using methods similar to the methods of mapping tags to gestures already described herein.

[0276] Fig.12 is an illustrative example of a classification detector subsystem in accordance with at least one embodiment of the present disclosure, wherein at least some of the data inputs are temporal and at least some of the data inputs are non-temporal.

[0277] Unsupervised classification of temporal data input

[0278] In yet another embodiment of the present disclosure, a deep learning algorithm can be used for machine classification. Classification using a deep learning algorithm is sometimes referred to as unsupervised classification. With unsupervised classification, a deep learning statistical algorithm performs the classification task based on directly processing the data, thereby eliminating the need for a feature generation step.

[0279] Fig.13 An illustrative example of a classifier training subsystem in accordance with at least one embodiment of the present disclosure is shown, wherein the classifier training module is based on a deep learning statistical algorithm for unsupervised classification.

[0280] The posture envelope detector 1349 calculates postures 1350 with corresponding posture time envelopes based on the time training data 1348. The data segmenter 1351 assigns an appropriate data segment or multiple data segments to each posture based on the information in the posture time envelope. For example, the data segmenter 1351 can view the start and end time information in the posture time envelope and assign one or more data segments corresponding to the entire posture duration. This is just one example. Data segments can be selected based on different segments or sub-segments defined by the posture time envelope. Data segments can also be selected based on time segments that are outside the posture time envelope but are directly or indirectly related to the posture time envelope. An example can be selecting a data segment corresponding to a period of time immediately before the start of the posture time envelope, or selecting a data segment corresponding to a period of time immediately after the end of the posture time envelope. Other examples of time segments that are outside the posture time envelope but are directly or indirectly related to the posture time envelope are also possible.

[0281] The posture comprising the data segment, the posture temporal envelope information and the label is fed to the classifier training module 1356. In some embodiments of the present disclosure, only a subset of the posture temporal envelope information may be fed to the classifier training module 1356. In some embodiments of the present disclosure, the posture temporal envelope information may be processed before it is applied to the classifier training module 1356. One example may be to align the time base of the posture temporal envelope with the beginning of the data segment, rather than with the time base of the original temporal training data stream. Other examples are also possible. The performance of the classifier training module may be improved by adding temporal envelope information that further characterizes the data segment.

[0282] For example, in the case of gesture recognition of an eating gesture based on motion sensor data input, feeding additional anchor time information such as the time when the pitch, roll, or yaw angle reaches a maximum or minimum value into the classifier training module can improve the performance of the trained classifier 1357, because the trained classifier 1357 can analyze the training data and look for features and correlations especially around the anchor time. Other examples of temporal envelope information that can be fed to the classifier training module are also possible.

[0283] Fig.14 The invention shows that the invention can be combined with at least one embodiment of the present disclosure. Fig.13 An illustrative example of a classification detector subsystem used by a classification training subsystem of FIG.

[0284] Classifier Ensemble

[0285] In some embodiments, multiple parallel classification systems based on gesture envelope detection may be used. Fig.15An example of a system with multiple parallel classifiers is shown in . The number of parallel classification systems may vary. Each classification system 1510, 1512, 1514 has its own training and detector subsystem and performs posture envelope detection on a different subset of the training data 1502 and label 1504 inputs to detect postures, or may use different thresholds or benchmarks to identify postures. Therefore, each individual posture envelope detector will produce an independent set of postures, each with a different posture time envelope. The feature generator logic of each classification system produces features of the postures, which are produced by its corresponding posture envelope detector logic. The features may be different for each classification system. The classifier models used by each parallel classifier may be the same or different, or some may be the same and others may be different. Because the posture time envelopes and features used for the training of each classifier model are different, the parallel classification systems will produce different classification outputs 1516, 1518, 1520.

[0286] The classification output 1516, 1518, 1520 of each classification system may be fed to a classifier combiner subsystem 1522. The classifier combiner subsystem 1522 may combine and weight the classification outputs 1516, 1518, 1520 of the individual classification systems 1510, 1512, 1514 to produce a single overall classification result, the combined classification output 1524. The weighting may be static or dynamic. For example, with respect to gesture recognition, some classifiers may be better at correctly predicting the gestures of one group of people, while other classifiers may be better at correctly predicting the gestures of another group of people. The classifier combiner subsystem 1522 may use different weights for different users or different background conditions to improve the performance of the entire classifier ensemble. The trained system may then be used to process the unlabeled data 1506.

[0287] Other examples of timing issues include, but are not limited to, autonomous driving, driver alert systems (which warn the driver when dangerous traffic conditions are detected), driver alertness detection, voice recognition, video classification (security camera surveillance, etc.), and weather pattern recognition.

[0288] Ignoring the temporal nature of the data input and any features related to the temporal envelope of the data input may limit the performance of the classifier and make it unsuitable for the classification task in cases where reliable detection depends on features that are essentially related to segments of the variable temporal envelope or to the entire variable temporal envelope. Performance and usability may be impaired if the appropriate time period cannot be reliably determined or if the time period varies from posture to posture, from person to person, etc.

[0289] As described herein, an improved method utilizes a variable temporal envelope to adjust for temporal issues such that information related to the entire variable temporal envelope or a segment thereof can be extracted and included in a feature set used to train a classifier. The proposed improved method improves performance and reduces the amount of training data required because features can be defined relative to the temporal boundaries of the variable temporal envelope, thereby reducing sensitivity to temporal and user variations.

[0290] In addition to finding the temporal envelope of a gesture, the system can also find the event temporal envelope. In such an approach, the system can determine the gesture and gesture envelope, but then do the same for additional gestures and then define the event envelope, such as the start and end of an eating event.

[0291] Background to improve overall accuracy

[0292] Fig.16 An example of a machine classification system including a cross-correlation analysis subsystem is shown. Classification output 1602 may be fed to a cross-correlation analysis subsystem 1604. The cross-correlation analysis subsystem 1604 may make adjustments based on one or more contextual cues to improve accuracy. In the example of gesture recognition, an example of a contextual cue may be proximity in time to other predicted gestures. For example, eating gestures tend to be clustered together in time as part of an eating activity such as a meal or snack. As an example, the cross-correlation analysis subsystem 1604 may increase the confidence level that the predicted gesture is an eating gesture based on the confidence level and proximity to nearby predictions.

[0293] In another embodiment, the cross-correlation analysis subsystem 1604 may take as input the individual predicted gestures 1614 from the classification output 1602 and may cluster the individual predicted gestures into predicted activities 1608. For example, the cross-correlation analysis subsystem 1604 may map multiple biting gestures to eating activities such as snacks or meals. Similarly, the cross-correlation analysis subsystem 1604 may map multiple sip gestures to drinking activities. Other examples of activity prediction based on gesture clustering are also possible. The cross-correlation analysis subsystem 1604 may modify the confidence level of the predicted gesture based on the time interval and sequence of the predicted activities. For example, the cross-correlation analysis subsystem 1604 may reduce the confidence level that the predicted gesture is an eating gesture if the gesture is detected immediately after or during the "brushing teeth" activity. In another example, the cross-correlation analysis subsystem 1604 may reduce the confidence level that the predicted gesture is a drinking gesture if the gesture is detected during or immediately after the brushing teeth activity. In this case, the cross-correlation analysis subsystem 1604 may decide to increase the confidence level that the gesture is a gargling gesture.

[0294] The cross-correlation analysis subsystem 1604 may adjust the classification output of the predicted gesture based on the historical information 1612 or other non-gesture metadata 1610 information such as location, date and time, other biometric input, calendar or phone call activity information. For example, the cross-correlation analysis subsystem 1604 may increase the confidence level that the predicted gesture is an eating gesture or the predicted activity is an eating activity if the GPS coordinates indicate that the individual is at a restaurant. In another example, the cross-correlation analysis subsystem 1604 may increase the confidence level that the predicted gesture is an eating gesture or the predicted activity is an eating activity if the predicted gesture or predicted activity occurs at a certain time of day and past behavior indicates that the user typically eats at this time. In yet another example of the present disclosure, in the case where a predicted gesture or predicted activity precedes or follows a calendar event or a phone call, if past behavior indicates that the user typically eats before or after similar calendar events (e.g., with the same attendees, at a certain location, for certain meeting schedules, etc.) or phone calls (e.g., from a specific phone number), then the cross-correlation analysis subsystem 1604 may increase the confidence level that the predicted gesture is an eating gesture or the predicted activity is an eating activity. Although the above example refers to eating, it is obvious to a technician in the field that this can also be applied to gestures other than eating. In general, a machine classifier with a cross-correlation analysis subsystem uses contextual cues, historical information, and insights based on temporal proximity sensing to improve accuracy, where the specific contextual cues, historical information, and insights based on temporal proximity sensing and how to apply them are determined by the methods disclosed or proposed herein.

[0295] In some embodiments of the present disclosure, the classification output 1602 may include additional features or gesture temporal envelope information. The cross-correlation analysis subsystem 1604 may process such additional features or gesture temporal envelope information to determine or extract additional characteristics of the gesture or activity. For example, in one embodiment of the present disclosure, the cross-correlation analysis subsystem 1604 derives an estimated duration of a drinking gesture from the gesture temporal envelope, and this information may be used by the cross-correlation analysis subsystem 1604 or by one or more systems external to the machine classifier system to estimate fluid intake associated with the drinking gesture.

[0296] In another embodiment, the cross-correlation analysis subsystem 1604 may derive an estimated duration of an eating gesture from the gesture time envelope, and this information may be used by the cross-correlation analysis subsystem 1604 or by one or more systems external to the machine classifier system to estimate a bite size associated with an eating gesture. The cross-correlation analysis subsystem 1604 may combine the predicted drinking gesture with other sensor data to more accurately predict whether a person is consuming a beverage containing alcohol and estimate the amount of alcohol consumed. Examples of other sensor data may include, but are not limited to, measuring hand vibrations, heart rate, voice analysis, skin temperature, measuring blood, breath chemistry, or body chemistry.

[0297] The detector subsystem 1600 may predict a particular eating or drinking method, and the cross-correlation analysis subsystem 1604 may combine information obtained from the detector subsystem 1600 about eating or drinking method details with additional metadata to estimate food content, healthiness, or calorie intake. Examples of eating / drinking methods may include, but are not limited to, eating with a fork, eating with a knife, eating with a spoon, eating with fingers, drinking from a glass, drinking from a cup, drinking from a straw, etc. Examples of metadata may include, but are not limited to, time of day, location, environmental, or social factors.

[0298] annotation

[0299] Connective language, such as phrases of the form "at least one of A, B, and C" or "at least one of A, B, and C", unless specifically stated otherwise or otherwise clearly contradicted by context, is generally understood in conjunction with the context as used to mean that an item, term, etc. can be A or B or C; or any non-empty subset of the set of A and B and C. For example, in the illustrative example of a set having three members, the connective phrases "at least one of A, B, and C" and "at least one of A, B, and C" refer to any of the following sets: {A}, {B}, {C}, {A, B}, {A, C}, {B, C}, {A, B, C}. Thus, such connective language is generally not intended to imply that certain embodiments require that at least one A, at least one B, and at least one C are all present.

[0300] Unless otherwise specified herein or otherwise clearly contradictory to the context, the operations of the methods described herein may be performed in any suitable order. The methods described herein (or variations and / or combinations thereof) may be performed under the control of one or more computer systems configured with executable instructions, and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is jointly executed by hardware or a combination thereof on one or more processors. The code may be stored on a computer-readable storage medium, such as in the form of a computer program that includes a plurality of instructions that may be executed by one or more processors. The computer-readable storage medium may be non-transitory.

[0301] Unless otherwise required, the use of any and all examples or exemplary language (e.g., "such as") provided herein is intended only to better illustrate embodiments of the invention and does not impose limitations on the scope of the invention. No language in this specification should be construed as indicating any non-claimed element is essential to the practice of the invention.

[0302] After reading this disclosure, other embodiments may be envisioned by those skilled in the art. In other embodiments, combinations or sub-combinations of the invention disclosed above may be advantageously made. Example configurations of components are shown for illustrative purposes, and it should be understood that combinations, additions, rearrangements, etc. are encompassed in alternative embodiments of the present invention. Therefore, although the present invention has been described with respect to exemplary embodiments, those skilled in the art will recognize that many modifications are possible.

[0303] For example, the methods described herein may be implemented using hardware components, software components, and / or any combination thereof. Therefore, the description and drawings should be viewed in an illustrative sense rather than a restrictive sense. However, it is apparent that various modifications and changes may be made to the present invention without departing from the broad spirit and scope of the invention as set forth in the claims, and the present invention is intended to cover all modifications and equivalents within the scope of the appended claims.

[0304] All references, including publications, patent applications, and patents, cited herein are hereby incorporated by reference to the same extent as if each reference was individually and specifically indicated to be incorporated by reference and were set forth in its entirety herein.

Claims

1. A method for sensing wearer activity, wherein the wearer activity includes a behavioral event of a wearer of at least one electronic device, the method comprising: determining, using a processor, sensor readings, wherein at least one sensor reading is from an accelerometer of the at least one electronic device that measures movement of a body part of the wearer; determining a first gesture indicative of a potential behavioral event based on the sensor readings; determining, using the processor, a stored state, the stored state being a state from a set of states, the set of states comprising an in-event state and an out-of-event state, wherein the processor processes gesture indications differently in the in-event state and the out-of-event state; while in the out-of-event state, determining, using the processor and based on the first gesture, the onset of at least one behavioral event corresponding to the potential behavioral event based on detection of a single event with a confidence level above a predetermined confidence level or based on a series of two or more sensed gestures; upon determining the onset of the at least one behavioral event, activating at least one sensor or component of the food intake tracking system; collecting data or information related to the at least one behavioral event from the at least one sensor or component after activation; providing storage of event-specific parameters initialized for the at least one behavioral event; When in the event state, determining a series of gestures associated with the at least one behavioral event; as well as The event-specific parameters regarding the at least one behavioral event are derived from the series of gestures.

2. A method according to claim 1, wherein the event-specific parameters include one or more of the following: a timestamp associated with the start of the at least one behavioral event, a speed associated with the at least one behavioral event, and / or a measurement of the amount of activity associated with the at least one behavioral event. The method of claim 2 , wherein the timestamp is equal to the start of the at least one behavioral event.

4. The method of claim 1, wherein the body part of the wearer is an arm, a hand and / or a wrist of the wearer of the at least one electronic device.

5. The method of claim 1 , wherein determining a sensor reading comprises: A signal is received from one or more of the accelerometer and an additional sensor for detecting the onset of the at least one behavioral event, and wherein the additional sensor includes a gyroscope.

6. The method according to claim 1, further comprising: The learning engine is trained to predict the occurrence of the at least one behavioral event based on tracking one or more of gender, age, weight, socioeconomic status, timing information about the at least one behavioral event, information about the location of the at least one behavioral event, and vital sign information.

7. The method according to claim 6, further comprising: identifying features of an event time envelope using the learning engine, thereby defining the event time envelope; identifying gestures within the event time envelope using limits of the event time envelope; as well as The at least one behavioral event is derived from the gesture.

8. The method according to claim 1, further comprising: storing a learned data set in a memory storage device, wherein the learned data set includes data that can be used to detect a type of gesture based on training examples; as well as Using a classifier trained on the learned data set, considering the output of the classifier to determine the possible posture of the wearer, whereby the classifier can be used in a behavioral event detection system to detect the start of at least one behavioral event, wherein the learned data set includes data records related to previous postures of the wearer or other user, wherein the learned data set includes data related to sleep patterns, stress levels, daily interactions and recent activities or data related to one of sleep patterns, stress levels, daily interactions and recent activities.

9. The method of claim 1, wherein the at least one behavioral event is a feeding event.

10. A method of sensing wearer activity using at least one electronic device worn by a wearer, the wearer activity including at least one eating event, the method comprising: determining, using a processor, sensor readings, wherein at least one sensor reading is from an accelerometer of the at least one electronic device that measures movement of a body part of the wearer; determining a first gesture indicative of a potential eating event based on the sensor readings; upon determining the first gesture indicative of the potential eating event, activating at least one sensor or component of a food intake tracking system; collecting data or information related to the potential eating event from the at least one sensor or component after activation; Corresponding to the potential eating event being the identified eating event, providing storage of event-specific parameters initialized for the identified eating event and an indication of whether the first gesture indicates that the at least one electronic device is in an event state; While in the in-event state, determining a series of gestures associated with the identified eating event; deriving the event-specific parameters regarding the identified eating event from the series of gestures; as well as At least one of the rhythm and / or duration of the gestures is detected and the amount of eating steps is calculated therefrom.

11. The method of claim 10, wherein the rhythm corresponds to a function of one or more of rotation, velocity, acceleration and / or deceleration of an arm, hand and / or wrist.

12. The method of claim 10, wherein the identified eating event is a drinking event and the eating step is sipping.

13. The method according to claim 10, further comprising: The total estimated consumption is calculated based on multiple eating steps.

14. The method of claim 13, wherein the identified eating event is a drinking event, the eating step is a sip, and the total estimated consumption is an estimated hydration intake. The method of claim 13 , wherein the total estimated consumption is compared to a meal portion goal.

16. The method of claim 15, wherein the identified eating event is a drinking event, the eating step is a sip, and the eating portion goal is a hydration goal.

17. The method of claim 10, wherein calculating the amount of the eating step comprises: Measuring the period of time that the wearer's hand, utensil, or liquid container is near the wearer's mouth; as well as An estimate of the eating steps is generated using the value of the time period as input.

18. The method of claim 17, wherein the estimated amount is an estimated amount of a sip, and the sip amount is estimated based on at least an amount of rotation of the wearer's wrist.

19. The method according to claim 17, further comprising: The time corresponding to the peak of the pitch of the hand-to-mouth movement was calculated.

20. The method of claim 10, wherein the event-specific parameters include one or more of: a timestamp associated with the start of the identified eating event, a speed associated with the identified eating event, and / or a measurement of the amount of food consumed in the identified eating event.

21. The method according to claim 10, further comprising: The learning engine is trained to predict the occurrence of the identified eating events based on tracking one or more of gender, age, weight, socioeconomic status, timing information about the identified eating events, information about the location of the identified eating events, vital sign information, and hydration level information.

22. The method of claim 10, further comprising: One or more of a duration, a bite count, a sip count, an eating rate, or a drinking rate of the identified eating events is estimated.

23. The method of claim 10, further comprising: identifying features of the event time envelope using a learning engine, thereby defining the event time envelope; identifying gestures within the event time envelope using limits of the event time envelope; as well as A recognized eating event is derived from the gesture.

24. A method of sensing wearer activity using at least one electronic device worn by a wearer, the wearer activity comprising at least one behavioral event, the method comprising: determining, using a processor, sensor readings, wherein at least one sensor reading is from an accelerometer of the at least one electronic device that measures movement of the wearer's arm; determining a first gesture indicative of a potential event based on the sensor readings; determining, using the processor, a stored state, the stored state being a state from a set of states, the set of states comprising an in-event state and an out-of-event state; determining, using the processor and based on the first gesture, a start of the at least one behavioral event; upon detecting the onset of the at least one behavioral event, sending an activation signal to an image capture device to perform an action related to facilitating the capture of an image related to the at least one behavioral event; collecting, from the image capture device after activation, captured images associated with the at least one behavioral event; providing storage of event-specific parameters initialized for said behavioral event; When in the event state, determining, using the processor, a series of gestures associated with the behavioral event; as well as The event-specific parameters regarding the behavioral event are derived from the series of gestures, the event-specific parameters comprising a timestamp related to the start of the behavioral event.

25. The method of claim 24, wherein the action related to facilitating capture of an image comprises capturing the image.

26. The method of claim 24, wherein the action related to facilitating capture of an image comprises opening a camera application.

27. The method of claim 24, wherein the action related to facilitating capture of an image comprises unlocking an input screen of the image capture device.

28. The method of claim 24, wherein the image capture device is a smartphone.

29. The method of claim 24, wherein the at least one behavioral event is a feeding event.

30. The method of claim 24, further comprising: The learning engine is trained to predict the occurrence of the at least one behavioral event based on tracking one or more of gender, age, weight, socioeconomic status, timing information about the at least one behavioral event, information about the location of the at least one behavioral event, vital sign information, and hydration level information.

Images