Systems and methods for food analysis, personalized recommendation, and health management

Abstract

The present disclosure provides methods and systems for providing personalized food and health management recommendations. The method may include mapping a food product by abstracting information from data related to the food product to develop a food product ontology. The method may include collecting and aggregating a plurality of data sets related to food, health, or nutritional information of a user. The plurality of data sets may be provided in two or more different formats from multiple sources. The method may include converting the plurality of data sets into a standardized format that may be individualized for the user. The method may include applying a predictive model to the food body and the plurality of data sets of the user in the standardized format to determine an impact of food consumption of the user body.

Description

[0001] This divisional application is a divisional application of a Chinese patent application with application number 201980020361.3, application date March 21, 2019, and invention title "Systems and Methods for Food Analysis, Personalized Recommendations, and Health Management".

[0002] Cross - reference to related applications

[0003] This application claims priority to U.S. Provisional Patent Application No. 62 / 647,552, filed on March 23, 2018; U.S. Provisional Patent Application No. 62 / 783,100, filed on December 20, 2018; U.S. Patent Application No. 15 / 981,832, filed on May 16, 2018; and U.S. Patent Application No. 16 / 359,611, filed on March 20, 2019, which are hereby incorporated by reference in their entirety for all purposes. Background of the Invention

[0004] A large number of databases and services are available for providing food and nutrition advice. Examples of such databases and services include healthcare providers, food or nutrition manufacturers, restaurants, online and offline food recipes, and scientific articles. Whether for entertainment, beauty, medical, or other purposes, individuals inevitably have to rely on multiple information sources on a daily basis to make food and nutrition - related decisions.

[0005] Numerous attempts have been made to generate a collection of data on the nutritional information of commonly consumed foods and their impact on human health. However, these databases often struggle with inconsistency, unreliability, and generally low quality because the data is typically collected from user input or crowdsourced. Additionally, since many attempts are targeted at specific populations, regions, or food categories within a defined time period, the resulting databases are often fragmented in scope and time. This fragmentation limits their applicability. Moreover, these databases rely on different data sources (e.g., mobile devices, glucose monitors, social media, etc.) that are often incompatible with each other. In the absence of alternatives, individuals will continue to rely on limited and incomplete databases and / or services to piece together decisions regarding food and nutrition.

[0006] Therefore, there is a need for systems and methods that can continuously collect large amounts of data (e.g., ingredients in dishes, nutritional information, glucose levels, blood pressure, temperature, etc.) from discrete sources, analyze the data and reconstruct it into a common format, evaluate and predict the correlation between the foods consumed by an individual and biomarkers, and provide personalized nutrition recommendations based on the health and metabolic status of an individual at any given time. Summary of the Invention

[0007] This disclosure provides a computer-implemented method for mapping food. Exemplary embodiments of the method involve obtaining food-related data from a plurality of different sources; and abstracting information from the food-related data using one or more algorithms, the one or more algorithms including at least one machine learning algorithm for developing a food ontology.

[0008] This disclosure also provides a computer-implemented system for determining the impact of the consumption of one or more foods on a user's body. Exemplary embodiments of the system involve: (a) a device and a data aggregator configured, with the assistance of one or more processors, to collect and aggregate a plurality of data sets from a plurality of application programming interfaces (APIs), where the plurality of data sets include (1) data indicating the one or more foods consumed by the user and (2) data indicating physiological inputs associated with the user, and where the plurality of data sets are provided in two or more different formats; and converting the plurality of data sets in the two or more different formats into a standardized format individualized for the user. Exemplary embodiments of the system further include: (b) an analysis engine configured, with the assistance of one or more processors, to determine the impact of the consumption of the one or more foods on the user's body by applying a prediction model to a plurality of standardized data sets including (1) the data indicating the one or more foods consumed by the user and (2) the data indicating the physiological inputs associated with the user, and by partially using (3) information about the one or more foods from a food ontology consumed by the user; and generating a plurality of personalized food and health metrics for the user based on the results output by the prediction model. The plurality of personalized food and health metrics for the user are configured to be displayed on an electronic display of a user device in the form of a set of graphical visual objects.

[0009] The present disclosure also discloses a computer-implemented method for determining the impact of the consumption of one or more foods on a user's body. Exemplary embodiments of the method involve: (a) with the help of a device and a data aggregator: collecting and aggregating multiple data sets from multiple application programming interfaces (APIs), wherein the multiple data sets include (1) data indicating the one or more foods consumed by the user and (2) data indicating physiological inputs associated with the user, and wherein the multiple data sets are provided in two or more different formats; and converting the multiple data sets in the two or more different formats into a standardized format individualized for the user. Exemplary embodiments of the method also involve: (b) with the help of an analysis engine: determining the impact of the consumption of the one or more foods on the user's body by applying a prediction model to multiple standardized data sets including (1) the data indicating the one or more foods consumed by the user and (2) the data indicating the physiological inputs associated with the user, by partially using (3) information about the one or more foods from a food ontology consumed by the user; and generating multiple personalized food and health metrics for the user based on the results output by the prediction model. The multiple personalized food and health metrics for the user are configured to be displayed in the form of a set of graphical visual objects on an electronic display of a user device.

[0010] The present disclosure also discloses a tangible computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform a computer-implemented method for determining the impact of the consumption of one or more foods on a user's body. Exemplary embodiments of the method involve: (a) with the help of a device and a data aggregator: collecting and aggregating multiple data sets from multiple application programming interfaces (APIs), where the multiple data sets include (1) data indicating the one or more foods consumed by the user and (2) data indicating physiological inputs associated with the user, and where the multiple data sets are provided in two or more different formats; and converting the multiple data sets in the two or more different formats into a standardized format individualized for the user. Exemplary embodiments of the method also involve: (b) with the help of an analysis engine: determining the impact of the consumption of the one or more foods on the user's body by applying a prediction model to multiple standardized data sets including (1) the data indicating the one or more foods consumed by the user and (2) the data indicating the physiological inputs associated with the user, by partially using (3) information about the one or more foods from a food ontology consumed by the user; and generating multiple personalized food and health metrics for the user based on the results output by the prediction model. The multiple personalized food and health metrics for the user are configured to be displayed in the form of a set of graphical visual objects on an electronic display of a user device.

[0011] The present disclosure also discloses a dietary glucose monitor. Exemplary embodiments of the dietary glucose monitor include a food analysis module in communication with a glucose level monitor. The food analysis module is configured to (1) analyze data indicating foods consumed by a user and (2) determine the impact of an individual food on the user's glucose level based on changes in the user's glucose level as measured by the glucose level monitor.

[0012] The present disclosure also discloses a method for determining the impact of a food on a user's glucose level. Exemplary embodiments of the method involve: (a) providing a food analysis module in communication with a glucose level monitor; (b) analyzing, using the food analysis module, data indicating foods consumed by the user; and (c) determining, using the food analysis module, the impact of an individual food on the user's glucose level based on changes in the user's glucose level as measured by the glucose level monitor.

[0013] The present disclosure also provides a computer program product having a non-transitory computer-readable medium encoded with computer-executable code. The computer-executable code is adapted to be executed to implement the methods summarized in the preceding paragraphs.

[0014] Incorporated by reference

[0015] All publications, patents, and patent applications mentioned in this specification are hereby incorporated by reference to the same extent as if each individual publication, patent, or patent application was specifically and individually indicated to be incorporated by reference. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The novel features of the disclosure are set forth with particularity in the appended claims. A better understanding of the features and advantages of the disclosure will be obtained from the following detailed description that illustrates exemplary embodiments, in which the principles of the disclosure are utilized, and the accompanying drawings:

[0017] Figure 1 shows an ecosystem for food analysis and personalized health management according to some embodiments;

[0018] Figure 2 shows examples of sources of food-related data according to some embodiments;

[0019] Figure 3 is a flowchart of a method of food analysis according to some embodiments;

[0020] Figure 4 shows an exemplary table of a training set for a food analysis algorithm according to some embodiments;

[0021] Figure 5 shows an exemplary table of results from testing a food analysis algorithm according to some embodiments;

[0022] Figure 6 shows an exemplary vocabulary list for a food analysis algorithm according to some embodiments;

[0023] Figure 7 shows an exemplary table of training sets for different food analysis algorithms according to some embodiments;

[0024] Figure 8 shows an exemplary pipeline of a food tagger according to some embodiments;

[0025] Figure 9 shows a second exemplary pipeline of a food tagger according to some embodiments;

[0026] Figure 10Shows a third exemplary pipeline of a food marker according to some embodiments;

[0027] Figure 11 Shows an exemplary table of a problem solver and its statistical analysis according to some embodiments;

[0028] Figures 12A - 12C Shows the abstraction and classification of information from consumer food packaging according to some embodiments;

[0029] Figure 13 Shows an exemplary table of abstracted and classified consumer food packaging according to some embodiments;

[0030] Figure 14 Shows an exemplary restaurant menu for food analysis according to some embodiments;

[0031] Figure 15 Shows an exemplary table of information abstracted from a restaurant menu according to some embodiments;

[0032] Figure 16 Shows an exemplary two - dimensional graphical representation of a food ontology according to some embodiments;

[0033] Figures 17A - 17D Shows an exemplary window for food image recording of a GUI - based software interface according to some embodiments;

[0034] Figures 18A - 18C Shows an exemplary window for speech recognition analysis of a GUI - based software interface according to some embodiments;

[0035] Figure 19A and 19B Shows a parallel comparison of foods with similar appearance but substantially different calorie content according to some embodiments;

[0036] Figure 20 Shows the features of an insights and recommendation engine according to some embodiments;

[0037] Figure 21 is a graph of the blood glucose levels of two individuals;

[0038] Figure 22 Shows potential entity partners leveraging embodiments of the present disclosure;

[0039] Figure 23 is a graph of an individual's blood glucose level as a function of time according to some embodiments;

[0040] Figure 24A and 24BShows the classification of foods based on the impact of foods on biomarkers, according to some embodiments;

[0041] Figures 25A - 25C Shows an exemplary window for personalized recommendations on menu items based on a GUI-based software interface, according to some embodiments;

[0042] Figure 26 Shows an exemplary window for blood glucose recording based on a GUI-based software interface, according to some embodiments;

[0043] Figure 27 Shows an exemplary window for recommendations based on automated blood glucose recording based on a GUI-based software interface, according to some embodiments;

[0044] FIG. 28 is a flowchart of a method for modeling the interaction of glucose and insulin in the body, according to some embodiments;

[0045] Figure 29 Is a graph plotting the measured and estimated blood glucose levels, according to some embodiments;

[0046] Figure 30 Is a graph of the delivery of exogenous insulin into the body, according to some embodiments;

[0047] Figure 31 Shows an exemplary fit of a glucose absorption and insulin assimilation model, according to some embodiments;

[0048] Figures 32A - 32B Shows an exemplary window of a GUI-based software interface showing a prediction of a dining pattern, according to some embodiments;

[0049] Figures 33A - 33C Shows an exemplary window of a GUI-based software interface showing multiple features, according to some embodiments;

[0050] Figures 34A - 34C Shows an exemplary window of a GUI-based software interface showing a comprehensive report via an insights and recommendations engine, according to some embodiments;

[0051] Figure 35 Illustrates an exemplary calibration kit, according to some embodiments;

[0052] Figure 36 Shows an exemplary window of a healthcare provider's portal, according to some embodiments;

[0053] Figure 37 (Parts A to F) Shows an exemplary window of a mobile application showing the initial setup of the mobile application, according to some embodiments;

[0054] Figure 38 (Parts A - C) show an exemplary window of a mobile application showing a baseline data set's components, according to some embodiments;

[0055] Figure 39 (Parts A - D) show an exemplary window of a mobile application showing a food image recording interface, according to some embodiments;

[0056] Figure 40A and 40B show an exemplary window of a GUI - based software interface showing an analysis report of a user's data, according to some embodiments;

[0057] Figure 41 show an exemplary network layout, according to some embodiments;

[0058] Figure 42 show an example of a standard deviation image region, according to some embodiments;

[0059] Figure 43 show an example of a rectangular detection image region, according to some embodiments;

[0060] Figure 44 show an example of stalactite cave - style OCR text type separation, according to some embodiments;

[0061] Figure 45 show an exemplary image of a packaged food label from which nutritional information is to be extracted;

[0062] Figure 46 show a nutrient NLP score histogram based on nutritional information extraction from over 40,000 food products, according to some embodiments;

[0063] Figure 47 show an ingredient NLP score histogram based on nutritional information extraction from over 40,000 food products, according to some embodiments;

[0064] Figure 48 show an allergen NLP score histogram based on nutritional information extraction from over 40,000 food products, according to some embodiments;

[0065] Figure 49 show a graph of image logo training results, according to some embodiments;

[0066] Figure 50 show a graph of text logo recognition results, according to some embodiments;

[0067] Figure 51A and 51BShows per-signature results for multiple different signatures according to some embodiments;

[0068] Figure 52 Shows a model for classifying whether a food item is soy-free according to some embodiments;

[0069] Figure 53 Shows an example of a learning curve according to some embodiments;

[0070] Figure 54 Illustrates a graphical user interface (GUI) for managing a training set for food classification according to some embodiments;

[0071] Figure 55 Illustrates a histogram of food classification success by confidence according to some embodiments;

[0072] Figure 56 Shows an exemplary graphical representation of a food ontology related to an ingredient taxonomy;

[0073] Figure 57A and 57B Shows an exemplary process for generating statistical correlations between foods using one or more computer algorithms;

[0074] Figure 58A and 58B Shows an exemplary process for grouping meals for meal generalization;

[0075] Figure 59 Shows an ecosystem for food analysis and personalized glucose level management of a dietary glucose monitor according to some embodiments;

[0076] Figure 60 Shows another ecosystem for food analysis and personalized glucose level management of a dietary glucose monitor according to some embodiments;

[0077] Figure 61A and 61B Shows an exemplary window for food image rating of a GUI-based software interface according to some embodiments;

[0078] Figure 62A and 62B Shows an exemplary window of a GUI-based software interface that shows an analysis report of a user's data according to some embodiments;

[0079] Figure 63 Shows an exemplary window of a GUI-based software interface that shows a video between a user and another person;

[0080] Figure 64Shows an exemplary window of a GUI-based software interface that depicts message exchange between a user and a coach;

[0081] Figure 65 Shows an exemplary calibration kit according to some embodiments; and

[0082] Figure 66 Shows a computer system programmed or otherwise configured to implement the methods provided herein. Detailed Description

[0083] Reference will now be made in detail to some embodiments of the present disclosure, examples of which are illustrated in the accompanying drawings. Wherever appropriate, the same reference numerals will be used throughout the drawings to refer to the same or like parts.

[0084] Introduction

[0085] Not only can two individuals react differently to the same food, but a single individual can also react differently to the same food at different times. However, the current state of food and nutrition services continues to rely on previously generated static databases to provide general, non-customized advice on food and health management.

[0086] The currently available databases for food and nutrition are inconsistent due to user input dependence, limited in their scope and time scale, and generally incompatible with each other. As a result, there is currently a lack of systems and methods that can curate fragmented food-related information into a single format, evaluate the relationship between food and biomarkers for each individual, and provide personalized nutrition recommendations as an individual's lifestyle continues to evolve.

[0087] The present disclosure can provide the above solutions by: (1) creating and / or using a food ontology that continues to be updated from various sources (e.g., from the Internet, pre-existing databases, user input, etc.) to organize and analyze any available information on all food types (e.g., primary foods, packaged foods, recipes, restaurant dishes, etc.); (2) generating a personalized data network in multiple data collection devices and services (e.g., mobile devices, glucose sensors, healthcare provider databases, etc.) to integrate any available information on biomarkers that may be affected by metabolism (e.g., sleep, exercise, blood tests, etc.) or may affect metabolism; and (3) connecting the food ontology with the personalized data network to gain insights into how food can affect each individual and generate personalized food, healthcare, and health recommendations for each individual.

[0088] Examples of food include primary food (e.g., fruits, vegetables, meat, salt, sugar, etc.), packaged food, and restaurant dishes. Food can also include beverages (e.g., liquids, water, coffee, tea, alcohol, juice, smoothies, powdered drinks, etc.).

[0089] Next, various embodiments of the present disclosure will be described with reference to the accompanying drawings.

[0090] Platform

[0091] Embodiments of the present disclosure can be implemented as an ecosystem of a device, one or more databases, and a platform for providing personalized nutritional insights to users.

[0092] Figure 1 An ecosystem 100 according to some embodiments is shown. In one aspect, the ecosystem 100 can include a platform 200. The platform 200 can include three components: a food analysis system 210, a device / data hub 220, and an insights and recommendations engine 230. The three components in the platform 200 can be separate or interconnected with each other. The ecosystem 100 can also include devices 110. The devices 110 can include wearable devices 112 (e.g., smartwatches, fitness trackers, etc.), mobile devices 114 (e.g., cell phones, smartphones, recorders, etc.), medical devices 116 (e.g., glucose monitors, insulin pumps, heart rate monitors (HRV), skin temperature sensors, etc.), or more. One or more of the devices 110 (e.g., the mobile device 114) can access one or more social network service (SNS) accounts (e.g., Facebook, Instagram, Twitter, Snapchat, Reddit, Yelp, Google+, Tumblr, etc.), thereby allowing the platform 200 to obtain any information or data of the user (e.g., images, videos, texts, audio recordings, etc.). The devices 110 can communicate with each other. The platform 200 can communicate with the devices 110. The platform 200 can communicate with the Internet 120 and one or more databases 130 (e.g., other food, nutrition, or healthcare providers). In some cases, one or more of the databases 130 can include a genomic database for retrieving genomic information of the user. Examples of genomic databases can include but are not limited to 23andMe, deCODE Genetics, Gene by Gene, GenePlanet, DNA Ancestry, uBiome, and healthcare providers. The platform 200 can also communicate with one or more additional databases 240 to store any data or information collected and generated by the platform 200. One or more of the additional databases 240 can be a collection of secure cloud databases.

[0093] The food analysis system 210 can create, update, and / or utilize a food ontology. The food analysis system 210 can be connected to various data sources, including devices 110 (e.g., wearable device 112, mobile device 114, etc.), the Internet 120, and existing databases 130. The food analysis system 210 can act as a content management system to continuously receive, analyze the nutritional information of all food types (e.g., primary foods, packaged foods, recipes, restaurant dishes, etc.) and organize the nutritional information into the food ontology.

[0094] The device / data aggregator 220 can generate a personalized data network of users among devices 110. The device / data aggregator 220 can automatically aggregate the biomarker and health data (e.g., sleep, exercise, blood tests, genetic tests, etc.) of users from multiple application programming interfaces (APIs) and healthcare provider databases.

[0095] The insights and recommendations engine 230 can communicate with the food analysis system 210 and the device / data aggregator 220. Thus, the engine 230 can create and analyze any correlations between the information from the food ontology and the information from the personalized data network. The engine 230 constitutes the "brain" of the platform 200 and acts as the user's food global positioning system (food GPS). The engine 230 can address the user's own daily nutritional needs based on how different foods react to the user's biomarkers at different times. Therefore, the engine 230 can generate personalized food, healthcare, and health recommendations for the user. The engine 230 can communicate with the device 110 directly or by communicating with the device / data aggregator 220. In one example, the engine 230 can use the device 110 to relay the recommendations to the user in a visible format.

[0096] One or more graphical user interfaces (GUI; Figure 1The platform 200 is implemented (not shown in the figure) to enable the user to select and adopt the features of the following three components: the food analysis system 210, the device / data hub 220, and the insights and recommendations engine 230. Generally, as opposed to a text-based interface, bright labels for input, or text navigation, a GUI can be the type of interface that allows a user to interact with an electronic device through graphical icons and visual indicators such as secondary symbols. The GUI can be presented on a display screen of the user device. Actions in the GUI can be performed by directly manipulating graphical elements. In addition to computers, the GUI can also be presented on handheld devices such as smartphones, portable media players, gaming devices, and office and industrial equipment. The GUI of the platform 200 can be provided in software, software applications, web browsers, etc. The GUI can be displayed on the user device. The GUI can be set up through a mobile application. One or more GUIs of the present disclosure can be referred to as platform GUIs. The platform GUI can communicate with other GUIs such as the wearable device 112, the mobile device 114, the medical device 116, the smart home device 118 (e.g., smart refrigerator), etc. The end users of the platform can include infants, teenagers, college students, adults, healthy individuals, patients, participants in health programs, insured persons of various providers, etc.

[0097] Food analysis system

[0098] The food analysis system 210 can map food by abstracting information from food-related data to develop a food ontology. The food analysis system 210 can continuously receive, analyze nutritional information from data related to all food types, and organize the nutritional information into the food ontology. The food ontology can also be referred to as a network of foods. Food-related data can be obtained from multiple sources. The multiple sources can include the device 110 (e.g., the wearable device 112, the mobile device 114, etc.), the Internet 120, and one or more existing databases 130. Examples of sources of food-related data are shown in Figure 2 The examples of sources of food-related data can include food manufacturers, restaurants, grocery stores, cafeterias, airline food, healthcare providers, etc. One or more existing databases 130 can be the food databases of competitors.

[0099] Food-related data can be classified into multiple different categories, such as 2, 3, 4, 5, or more categories. In some cases, a category can include one or more sub-categories.

[0100] In some embodiments, the food-related data analyzed by the food analysis system 210 can be divided into four different categories, such as (1) primary foods, (2) recipes, (3) packaged foods, and (4) restaurant dishes. Food can include beverages (e.g., water, coffee, tea, alcohol, etc.).

[0101] The primary foods in category (1) can include foods and ingredients for general consumption. The primary foods can be further divided into two or more components, such as (a) primary ingredients and (b) primary recipes. A primary ingredient is a food item that contains a single ingredient other than water. Examples of primary ingredients can include bananas, almonds, frozen blueberries, raw blueberries, etc. A primary recipe is an abstraction of a commonly consumed food that contains more than one single ingredient. Examples of primary recipes can include pad thai, chicken lo mein, French fries, etc.

[0102] The recipes in category (2) can include foods that include multiple ingredients along with preparation instructions. The recipes in category (2) can include only specific recipes and may be fundamentally different from the primary recipes in category (1). For example, the general "pad thai" in category (1) is an abstraction of the concept of a dish called "pad thai" (which typically includes noodles, oil, peanuts, etc.) and is thus a primary recipe. In contrast, different pad thai recipes (such as those found on various internet sources) are specific recipes and are implementations of the primary recipe "pad thai" and are thus (non-primary) recipes. Other examples of recipes in category (2) can include multiple recipes for fettuccine alfredo from different sources.

[0103] The packaged foods in category (3) can include foods that are sold in a packaged form with a barcode, such as a Clif bar chocolate with a specific Universal Product Code (UPC). Most packaged foods can be considered to contain the recipes of food manufacturers because recipes are required to prepare the packaged foods. However, since the amount of each ingredient and the exact preparation instructions are usually not provided or listed on the packaged foods, the category of packaged foods can be separated from the recipes. A separate category may be needed because packaged foods and recipes usually need to be analyzed differently by machine learning models disclosed elsewhere in this document.

[0104] The restaurant dishes in category (4) can include menu items from a restaurant, such as a Big Mac from McDonald's. In fact, these restaurant dishes are created according to the recipes within the restaurant itself. However, in most restaurants, the exact ingredient list and preparation instructions are not provided to the customers or are unknown to the customers. Therefore, restaurant dishes are different from recipes because they need to be processed (analyzed) in a different way from recipes.

[0105] Typically, the nutrition-related data about food is usually partial (incomplete) because some information in the information is usually lost or difficult to obtain. For example, the primary food in category (1) may lack labels, and it may be challenging to add new food items and new nutrients. The packaged food in category (2) usually does not display the amounts of different ingredients. Similarly, most food labels on packaged food can contain only a limited number of nutrients (e.g., 10 - 14) when there may be more nutrients. In addition, the data of food labels can be captured in image form, and there may be a difference between the actual content of the packaged food and the information on the food label. For the recipes in category (3), there are usually no labels on the recipes, and thus the nutrition information may be inaccurate. Similarly, for the restaurant dishes in category (4), the restaurant menus usually do not have labels and may not contain nutrition information. The ingredients in restaurant dishes are usually part of the free text in the description of the restaurant dishes. In addition, the data of restaurant dishes is usually captured in image form (e.g., in the form of an image of the menu), and there are significant differences in the way the dishes are described or depicted on the restaurant menu.

[0106] In view of this, each of the above food categories (1)-(4) may individually have certain gaps or limitations. However, these gaps or limitations can be addressed by the algorithms described herein, enabling the food analysis system to generate a complete understanding of each food object by leveraging the data within and between different categories. In most cases, knowledge of the ingredients and amounts of each food can allow for the determination of additional data about each food.

[0107] The food analysis system 210 can utilize one or more algorithms including at least one machine learning algorithm to abstract information from food-related data. The food analysis system can classify the abstracted data into one or more categories of a food ontology. The categories can be an abstract layer. The one or more algorithms can include natural language processing (NLP), a computer vision system, or a statistical model. The computer vision system can include artificial intelligence (AI), deep learning, or optical character recognition (OCR) capabilities. The combination of the computer vision system and NLP can convert an image of consumer-packaged food or an image of a restaurant menu into structured data, which can be analyzed and classified into the food ontology.

[0108] The category or abstraction level of a food in a food ontology can include a name, description, user rating, image, characteristics (such as dietary requirements, allergies, cuisine, flavorings, texture, etc.), ingredient breakdown (type and amount), nutrient breakdown (type and amount), processing information, and food geographical location and / or availability information. A food analysis system can use at least one machine learning algorithm to generate additional abstraction levels or metadata for a food. The additional abstraction levels or metadata of a food can be incorporated into the food ontology. A food item can have one or more abstraction levels. The abstraction levels can be used to describe one or more food items.

[0109] Dietary requirements can be an abstraction level of a food ontology. Dietary requirements can include a vegetarian diet, a lacto-ovo vegetarian diet, a pescatarian diet, a lacto vegetarian diet, an ovo vegetarian diet, a vegan diet, or any other diet, depending on or based on the popularity in a particular region. Vegetarians generally may not eat meat or fish. Lacto-ovo vegetarians may avoid all animal meat (both meat and fish). Pescatarians may eat fish but not meat. Lacto vegetarians can consume dairy products but not eggs. Ovo vegetarians can consume eggs but not dairy. Vegans may avoid all animal-based foods, including honey.

[0110] Dietary requirements can include a gluten-free diet. A gluten-free diet can be important for individuals with celiac disease (celiac sprue), a severe autoimmune disorder of gluten malabsorption that can damage the small intestine. Celiac disease may affect 1 in 100 people worldwide. Examples of foods not suitable for a gluten-free diet include wheat, barley, rye, and oats. Examples of common processed foods containing wheat, barley, rye, or oats can include products with malt, cereals, cold cuts, gravy, flavored rice mixes, trail mixes, and imitation fish or bacon.

[0111] Dietary requirements can include a diabetic diet for individuals with type 1 or type 2 diabetes. Diabetes can be a disease that causes too much sugar (such as glucose) in the blood. A diabetic diet can include foods rich in fiber, including fruits, vegetables, whole grains, legumes (beans, lentils, and chickpeas), and low-fat dairy products. A diabetic diet can include fish rich in omega-3 fatty acids, including salmon and mackerel. A diabetic diet may not include foods high in sugar (carbohydrates). For example, for an individual with diabetes on a 1,600-calorie diet, no more than about 50% of those calories can come from carbohydrates.

[0112] Dietary requirements can include various religious diets, including halal and kosher diets. Halal diets can be those permitted or lawful according to traditional Islamic dietary laws. In one example, halal diets may not include pork or pork products. Kosher diets can be those permitted or lawful according to a set of Jewish religious dietary laws known as kashrut. In one example, kosher diets may not include hares, hyraxes, camels, and pigs. The food analysis system 210 can communicate with several food certification programs, including the Islamic Food and Nutrition Council of America (IFANCA), Kosher Supervision of America (KSA), etc., to continuously monitor and update its algorithms for religious diets.

[0113] Dietary requirements can include lactose-free diets for individuals with lactose intolerance. Lactose intolerance can be a condition associated with a reduced ability to digest lactose, a sugar found in dairy products. Individuals with lactose intolerance will exhibit symptoms including abdominal pain, bloating, diarrhea, flatulence, and nausea after consuming dairy products or milk without any medical treatment (such as lactase). Therefore, the food analysis system 210 can analyze and report whether a food item can contain milk or dairy ingredients.

[0114] Dietary requirements can include organic food diets. Organic foods can include products from animals not given any antibiotics or growth hormones. Organic foods can include plants that do not use conventional pesticides or fertilizers made from synthetic ingredients. Examples of terms used on the labels of commercially available organic products can include "100% organic", "organic", and "made with organic ingredients".

[0115] Dietary needs can include non-Genetically Modified Organism (non-GMO) diets. GMO ingredients can be plants or animals created through genetic engineering in a laboratory environment beyond traditional hybridization. Genetic engineering can combine genes from different species to create new species. Since organic foods may be prohibited from containing one or more GMO ingredients, organic foods are generally non-GMO foods. Examples of terms used on the labels of commercially available non-GMO products can include "Non-GMO Project Verified". The "Non-GMO Project Verified" label can be certified by the Non-GMO Project.

[0116] Dietary requirements can include other diets preferred by the user, such as the Atkins diet, the Zone diet, the ketogenic diet, and the raw food diet. The Atkins diet can be a weight loss program envisioned by Robert Atkins. The Atkins diet can be a first variation of a low-carbohydrate diet. The Zone diet can be a second variation of a low-carbohydrate diet. The Zone diet may require a specific food ratio in each meal of 40% carbohydrates, 30% fat, and 30% protein. The Zone diet can recommend eating five times a day to help prevent overeating. The ketogenic diet can be used for children with epilepsy. The ketogenic diet can be a third variation of a low-carbohydrate diet. The ketogenic diet can encourage a high-fat diet. Ketosis may cause the breakdown of fat deposits in the body to be used as fuel and produce substances called ketones through a process called ketosis. The ketogenic diet can encourage the consumption of oils from avocados, coconuts, Brazil nuts, olives, and oily fish. The raw food diet can encourage the consumption of unprocessed foods and beverages. The raw food diet may not include food products with artificial food preservatives, which include calcium propionate, sodium nitrate, butylated hydroxyanisole (BHA), and butylated hydroxytoluene (BHT). The food analysis system 210 can use at least one machine learning algorithm to detect or estimate the presence of one or more artificial food preservatives from consumer-packaged foods.

[0117] Allergies can be an abstraction layer of the food entity. For information on food allergies, the food analysis system 210 can communicate with existing databases such as the Food Allergy Research and Resource Program (FARRP) to continuously monitor and update its algorithms to abstract or classify information from allergenic foods. Allergenic foods can be broken down into one or more groups, including wheat or gluten (barley, corn, maize, oats, rice, rye, wheat, other gluten-containing grains, etc.), lactose or dairy products (milk, goat milk, sheep milk, etc.), eggs (hen, goose, duck), tree nuts (almonds, Brazil nuts, cashews, chestnuts, hazelnuts, macadamia nuts, pecans, pistachios, walnuts, etc.), legumes (chickpeas, lentils, lupins, peanuts, peas, etc.), fish (Alaskan cod, carp, cod, dogfish, mackerel, salmon, sole, tuna, etc.), and crustacean shellfish (crab, lobster, shrimp, etc.). Additional allergenic foods can include fruits (acerola cherry, apple, apricot, banana, cherry, coconut, date, fig, grape, mango, melon, orange, peach, pineapple, etc.) and vegetables (asparagus, avocado, carrot, celery, etc.).

[0118] Food flavorants can be an abstract layer of the food body. Food flavorants can include the sensory impressions of the food. Food flavorants can be determined by the chemical senses of taste and smell. Some examples of taste can include sweet, sour, bitter, salty, and savory (also known as umami). Some examples of odors or scents distinguishable by the human olfactory system can include aromatic (e.g., floral and perfume), fruity (all non-citrus fruits), citrus (e.g., lemon, lime, orange), woody or resinous (e.g., pine or freshly cut grass), chemical (e.g., ammonia, bleach), sweet (e.g., chocolate, vanilla, caramel), minty (e.g., eucalyptus and camphor), roasted or nutty (e.g., popcorn, peanut butter, almond), pungent (e.g., blue cheese, cigar smoke), and putrid (e.g., spoiled meat, sour milk). Alternatively or additionally, food flavorants can be determined by temperature ranges (e.g., hot, room temperature, cold, frozen, etc.).

[0119] As substances in food, food flavorants can be edible spices or seasonings. Some examples of natural or artificial edible spices for taste can include glutamic acid, glycine, guanylic acid, inosinic acid, disodium 5'-ribonucleotide, acetic acid, ascorbic acid, citric acid, fumaric acid, lactic acid, malic acid, phosphoric acid, and tartaric acid. Some examples of natural or artificial edible spices for scent can include diacetyl, acetyl propionyl, acetoin, isoamyl acetate, benzaldehyde cinnamaldehyde, ethyl propionate, methyl anthranilate, limonene, ethyl decadienoate, allyl hexanoate, ethyl maltol, ethyl vanillin, methyl salicylate, and matricin.

[0120] Food flavorants can also include color. The color of food can affect an individual's expectations of one or more flavorants of the food. In one example, adding more red color to a beverage can increase the perceived sweetness of the beverage. The food analysis system 210 can label color in one or more ways. Color can be labeled by standard nomenclature, including red, orange, yellow, green, blue, navy, purple, black, etc. Color can be labeled in the form of a position in a predefined color palette or color wheel. Alternatively or additionally, color can be labeled according to the characteristic absorption curve of the food in at least a portion of the electromagnetic spectrum. The characteristic absorption curve can be defined by the position and intensity of at least a portion of the electromagnetic spectrum. At least a portion of the electromagnetic spectrum can be the visible spectrum. The visible spectrum can include electromagnetic radiation in the range of approximately 400 nanometers to approximately 750 nanometers. Labeling color in the form of position and intensity within the electromagnetic spectrum can avoid bias towards users with or without color blindness.

[0121] Nutritional characteristics can be an abstract layer of the food entity. Nutritional characteristics can include one or more dietary goals or constraints recommended to or defined by the user. Dietary goals can include low fat, high fat, and high calcium. Nutritional characteristics can include nutritional recommendations for pregnancy. Nutritional recommendations for pregnancy can include: pasteurized foods to prevent Listeria infection; foods with high folate, calcium, or iron; foods or beverages with low caffeine; and avoiding or consuming no more than 6 ounces per week of fish with high levels of mercury. Fish with high levels of mercury can include king mackerel, marlin, swordfish, tilefish, and tuna.

[0122] Texture can be an abstract layer of the food entity. Texture can include: soft, firm, creamy, brittle, crispy, crunchy, fragile, tender, chewy, tough, thick, thin, viscous, airy, fluffy, greasy, sticky, moist, pasty, lumpy, pulpy, granular, etc.

[0123] The food entity generated by the food analysis system 210 can be compatible with any suitable food database containing an existing food database. General databases of foods include the United States Department of Agriculture (USDA) database, Open Food Facts, and ItemMaster. Databases of packaged foods include Nutritionix, Fatsecret, and Myfitnesspal. Databases of recipes include Yummly, BBC Good Food, Allrecipes, The Kitchn, EatingWell, and MyRecipes. Databases of restaurant dishes include HealthyDiningFinder, Nutritionix, MyNetDiary, FatSecret, HealthyOut, and OpenMenu. Alternatively or additionally, the food entity generated by the food analysis system 210 can supplement the limitations of existing food databases. Due to being fully or partially dependent on user-generated and reported content, the data quality of existing food databases may be low. The scope of existing food databases may be limited because they may not cover all food categories and / or were targeted at specific populations and / or geographical locations during data collection. Similarly, due to the dependence on data collected in the past, existing food databases may be static or outdated. Additionally, existing food databases may not be able to maintain a robust entity of foods with detailed abstract layers (such as characteristics, ingredient breakdown, etc.).

[0124] Figure 3It is a flowchart of method 300 of food analysis system 210. Food analysis system 210 can be connected to the Internet 310 and collect food-related images. The food can include consumer-packaged food. The images can include front and back pictures of Immaculate Bakery gluten-free chocolate chunk cookies 315. The images can include nutrition label stickers. Food analysis system 210 can also be configured to connect to the user's mobile device 320 and collect food-related images 325 taken or saved by the user. Food analysis system 210 can also collect food-related text (such as publications, text messages, etc.) from the Internet 310, mobile device 320 or other sources. Food analysis system 210 can use various algorithms including OCR to automatically separate food-related text information 330 from images 315, 325. Food analysis system 210 can use NLP and other machine learning algorithms to parse the collected food-related text into a structured format 340, analyze its characteristics 350 and verify that the result analysis is correct 360. Food analysis system 210 can use the verified characteristics to map the food in the food ontology.

[0125] In some embodiments, the food analysis system can perform optical character recognition (OCR) to extract text information from images. The food analysis system can also implement a convolutional neural network to extract information from various icons that often appear on packaged food. Examples of OCR techniques used in the embodiments of the present disclosure are described below.

[0126] Nutrition information extraction as described herein can include extracting the nutrition label, ingredients, or allergens of a consumer product using a picture of the product. This can be implemented by first dividing the image into multiple sub-images containing regions with text using, for example, an image cropping algorithm. The image cropping algorithm can include methods for extracting the parts of the image containing text. The method implemented by the image cropping algorithm can include the following steps: (1) searching for and extracting rectangles in the image; (2) calculating the standard deviation image and pulling regions with a large standard deviation; (3) removing overlapping regions and calculating the difference between the standard deviation regions overlapping with the rectangular regions; (4) returning a list of the regions pulled from the image. Figure 42 An example of a standard deviation image region is shown, and Figure 43 An example of a rectangle detection image region is shown. Referring to Figure 42 , the standard deviation image region can include multiple free-form contours 4202 around the text and the blank space 4204 therebetween. Referring to Figure 43 , the rectangle detection image region includes multiple rectangular boxes 4302 around the text. Therefore, regions (sub-images) of various shapes and sizes can be cropped from the original image.

[0127] Next, the OCR algorithm can be applied to each of the cropped sub-images in the cropped sub-images. Words related to allergens, ingredients, or nutritional labels can be searched for in the paragraphs of the extracted text.

[0128] For allergens and ingredients, their spellings can first be corrected using a specialized spell checker. The spell checker can be used to correct misspelled words by OCR. The probability of each word is determined using its frequency in the training set, where the probability is set, for example, by Zipf's law, which states that given a certain corpus of natural language discourse, the frequency of any word is inversely proportional to its rank in the frequency table. For each word detected by OCR, the food analysis system can check whether the word is in the stored vocabulary. If the word is not in the stored vocabulary, the following options may be available: (1) Assume a spelling error and look for the top matching predetermined number (e.g., the top eight matches) in the vocabulary and select the match with the highest probability, or (2) Assume the detected word is the result of concatenating two different words by OCR and then select the split that provides the highest probability. After spelling correction, each extracted text can be divided into sections for ingredients and allergens by maximizing its confidence using, for example, the stalactite cave algorithm. The stalactite cave algorithm can be used to detect the parts of the text that contain paragraphs of ingredients and allergens. The stalactite cave algorithm can be implemented by training a model to classify expressions into three categories, e.g., (i) ingredients, (ii) allergens, and (iii) others. The model can include a tfidf transformation and a logistic regression classifier. The confidence of every three words can be checked using the model. Dynamic programming can be used to assert the parts (ingredients and allergens) that contain each group such that the area under the confidence graph is maximized. Figure 44 An example of stalactite cave OCR text type separation is shown. Figure 44 The x-axis of the graph shows the index of the words in the text (e.g., for a food dish). Figure 44 The y-axis of the graph shows the probability that the word pair starting from the index belongs to each class (e.g., ingredients, allergens, etc.). After dividing each extracted text into sections for ingredients and allergens, the paragraph with the highest confidence is selected as the most relevant for allergens and ingredients.

[0129] For nutritional labels, the food analysis system described herein can assert whether each strain contains a nutritional label. Next, the strain is divided into a name (by fitting to a word bank), an amount, and a measure. Then, the paragraph with the highest confidence is selected as the most relevant for the nutritional label.

[0130] Next, refer to Figure 45Describing an example of nutritional information extraction, the figure shows an image of a packaged food label. The OCR output is provided as follows:

[0131] Allergens: Contains wheat and soy ingredients.

[0132] Ingredients: Strawberry Filled Pear Juice Concentrate, Tapioca Syrup, Dried Cane Syrup, Apple Powder, Strawberry Puree Concentrate, Corn Starch, Vegetable Glycerin, Natural Flavoring, Elderberry Juice Concentrate, For Seven Color Kashi Whole Grain Whole Flour: Cat, Hard Red Wheat, Brown Rice, Rye, Triticale, Barley, Buckwheat Whole Wheat Flour, Inverted Cane Syrup, Expeller Pressed Rapeseed Oil, Rolled Cat, Honey, Tapioca Syrup, Gum Arabic, Vegetable Glycerin, Hair Fiber, Fermented Sodium Acid Pyrophosphate, Baking Soda Soy Lecithin, Xanthan Gum, Natural Flavoring.

[0133] Nutrition label:

[0134] Figures 46 - 48 The results of extracting nutritional information about more than 40,000 food products are presented. Specifically, Figure 46 Nutrition label accuracy is illustrated, showing a nutrient NLP score histogram of 0.843. Figure 47 The component accuracy is illustrated, showing a component NLP score histogram of 0.887. Figure 48 The allergen accuracy is illustrated, showing an allergen NLP score histogram of 0.937. Based on the above results, it can be observed that the various algorithms utilized in the nutritional information extraction process described herein are capable of extracting highly accurate information about nutrition labels, ingredients, and allergens.

[0135] In some embodiments, the packaged food OCR described herein may further include logo recognition technology. Logo recognition technology may be applied alone or in combination with the nutritional information extraction technology described above. Two main classifiers may be provided for logo recognition: 1) a text-based classifier, and (2) an image classifier.

[0136] The text classifier can extract text from an image using OCR. The text can be converted into a numeric vector using the tfidf vectorizer. Classification of the text vector can be performed using one or more of the following: logistic regression, decision trees, or random forests.

[0137] The image classifier can be based on deep residual learning of image recognition (ResNet50) pre-trained on ImageNet data. A multi-layer (e.g., 3-layer) multi-layer perceptron (MLP) can be attached to the output of ResNet50 and trained. The neural network can be fine-tuned by training all layers except the first n layers (e.g., the first 50 layers). Training can be performed on a small set of images by including specialized image augmentation. The image augmentation can include the following random variations on images with logos: (a) image rotation, (b) gamma correction, (c) brightness variation, and (d) conversion to grayscale. Conversely, images without logos can be used to enlarge the logo image training set by inserting logos into images that do not have them. The following steps can be performed to ensure a large diversity of logo insertions: (i) warping the logo image using a homography transformation, and (ii) adding the logo to random positions in the image. Figure 49 An example showing the result of image logo training. Fine-tuning is performed after 100 epochs. Figure 50 An example showing the result of text logo recognition, particularly the accuracy per average sample number for more than 1000 runs for multiple different logos. Figure 51A and 51B Further shows the results for each of multiple different logos.

[0138] In some embodiments, the food analysis system disclosed herein can include a food image recognition engine. The food image recognition engine can be a computer vision system configured to classify food based on an image and analyze the content, volume, and nutritional value of the food.

[0139] Conventional commercially available food image analysis systems are typically limited to individual food classification and lack the ability to estimate the volume of individual contents within the food. Existing food image analysis systems may also have other deficiencies. For example, existing food image analysis systems analyze food based on the food content shown in the image and cannot account for ingredients that are not visually obvious from the image. Some of these ingredients (invisible from the image) can significantly disrupt and alter the nutritional estimate of a given food. An example of such an ingredient is the oil family, where 1 tbsp of oil can contribute over 100 calories. Figure 19A and 19B Each shows a parallel comparison of foods with similar appearances but substantially different calorie contents due to differences in the amount of individual food contents and other ingredients (such as oil) that are not visually obvious. For example, in Figure 19A , the two bowls contain the same ingredients but in different amounts. Figure 19AThe bowl on the left contains: 180 g of unmodified steak cooked with 1 tsp of oil. 2 cups of lettuce, 4 rings of red onion, 50 g of avocado, 1 cup of cooked rice noodles, 30 g of cucumber, 2 cherry tomatoes, 2 tsp of sesame oil, 2 tsp of lemon juice, 1 tsp of soy sauce. Figure 19A The bowl on the right contains: 100 g of modified steak (roasted), and has the same amounts of lettuce, red onion, and lime juice as the bowl on the left. However, compared to the bowl on the left, the bowl on the right has half the avocado and rice noodles, 50 g of cucumber, twice the cherry tomatoes, 1 / 2 medium carrot, and half the sesame oil. Thus, the bowl on the left contains 770 calories, while the bowl on the right contains only 405 calories. Refer to Figure 19B The bowl on the left contains: 200 g of chicken cooked with 2 tsp of extra-virgin olive oil, 30 g of semi-modified bacon, 2 cups of cooked pasta, 30 g of full-fat cheddar cheese, 2 large broccoli florets, 1 medium mushroom. Figure 19B The bowl on the right contains: 100 g of poached chicken, 1 cup of cooked pasta, 1 tsp of capers, 10 g of low-fat cheddar cheese, 1 / 4 large pepper, 4 large broccoli florets, 2 medium mushrooms. Due to the extra pasta, the bowl on the left contains 800 calories, while the bowl on the right contains only 380 calories.

[0140] Conventional / existing food image analysis systems may be limited for the following reasons.

[0141] First, research has shown that in the United States, 50% of meals are eaten out. This means that for a model to produce good real-world results, it must be able to process restaurant dishes with high accuracy. Most leading academic papers on the subject, as well as available commercial food image analysis solutions, expect to have a sufficient number of images of each dish in each restaurant. In the United States alone, there are over 700,000 restaurants, with an average of 61 dishes per restaurant, which results in a huge lower bound of 42 million dishes. Unfortunately, most restaurants do not provide a sufficient number of images of their dishes (even each dish in a chain restaurant typically has only a few images). Additionally, online sources of dish images (such as images found on Yelp TM ) do not solve the problem. This means that for a solution to provide very accurate results in real data, it must overcome the problem of the "dining-out data gap". That is, even if the model has never been exposed to images from a particular restaurant, it must be able to identify images of dishes.

[0142] Second, as referred to in Figure 19A and 19BAs discussed, many foods may seem similar but can be quite different (e.g., pasta dishes such as Pad Thai, Lo Mein, etc.). Although in many cases, deep neural networks may perform reasonably well in differentiating objects, this is not always possible. For this purpose, there is sufficient value in identifying both the dish and its ingredients and attempting to use the former to provide more information about the latter and vice versa in both one-way and multi-way manners. To achieve this, one would have to leverage the structure of the dish and its relationship with the ingredients (i.e., the food ontology as described in this article).

[0143] Third, in many cases, existing systems may provide several suggestions for a given image, and the classification of many of the images in the image may be poor. This is mainly due to the fact that when analyzed based on images, many foods are deceptive and no computer vision system can perform perfectly. Especially in the case of digital nutrition (and other digital healthcare solutions), receiving poor results may reduce the user's confidence in the computer vision system. The user may stop using such systems. This means that there is sufficient value in only providing correct or sufficiently accurate results. While it is generally not possible for computer vision systems to always be accurate, there is a delicate balance between the particularity of the classification (e.g., pasta dish -> Pad Thai -> Chicken Pad Thai) and the confidence of the system. Current existing systems tend not to generalize their uncertain results. For example, current existing systems may not be able to generate more accurate (but more generalized) results for the user.

[0144] Finally, there is a problem in the non-uniqueness of the mapping from food images to nutritional values. Many existing systems attempt to map food images to nutritional values but are not configured to understand the missing features to complete the mapping. Existing systems are also not configured to assist users in bridging the knowledge gap.

[0145] The food image recognition engine described in this article can address the above-mentioned drawbacks of existing systems. The food image recognition engine disclosed in this article is capable of performing one or more of the following: (1) identifying which foods are definitely in the image (i.e., 100% probability); (2) identifying the foods that may be present in the image (probability) by studying the image, the environment in which the image was taken, the user's history, or the likelihood of certain foods occurring together; (3) differentiating a dish (e.g., Pad Thai) from the ingredients the dish may contain (e.g., peanuts); and (4) estimating the volume and thus the nutritional value using the user's historical dining patterns and the environment in which the user dines. In particular, this can include using a known menu when the user dines out.

[0146] To achieve the above objectives, the food image recognition engine may include algorithms including the following. First, a complete ontology of Visual Indicators of Culinary Items for food (VICUF) was constructed. This is an ontology of any visual indicator that a human (or computer) can use to identify the food in front of them and may include: (1) combined food items (such as Greek salad or burrito); (2) ingredients (such as banana, apple, shrimp); and (3) other indicators (such as cup, liquid, fried food, etc.). Understanding the entire ontology of VICUF can be used in combination with other inputs to obtain a more accurate identification of restaurant dishes. Next, a robust corpus for each tag in VICUF was created. Ideally, each tag in the training set should be annotated. Next, a Convolutional Neural Network (CNN) was trained for each tag in VICUF (binary classifier), or a CNN capable of multi-labeling was trained. The latter CNN can provide better results because some foods usually occur together while others do not. Next, each time a new image is supplied, the CNN maps it to a probability vector, where each component represents the probability that a specific food-related visual indicator is present in the image. The above steps may be sufficient to create a food recording experience. Given an image, the food image recognition engine can classify items in the food recorder according to their probability values in the output vector. A threshold may be included such that items with low probabilities do not appear.

[0147] In some embodiments, additional sensors and inputs such as those based on spectroscopy and invisible light (infrared) can be used to detect subtle differences between visually similar foods and single volume / contents. In some embodiments, the volume / contents can be estimated based on a sequence of images or videos for 3D reconstruction of the food. In some embodiments, an infrared system can be used to perform distance estimation, the infrared system being configured to measure distances to various points on the food plate.

[0148] The food analysis system 210 may include a labeler. The labeler may be a machine learning system for discovering the category or abstraction level of the food (also referred to as a label herein). The labeler may be an automated system for performing text analysis on food objects and labeling them. This allows adding another layer of metadata that the system is using to understand the various characteristics of each food. These characteristics (labels) can be used in different ways by the system, such as by a personalized recommendation engine.

[0149] Examples of tags can include at least one ingredient (e.g., beef, pork, acorns, celery, etc.), at least one nutrient (e.g., vitamin A, vitamin C, calcium, iron, etc.), at least one dietary requirement (e.g., vegetarian, vegan, gluten-free, etc.), at least one allergy (e.g., peanut-free, gluten-free, etc.), at least one dish type (e.g., salad, sandwich, soup, etc.), at least one cuisine (e.g., ethnic and / or religious cuisines, etc.), at least one flavoring (e.g., sweet, fruity, etc.), at least one nutritional property (e.g., low-fat, high-protein, etc.), and at least one texture (e.g., soft, firm, etc.). Tags can generally be classified into one or more of the following categories, e.g.: dietary requirements (including diet and allergens), processing methods, flavors, meals and dishes. Dietary requirements are food labels that allow a personalized recommendation engine to filter out foods that certain individuals will never eat due to certain dietary constraints. Tags can be automatically generated by a labeling machine. Dietary requirements can generally be divided into two categories: (1) dietary requirements caused by food allergies; (2) dietary requirements caused by a specific diet that is often followed for ethical, religious, or environmental reasons. Examples of dietary requirements related to food allergies can include gluten-free, dairy-free, shellfish-free, fish-free, soy-free, egg-free, peanut-free, etc. Other examples of dietary requirements can include vegetarian, pescatarian, or vegan.

[0150] The labeling machine can use machine learning methods or non-machine learning heuristic methods to develop a food tagger that can discover properties about food. Machine learning methods can be referred to as "tag classifier" methods. Non-machine learning heuristic methods can be referred to as "heuristic tagger" methods. The tag classifier method or the heuristic tagger method can generate at least one algorithm for at least one food tagger to identify and label at least one specific abstraction layer of at least one food. The identification and labeling of a specific abstraction layer of food can be referred to as food tagging. The tag classifier and the heuristic tagger can use one or more logical entities called analyzers and problem solvers to perform food tagging. Both the tag classifier and the heuristic tagger can be evaluated using a corpus of pre-defined food data to assess how well each food tagger performs. The labeling machine can use various statistical models (e.g., precision, recall, etc.) to evaluate the performance of the food tagger.

[0151] As a logical entity of the labeling machine, the analyzer can be a single model with specific logic. The analyzer can be used to determine specific attributes of a food item (e.g., whether the item is or is not vegan). Thus, multiple analyzers may be required to determine multiple attributes of a food item. The analyzer can contain a training set. For each analyzer, one or more training sets can be created, and each training set can have a distinguishable name (e.g., "English" or "Spanish" for different languages, or "gluten-free" or "vegan" for different dietary requirements). The problem solver can be a combination of two or more analyzers used to determine the attributes of a food item based on the combined logic of the analyzers it encompasses. The problem solver can contain at least one label classifier analyzer or a heuristic label analyzer. The problem solver can encompass at least one additional problem solver.

[0152] For both the label classifier and heuristic tagger methods, labeling and classifying food data is a process with the following three main processes: (1) training various analyzers; (2) defining at least one combination of two or more analyzers to form a problem solver; and (3) running the problem solver to identify one or more features from the food data and labeling and classifying the one or more features. Additionally, the process of training different analyzers can contain two subprocesses: (i) establishing at least one training set for each analyzer, and (ii) performing training based on the at least one training set.

[0153] The labeling machine can use the label classifier method to develop analyzers. In the label classifier method, the training set of an analyzer can contain at least one input data pair and its corresponding correct answer (also known as the target). The target can be the "gold standard" of the analyzer. The learning algorithm of the analyzer can find one or more patterns in the training set between the input data and the target, and generate an improved machine learning algorithm that can capture the one or more patterns. The analyzer can be trained using more than one training set separately to generate more than one improved machine learning algorithm for the same characteristic. Subsequently, a test set can be used to test the accuracy of the more than one improved machine learning algorithm of the analyzer, and the algorithm with the best performance can be selected for use. Selecting the algorithm with the best performance can involve comparing the statistical analysis of the accuracy tests of each algorithm, including the recall value, precision value, and F1 score. The recall value can indicate how many items that should be labeled were actually selected. The precision value can indicate how many items were correctly labeled. The F1 score can be the harmonic mean of the recall rate and the precision value as a measure of accuracy according to the following equation:

[0154]

[0155] where the F1 score can reach its best value at 1 and its worst value at 0.

[0156] Thus, if multiple analyzer algorithms are trained to label the same food characteristic, the analyzer algorithm with the highest F1 score can be selected as the working analyzer.

[0157] Figure 4 Shows a table of the training set 400 for an analyzer based on the label classifier method. The analyzer can be assigned tasks to classify ingredients from input data 410 (raw ingredient text). A specific training set can be named "English" as the training set is designed to train the analyzer to classify ingredients written in English. Each input data is provided with a corresponding target 420 (matched primary food item). In this example, the input data of "1 pound 90% lean ground beef" can have a corresponding target of "90% lean ground beef". The table of the training set 400 can also indicate when an input data-corresponding target pair is selected for use 430 (approved). The content developer can deselect an input data-corresponding target pair from the training set. The table of the training set 400 can also display multiple samples 440 (Tot # samples) in one or more databases of the platform 200, and the databases contain the same corresponding target. Figure 5 Shows a table of the results 500 from testing multiple trained analyzers. Each analyzer is trained to identify and label ingredients written in English. The table indicates the call value 510, precision value 520, and F1 score 530 for each analyzer tested. In some embodiments, the label classifier can use a pipeline that includes word singularization. For example, a count vectorizer and logistic regression can be used to determine whether a food item is labeled with a certain label. The label classifier can use the same validation set as used by the heuristic tagger described herein as the training set. For example, a complete model for classifying whether a food item is soy-free or soy-containing can be constructed as shown in Figure 52 shown. Thus, only food items that are classified as soy-free by both classifiers and whose combined classification confidence exceeds 85% can be considered soy-free. To study the impact of model composition, the precision and recall of individual models, as well as the composed model (for the example of the soy-free tagger above), can be examined and summarized in the following table. It can be observed that the model benefits greatly from the composition, increasing the precision by approximately 20% (in this case):

[0158] Figure 53 Shows an example of the learning curves of most of the classifiers described herein. It can be observed that the model can learn the training set almost completely, and the model can generalize. The learning curve also shows that obtaining more data may further improve the generalization.

[0159] Alternatively or additionally, the labeling machine can use a heuristic tagger method to develop an analyzer. The analyzer of the heuristic tagger method can analyze food items for specific food characteristics (e.g., dietary requirements such as vegetarian, vegan, gluten-free, etc.). This analyzer can use a special vocabulary predefined for each specific characteristic to determine whether a new food item should be labeled with the characteristic or not. Each analyzer of the heuristic tagger method can contain three components: a vocabulary, tagging logic, and a training set.

[0160] The analyzer of the heuristic tagger method can include a vocabulary. The vocabulary can contain groups of words related to food characteristics (e.g., gluten-free), also known as food labels. When text data from a food item is provided as input, the heuristic tagger method compares the words found in the text data with the groups of words in the vocabulary. The groups of words in the vocabulary can include negative terms, positive terms, menu positive terms, and non-negative terms. Having at least one matching negative term can indicate that the food item may not belong to the food label. Having at least one matching positive term (even though having at least one matching negative term) can indicate that the food item may belong to the food label. Having at least one matching menu positive term can indicate that the food item may belong to a specific food label only if the word comes from a menu category title. Finally, non-negative terms can be words that were once negative terms but have been proven otherwise. Non-negative terms can be retained to ensure that such terms are not added as negative terms again.

[0161] Figure 6 An exemplary vocabulary for the food label "pescatarian" is shown. A pescatarian can be a person who does not eat meat but eats fish. The vocabulary for the food label "pescatarian" can have negative terms, positive terms, positive terms, and optionally non-negative terms. The negative terms can include Caesar, fillet, sirloin, brown gravy, pancetta, brisket, country gravy, steak, pork, hamburger, beef burger, cheeseburger, duck, chicken, beef, lamb, veal, corned beef, sausage, meatball, ham, coppa, turkey, rabbit, bacon, chorizo, meat, wing, tenderloin, rib, dog meat, sausage, meatloaf, steak burger, lasagna, cheesesteak, lasagna, bison, rib, salami, capocollo, philly, prosciutto, and worcestershire. The positive terms can include veggie burger and meatless. The positive menu terms can include vegan and vegetarian. Non-negative terms may or may not be found.

[0162] The analyzer of the heuristic tagger method may include tagging logic. Based on the vocabulary, the tagging logic may have three rules for tagging food items. First, if there are no matching negative terms, positive terms, or menu positive terms in the name, description, or menu category title of a food item, the food item can be tagged with a label. Second, if there is at least one matching negative term and no other matching terms, the food item cannot be tagged with a label. Third, if there is at least one matching positive term or at least one matching menu positive term in the menu or submenu of a food item, the food item can be tagged with a label. Even if there is at least one matching negative term, the third rule of the tagging logic can remain valid.

[0163] In addition to the vocabulary and tagging logic, the analyzer of the heuristic tagger method may also include a training set. The purpose of the training set may be to verify whether the vocabulary generated for a certain food label is sufficient. The training set may include a list of food items with known characteristics. The corresponding vocabulary can be applied to each food item in the training set to evaluate whether a tag should be applied or not. Afterwards, the training set can determine whether the tag evaluation is correct and report the precision value (%) of the vocabulary.

[0164] Figure 7 An exemplary table of a training set for the food label "fish vegetarian" based on the heuristic tagger method is shown. The training set may include various food items, including food items that a fish vegetarian can or cannot eat. Each food item in the various food items can be defined by the following list: name (e.g., strawberry swiss roll), description (e.g., mascarpone cream, honey oats, and strawberry sorbet), menu (dessert), and submenu (dessert). If a food item is listed in a menu without a submenu, the submenu can be listed as the same as the menu. The test vocabulary may involve applying the vocabulary to analyze the various food items in the training set and recording whether the vocabulary can correctly distinguish food items that are friendly to fish vegetarians. The training set can also report a precision value (e.g., 98%) and a recall value (e.g., 92%) to determine the reliability and effectiveness of the vocabulary. In this example, the precision value can indicate how many of the analyzed items are correctly tagged as "fish vegetarian". The recall value will tell how many items that should be tagged as "fish vegetarian" are actually selected.

[0165] During training or in use, an analyzer based on a heuristic tokenizer method can detect one or more words extracted from a food item that are not found in an appropriate vocabulary. If one or more unknown words (e.g., via machine learning) are evaluated as being related to at least one negative term, the analyzer can store the one or more words, along with at least one word that may be related and found in the vocabulary, in database 240. The one or more words can then be examined (e.g., via machine learning or a content developer) and added as new terms to the appropriate vocabulary of the analyzer (e.g., as new negative terms). The added one or more words can expand the capacity and efficiency of the analyzer.

[0166] As a logical entity of the labeler, the problem solver can be a combination of at least two analyzers to integrate the logic of at least two analyzers. In some instances, an analyzer for a specific food attribute from a machine learning method (label classifier) and an analyzer for a specific food attribute from a heuristic method (heuristic tokenizer) can be combined as the problem solver to label foods with a specific food attribute. The resulting problem solver can be represented as an "attribute" food labeler. For example, a problem solver for identifying and labeling gluten-free foods can be referred to as a "gluten-free" food labeler.

[0167] Figure 8 An exemplary pipeline 800 of the "gluten-free" food labeler is shown. The pipeline can be a graph representing a combination of different analyzers. The "gluten-free" food labeler 810 can be designed by combining two analyzer nodes: a label classifier 820 and a heuristic tokenizer 830, both of which are trained to identify the "gluten-free" attribute. The relationship between the two analyzer nodes for defining the food labeler 810 can be described by a logic node. The logic node can be an AND node or an OR node. The AND node can combine the results of the two analyzers in a strict manner. As Figure 8 shown, if the "gluten-free" food labeler is defined by the AND node 840, the food item can be labeled "gluten-free" only if both analyzer nodes consider the food item to be "gluten-free". On the other hand, an OR node in use can combine the results of the two analyzers in a less strict manner. For example ( Figure 8 not shown), if the "gluten-free" food labeler is defined by the OR node, the food item can be labeled "gluten-free" when at least one of the two analyzers considers it to be "gluten-free". The pipeline also includes arrows 850, 852, 854 that connect the relationships between the nodes.

[0168] The problem solver can be a combination of multiple problem solvers that integrates the logic of multiple problem solvers to refine food attributes. The resulting problem solver can also be represented as an "attribute" food tagger. Figure 9 An exemplary pipeline of a "vegan" food tagger is shown. The "vegan" food tagger can be defined by a combination of multiple nodes, including a label classifier for the "vegan" attribute, a heuristic classifier for the "vegan" attribute, and additional pre-defined food taggers for additional food attributes. The relationship between the "vegan" label classifier and the "vegan" heuristic classifier is defined by an AND logic node. Each of the additional pre-defined food taggers can be a combination of a label classifier and a heuristic classifier for one of the additional food attributes. The additional food attributes can be "dairy-free", "egg-free", and "vegetarian", and their relationships are defined by AND logic nodes. The resulting "vegan" food tagger can label a food item as "vegan" only if the following four requirements are true: (1) both the "vegan" label classifier and the "vegan" heuristic classifier consider the food item to be "vegan"; (2) the "dairy-free" food tagger considers the food item to be "dairy-free"; (3) the "egg-free" food tagger considers the food item to be "egg-free"; (4) the "vegetarian" food tagger considers the food item to be "vegetarian". If any of the four requirements is not true, then Figure 9 the "vegan" food tagger in

[0169] Each version of the problem solver (or food tagger) can be tested using a training set and analyzed using statistics. The statistics from the problem solver can be different from the statistics of the analyzer because the problem solver is a combination that includes the results of the analyzer. Along with precision, recall rate, and F1 score, a utilization value can also be reported. The utilization value can be the fraction of the training set items that can pass the confidence threshold defined in the pipeline of the problem solver. A problem solver with a pre-defined confidence threshold can have a high precision and recall value but a low utilization value. A low utilization value can indicate that a high portion of the results of the problem solver are inaccurate, and the problem solver may not be usable as a food tagger.

[0170] In one instance, a 70% confidence threshold node can be added to Figure 9 the pipeline of the "vegan" food tagger shown in Figure 10The pipeline shown in generates a new "vegan" food tagger. The new "vegan" food tagger can be tested using a training set and can report statistics containing utilization values. The utilization value of the new "vegan" food tagger can represent the fraction of training set items that can pass the 70% confidence threshold defined in the problem solver pipeline of the new "vegan" food tagger. A table of multiple problem solvers and their corresponding statistical analyses is shown in Figure 11 it.

[0171] Food analysis system 210 can use a labeler to analyze various types of data related to food and map the data to a food ontology. Food analysis system 210 can automatically obtain data (images or text) related to food (such as nutrition label tags, manufacturer product information, restaurant menus, recipes, etc.) from one or more sources (e.g., Internet 120, grocery store websites, restaurant websites, recipe blogs, user input, etc.). Food analysis system 210 can convert an image of at least one consumer packaged food, at least one restaurant menu item, or at least one food recipe into structured data using deep learning, OCR, and / or NLP capabilities according to the preferred format of the system. Food analysis system 210 can also reorganize the obtained text data into structured data according to the preferred format of the system. Using the new structured data, food analysis system 210 can analyze and classify the characteristics of at least one consumer packaged food, at least one restaurant menu item, or at least one food recipe in real time and map at least one consumer packaged food, at least one restaurant menu item, or at least one food recipe to a food ontology.

[0172] During the analysis and classification of food, food analysis system 210 can automatically parse and classify the types and amounts of ingredients specified in the obtained data. During the analysis and classification of food, food analysis system 210 can automatically estimate the types and amounts of unknown ingredients based on known ingredients specified in the obtained data or other similar foods in the food ontology. Food analysis system 210 can use at least a probability model to estimate the ingredients that must or may be present in food (e.g., restaurant dishes). The food analysis system can also calculate the probability (or confidence level) of the food having the estimated ingredients and the expected range of the amount of each estimated ingredient.

[0173] Food classification can include matching the text of raw ingredients to equivalent primary food items in the food ontology. Food classification can be implemented in scikit-learn, for example, via the following pipeline. First, word singularization can be performed by processing the raw text using the inflect library to convert all nouns to their singular forms. The inflect library can be used to correctly generate plurals, singular nouns, ordinals, indefinite articles, and convert numbers to words. Next, a count vectorizer can be used to extract features as word vectors. Next, k best features can be selected according to the chi-squared test. Finally, multinomial logistic regression can be utilized for classification. Cross-validation can be utilized to perform the above pipeline in grid search to find the optimal k value.

[0174] Figure 54 A graphical user interface (GUI) for managing a training set for food classification is shown. The GUI implements the following capabilities for training set management. For example, training samples (text-to-primary food mappings) can be added. When adding samples, it may be possible to search for recipe ingredients for the sample text. Conversely, training samples can be removed. The interface also allows viewing of existing training samples and changing the food items to which they are mapped. Samples can be approved. Additionally, if those samples are automatically created and not very accurate, the approval of some samples can be removed. The interface can also allow filtering of data by text / food item / approval status. The interface can allow users to easily / conveniently access relevant information.

[0175] Food classification can include a confidence score, which can be used to filter or add human verification input to the results. Figure 55 A histogram of food classification success by confidence is shown according to some embodiments. For example, if a confidence threshold of 0.7 is selected for a sample data set, there may be 971 correct ingredient matches and 29 incorrect ingredient matches above the threshold, and 353 correct ingredient matches and 55 incorrect ingredient matches below the threshold.

[0176] As previously described, different food categories can present ingredient information in different ways and may therefore need to be analyzed in different ways. In some embodiments, food categories can be classified into multiple different models (e.g., 4 different models), each of which is built on the results of the previous model.

[0177] The first model may include a food classification model. The first model may utilize the food classification techniques described elsewhere in this document. The relevant food categories of the first model may include packaged foods, certain restaurant menus, and in some cases recipes (although recipes may not typically be input into the first model). The input to the first model may include free text describing a food item, such as "sucrose" or "brown rice", and the output of the first model may include a food identifier (food ID).

[0178] The second model may include a food parsing model. The relevant food categories of the second model may include recipes and in some cases packaged foods (but packaged foods may not typically be input into the second model) and in some cases restaurant dishes (but restaurant dishes may rarely be input into the second model). The input to the second model may include free text describing a food item and its quantity, such as "1 cup of couscous", and the output of the second model may include a food identifier (food ID), the quantity of ingredients, and a measure ID. Food parsing may include converting a single raw text ingredient (e.g., "1 cup of brown sugar") into an equivalent food item, serving, and / or quantity according to a food ontology. Food parsing may be performed, for example, using the following process. First, the food classification from the first model may be used to determine the primary food item. A list of all possible measurements may be extracted from the text and standardized using a standard unit library to associate each unit with a corresponding quantity. The above list may be further extended using conversions between different units. For example, if "tablespoon" is found in the text but "teaspoon" is not, then a "teaspoon" measure corresponding to 3 times the quantity of "tablespoon" may be added. All measure units (e.g., "cup", "tablespoon", etc.) of the matching food item may be retrieved from a database. Then, the two lists are matched to select the most likely measure and its quantity. The food parsing function may be implemented as part of the food analysis system described in this document.

[0179] The third model may include a free text analysis model. The relevant food categories of the third model may include restaurant dishes, and in some cases packaged foods (but packaged foods may rarely be input into the third model). The input to the third model may include, for example, a food description string: "Homemade deep - fried fries tossed in creamy red pepper salsa. Topped with cashew cream and cilantro. And prosciutto and two hard - boiled eggs." The output of the third model may include a list containing a food ID, the quantity of ingredients, and a measure ID. In some cases, the quantity or measure is not provided as part of the free text. In those cases, the model may be configured to extract the ingredient ID and mark the measure ID and quantity as unknown.

[0180] Free text analysis may include using any of the NLP algorithms described herein. Free text analysis may include performing entity extraction on free text to extract food information provided in the text. Entities may include objects representing ingredients. Additionally, entities may also include objects that do not necessarily represent ingredients but can nevertheless provide insights into the label or nutrients of the food. For example, in the case of describing a food item named "Totopos":

[0181] {{Food item [homemade] [deep fried] processing method [potato chips] food mixed with cream [cream red pepper salsa]. Covered with [cashew cream] food and [coriander] food. Served with [chorizo] food [two] amounts [hard-boiled eggs] food}}.

[0182] Entities may include, for example: food items, amounts, measures, processing methods, dietary requirements, restaurant names, etc. Free text analysis may associate each entity with one or more other related entities. For example, "two" at the end of the string is related to "eggs". Once an entity has been detected as a "food entity", the entity can be run through an ingredient classifier to determine the exact food ID. Entity extraction can be solved using statistical methods such as conditional random fields (CRF). In some cases, if the corpus of entities is large enough, deep learning techniques can also be used to solve entity extraction. In the case where the string contains food items separated by commas, entity extraction can be simplified because the string can be split based on comma separation before running food parsing.

[0183] The fourth model may include a menu item analysis model. The relevant food categories of the fourth model may include restaurant dishes. The input to the fourth model may include the name and description of the dish. The output of the fourth model may include a list containing the food ID, the amount of ingredients, the measure ID, and the probability that each ingredient is present in the dish.

[0184] Under menu item analysis, a menu item is different from a pure text description because a menu item contains both the name and description of the dish. The description does not always contain all the ingredients, and therefore the name of the menu item can be used to estimate potential ingredients using any of the statistical methods described herein.

[0185] For example, consider the following two items from a menu:

[0186] (1) Name: Pad Thai with Chicken. Description: Steamed rice noodles stir-fried with chicken, eggs, mushrooms, onions, coriander, and peanuts.

[0187] (2) Name: Fried Rice. Description: Choose beef, pork, or chicken.

[0188] The menu item analysis pipeline can include multiple steps. Under statistical name analysis, assume there is a complete ontology of food (which can be represented as a tree by a taxonomy), and the branches in the taxonomy can be found based on the food name.

[0189] The union of all ingredients in the element can be performed within the branch, and the probability of each ingredient appearing in the food item can be calculated. For example: "Pad Thai with Chicken" can be its own branch or part of the regular branch "Pad Thai". All Pad Thai has rice noodles, 80% of which may have fish sauce, 23% of which may contain mushrooms, etc. The probability of each food item appearing (calculated through the branch) can be given as follows:

[0190] P(ingredient in food) = (number of food items containing the ingredient) / (number of food items)

[0191] If an element in the branch has too few instances, the probability can be recalculated by ascending in the taxonomy.

[0192] The probability of each food item appearing can be used to extract more information from the menu. In some embodiments, the data can be further refined by crowdsourcing from users and / or restaurants about the appearance of specific ingredients that the user is unsure of.

[0193] Free text analysis can be performed on the description, and then the probability of certain ingredients found in the previous step can be increased to 100% (or close to 100%). For example, in the above example (1), mushrooms can appear as an ingredient, and thus the confidence that the food item has mushrooms can be increased (e.g., from 23% to 100%). By estimating the amount of the ingredient, a list of probabilities for the appearance of each ingredient can be generated. Next, a probability model can be applied to the amount of each ingredient. Considering a threshold θ, above which the ingredient is assumed to be part of the dish. Initially, this threshold can be assumed to be a value (e.g., 50%). In some embodiments, the threshold can be estimated by machine learning techniques to maximize the accuracy of the results.

[0194] To estimate the amount, again consider the branch where the food item appears, and fit the probability model to the amount of each ingredient. The probability model is applied to all samples in the branch that contain the ingredient. Since the amount is always positive, the probability model can be a lognormal distribution, which can provide both the expected amount and the standard deviation.

[0195] The food analysis system 210 can estimate unknown nutrients from at least one consumer-packaged food, at least one restaurant menu item, or at least one food recipe. Once the types and amounts of known and unknown ingredients are abstracted and estimated using OCR, NLP, labelers, or other algorithms of the food analysis system 210, such information can be used to estimate nutrients not shown in at least one consumer-packaged food, at least one restaurant menu item, or at least one food recipe. The food analysis system 210 can estimate the range of nutrients not shown. In some instances, the range of nutrients can be based on their amount (e.g., grams, milligrams, etc.) or the percentage (%) of the recommended total daily requirement for each nutrient in the nutrient. In some instances, the nutrients not shown can be macronutrients, including the breakdown of total fat (e.g., saturated fat, trans fat) or total carbohydrates (e.g., dietary fiber, total sugars, added sugars). In some instances, the nutrients not shown can be micronutrients, including vitamins, macrominerals, and microminerals. Vitamins can include biotin, folate, niacin, pantothenic acid, riboflavin, thiamine, vitamin A, vitamin B 12 , vitamin C, vitamin D, vitamin E, and vitamin K. Macrominerals can include calcium, phosphorus, magnesium, sodium, potassium, chlorine, and sulfur. Microminerals can include iron, manganese, copper, iodine, zinc, cobalt, fluorine, and selenium. In some instances, the nutrients not shown can be phytonutrients. Phytonutrients can be anthocyanins, ellagitannins, flavonoids, allyl sulfides, and isoflavones.

[0196] The food analysis system 210 can estimate unknown nutrients from at least one consumer-packaged food. The food analysis system 210 can parse and classify data obtained from nutrition label tags and ingredient lists of at least one consumer-packaged food. The nutrition label tag can show some but not all of the nutrients found in at least one consumer-packaged food. Additionally, the nutrition label tag may not disclose the amount (e.g., milligrams) of such nutrients. Thus, the food analysis system 210 can infer known or unknown ingredients and their amounts from data obtained using labelers and other algorithms described in this disclosure. Using the nutritional breakdown of each ingredient of known or unknown ingredients from one or more databases 240 of the platform 200, the food analysis system 210 can calculate the types and amounts of nutrients that can be found in at least one consumer-packaged food. By using the items shown in the nutrition label tag (e.g., calories per serving, total fat, total carbohydrates, protein), the food analysis system 210 can compare its calculated values for the corresponding shown items to verify its estimate.

[0197] The food analysis system 210 can estimate unknown nutrients from at least one food recipe. The food analysis system 210 can parse and classify a list of ingredients and their corresponding amounts from at least one food recipe using a labeler and other algorithms described in the present disclosure. The food analysis system 210 can aggregate the types and amounts of nutrients found in the ingredients and provide an estimate of the nutritional value of the resulting recipe. Nutrients can include at least one macronutrient, at least one micronutrient, or both. This estimate can result in high reliability because at least one macronutrient and at least one micronutrient are minimally affected by cooking methods. A pipeline for estimating unknown nutrients from at least one food recipe can include: (1) receiving free text of a food recipe containing ingredients, (2) parsing and classifying the ingredients and their corresponding amounts, and (3) overlaying the estimated nutritional values of the ingredients using the nutritional breakdown of each ingredient from one or more databases. Alternatively or additionally, the food analysis system 210 can include at least one machine learning algorithm based on at least NLP, statistical analysis, and multiple chemical databases for estimating one or more effects of food processing (e.g., frying, boiling, microwaving, cooking time, etc.) on the nutrients and their nutritional values of food. In some instances, this machine learning algorithm can be referred to as a food processing algorithm. The food processing algorithm can include specific processing parameters (e.g., absorption coefficient, evaporation coefficient, diffusion coefficient, etc.) for one or more nutrients. The diffusion coefficient of each of the one or more nutrients can take into account the size of the corresponding ingredient during cooking. Another pipeline for estimating unknown nutrients from at least one food recipe can include: (1) receiving free text of a food recipe containing ingredients, cooking method, and cooking time, (2) parsing and classifying the ingredients and their corresponding amounts by using at least a machine tagger and a food processing algorithm, and (3) overlaying the estimated nutritional values of the ingredients. During the cooking process, at least one ingredient can be used in two or more steps during cooking. The nutritional value of at least one ingredient can be calculated multiple times for two or more steps during cooking.

[0198] The food analysis system 210 can estimate unknown nutrients from at least one restaurant dish. The food analysis system 210 can map at least one restaurant dish to a food ontology. Subsequently, the food analysis system 210 can use its algorithms described in the present disclosure to detect which ingredients must or can be present in at least one restaurant dish. For each ingredient, the food analysis system 210 can calculate the types and expected amounts of nutrients that may be found.

[0199] The food analysis system 210 can utilize an automated spider builder (also known as a "crawler") to crawl the Internet to obtain all food-related data, abstract and classify food-related information, and store the information in one or more databases 240. The Internet 120 can include websites of restaurants with their menus, food manufacturers, and recipe blogs. Each website may have a different structure and may be disorganized and outdated. The automated spider generator of the food analysis system 210 can detect the layout of each website by detecting the XPath corresponding to each food name, description, price, ingredients, etc. XPath (or XML Path Language) can be a query language for selecting data points from an XML (Extensible Markup Language) data structure, such as data points of a restaurant website. The automated spider builder can scale up the data collection process of the food analysis system 210. The automated spider builder can allow a content management team without technical skills to easily add thousands of new foods to the food analysis system 210.

[0200] The food analysis system can analyze pictures of food dishes that do not have any text information about the food. In one instance, a user can take a picture of a food item at a restaurant using a mobile device. The food analysis system 210 connected to the mobile device can automatically receive the picture of the food item. The food analysis system 210 can use at least its deep learning and OCR capabilities to abstract and classify the ingredients that may appear in the food item. The food analysis system 210 can compare the food item with similar dishes already in the food ontology to abstract and classify the ingredients that may appear in the food item. If proven successful by the internal test model of the food analysis system 210, the food item can be mapped to the food ontology. The food analysis system 210 can compare the first characteristic absorption curve in at least the first part of the electromagnetic spectrum of the picture of the user's food item with the second characteristic absorption curve in at least the second part of the electromagnetic spectrum of at least one of the similar dishes already in the food ontology.

[0201] An example of abstracting and classifying information from consumer food packaging will be described in detail below with reference to Figures 12A - 12C be described in detail.

[0202] Figure 12ADisplays a picture 1200 of a lemon cheesecake 1210 from Dierbergs Bakehouse. Once imported into the platform 200, the food labeling system 210 in the platform 200 uses machine learning and OCR to automatically identify one or more information sections 1211, 1212 of the picture 1200 and draw bounding boxes around the sections. The first information section 1211 contains the ingredients in the lemon cheesecake, including milk, cream, sugar, barley flour, etc. The second information section 1212 contains warning statements from the manufacturer, including other ingredients that have been utilized in the manufacturer's facilities. Then, the food labeling system 210 uses NLP algorithms to abstract each word from the information sections 1211, 1212 in real time. Subsequently, the food labeling system 210 uses a labeling machine to label and classify the information.

[0203] Figure 12B Displays a table 1220 of the abstracted and classified information from the lemon cheesecake 1210. In the table 1220, the lemon cheesecake 1210 is identified by its product code 1230. The table 1220 presents nutrients (e.g., calories, total fat, sodium, etc.) and their corresponding amounts by weight (e.g., 130 g, 6 g, 170 mg, etc.) 1240 - 1250. The table 1220 presents a list of ingredients 1260, 1261. The table 1220 presents additional information from the manufacturer 1270 - 1275, including what the packaging contains, what the packaging may contain, what other products have been prepared using the same equipment from the same facility, and ingredients that have been utilized in the same facility. Importantly, the table 1220 presents one or more abstract labels 1280, 1281 (e.g., contains added sugar, all natural ingredients, no artificial colors, no artificial flavors, etc.) generated by the labeling machine of the food analysis system 210 based on the abstracted and structured data in the table 1220. The table 1220 also contains confidence scores and accuracy scores for all food labelers used in OCR and 1290 - 1297. The information generated by artificial intelligence can also be formatted in other formats, such as Figure 12C as shown in. The labeling machine can process a single picture or a batch of an unlimited number of pictures and can complete the analysis in a total time of approximately 3 minutes to approximately 5 minutes. If a picture has more than one packaged food, the food analysis system 210 can break the picture into multiple sub - pictures. Each sub - picture can contain one packaged food.

[0204] Figure 13Displays an exemplary table of various consumer-packaged foods after abstraction and classification. The various consumer-packaged foods can be organized by a machine-generated numbering system or by their corresponding product codes. Each consumer-packaged food can contain multiple pictures (e.g., pictures from different years or different angles, etc.). The table also keeps track of many ingredients, nutrients, detected allergens, and many food characteristic labels that have been found to be true (positive) and many food characteristic labels that have been found to be inapplicable to the labeling machine (negative).

[0205] Figure 14 Displays an exemplary restaurant menu obtained and analyzed by the food analysis system 210. The food analysis system 210 can use OCR and other algorithms to detect food dishes and their corresponding menu and sub-menu titles and draw bounding boxes around the food dishes and their corresponding menu and sub-menu titles. As Figure 14 shown, by way of example only, the food analysis system 210 can decompose the menu 1400 into food dishes and their titles 1410 and other non-food-related information 1420. Figure 15 Displays an exemplary table of information abstracted from a restaurant menu. This is an extensible method that can use AI, OCR, and / or other algorithms to capture menus located at at least approximately 350,000 restaurant locations across the United States. Additionally, the AI can continuously monitor the web of new or previously analyzed websites to identify new menus and expand the database.

[0206] The food analysis system 210 of the platform 200 can compile all the analyzed information into a food ontology. The food ontology can be used as a network for all foods. The food ontology can describe the relationships between foods, ingredients, nutrients, and other characteristics. The food ontology provided in the present disclosure can describe one or more relationships between two or more categories including or of foods, ingredients, nutrients, and / or characteristics. The food ontology provided in the present disclosure can describe one or more relationships between two or more components within one of the above categories.

[0207] The food ontology can include a graphical representation depicting information related to different foods. The food ontology can further include displaying the graphical representation of the food ontology on an electronic display. The graphical representation of the food ontology can be two-dimensional. The graphical representation of the food ontology can be multi-dimensional, including three or more dimensions.

[0208] The food ontology can be a graph database. The graph database can include one or more nodes. One or more nodes can be connected to one or more edges. Each edge can represent a relationship between two nodes. Instances of one or more nodes and one or more edges used in an embodiment of the food ontology are mentioned above in the present disclosure.

[0209] The graph database may include one or more “taxonomies” of food-related concepts (e.g., food items, ingredients, amounts, sizes, volumes, processing methods, dietary requirements, nutritional values, restaurant names, food dish names, dietary restrictions, geographical origins, etc.). The taxonomy may be a basic building block of the food ontology. The taxonomy may be represented by one or more nodes (e.g., one or more food item nodes) and one or more edges. The taxonomy may be at least a dimension in the “food space”. Thus, one or more combinations of taxonomies may present a food ontology in multiple dimensions. One or more combinations of taxonomies may allow for the analysis and / or visualization of the food space along any of the multiple dimensions. In some cases, the taxonomy of the food ontology may be represented as a tree diagram having one or more nodes and one or more edges, as mentioned and described elsewhere above in the present disclosure. In some instances, the edge may represent an “IS A” relationship between two nodes (e.g., “chicken” is “poultry product”; “cooking” is “processing method”, etc.). In other instances, the edge may represent a “good substitute” relationship between two nodes.

[0210] In some cases, the taxonomy of the food ontology can be the ingredients. The ingredient taxonomy can be operably connected to one or more food databases (e.g., primary food databases such as the USDA National Nutrient Database for Standard Reference, the USDA Branded Food Products Database, etc.). In the ingredient taxonomy, each node can represent a food. The food can be a specific food (e.g., Coca-Cola, banana, etc.), or can be an abstract food in the form of a combination of one or more specific foods (e.g., salad, tuna sandwich, mac and cheese, etc.). In some cases, the food can be an abstract category of a specific food genus (e.g., beef, nuts, vegetables, fruits, etc.). The edges can represent the relationships between two foods. Several types of relationships between two foods can be allowed. In one instance, two types of relationships between two foods can be allowed. The first type of relationship between two foods can be "IS_A", indicating a food that can be a variant or a part of another food. Some examples can include (1) "yellowfin tuna IS_A tuna (variant)" and "tuna IS_A (variant of fish)", and (2) "chicken thigh IS_A chicken (part)" and "chicken IS_A poultry meat (variant)". The second type of relationship between two foods can be "contains", indicating a first food that can contain a second food. The first food can contain the second food after a certain type of processing, transformation, or directly. If this containment involves the processing of a single ingredient, the edge can contain processing method information. If this containment involves the processing of two or more ingredients, the node can contain processing method information. Some examples can include: (1) "almond flour contains (ground) almonds", and (2) "cooked white rice contains (white rice, water)". The food ontology can have additional types of predefined edges to describe additional relationships between any two node pairs. The food analysis system 210 can generate one or more new edges for the food ontology.

[0211] As described above, in some cases, the taxonomy of the food ontology can be one or more processing methods. One or more processing methods can represent the various processes (e.g., growth, harvest, transportation, cooking, etc.) that food components undergo during preparation at various stages. Examples of the types of cooking that can be employed can include: curing, baking, roasting, grilling, sautéing, burning, steaming, boiling, broiling, deep frying, pan frying, air frying, impact cooking, stewing, simmering, boiling, slow cooking, sous vide, smoking, cold smoking, and / or any combination of the above, etc.

[0212] Nodes in the food ontology can be specific food items. Specific food items can include specific kinds of seeds, specific kinds of vegetables, specific kinds of tubers, specific kinds of edible fungi, specific kinds of meats, etc. Specific food items can be referred to as leaf nodes (or leaves) of the graphical database of the food ontology. A leaf node can be a node that does not have child nodes. For example, a potato (Yukon Gold Potato) cannot be decomposed into sub-foods and can be registered as a leaf node in the food ontology. Thus, each leaf node in the leaf nodes can be a meta-object existing above each specific food item. Additionally or alternatively, the food analysis system 210 can (i) receive data of specific food items from other nutrition services (e.g., USDA), (ii) analyze the data and map the data in the food ontology, and (iii) use at least one machine learning algorithm to learn how to generalize specific food groups and create new leaf nodes. For example, "long grain brown rice" and "short grain brown rice" can be generalized as "brown rice", and the food ontology can have leaf nodes of "long grain brown rice", "short grain brown rice", and "brown rice". If the food analysis system 210 determines that the incoming data from a third-party database (e.g., "brown rice") has been generalized, then the data can be plotted as a leaf node in the food ontology.

[0213] The graphical representation of the food ontology can be multi-dimensional, containing two, three, or more dimensions. Figure 16 An exemplary two-dimensional graphical representation of the food ontology 1600 related to the tuna salad from Panera Bread is shown. The food ontology 1600 contains multiple types of nodes. The food ontology contains multiple types of edges. The thick edges represent the relationships between two nodes directly abstracted from the original input data. The thin edges represent the relationships between two nodes indirectly estimated by the food analysis system 210. The first type of node can be a consumer-packaged food (e.g., the tuna salad from Panera Bread), an abstract food type (e.g., tuna salad, salad, wheat wrap, tuna salad wrap, or wrap), or an ingredient (e.g., fish, tuna, pickles, mayonnaise, egg, vinegar, or salt). A node can be a subclass of a higher-class node, and this relationship can be represented by an IS_A edge. Some examples include "the tuna salad from Panera Bread IS_A (subclass of) tuna salad" and "tuna salad IS_A (subclass of) salad" and "tuna salad wrap IS_A (subclass of) wrap". If the subclass is included as a whole in the higher-class node, then this relationship can be represented by a contains edge. In one example, "tuna salad wrap contains (tuna salad, wheat wrap)". Alternatively, a node can be an ingredient included in a higher ingredient node, and this relationship can be represented by a contains edge. As Figure 16As shown, the ingredients of tuna salad include tuna, mayonnaise, pickles, eggs, vinegar, and salt. Since tuna and mayonnaise are known ingredients, the food ontology uses a thick containment edge to describe tuna and mayonnaise as in "tuna salad contains (tuna, mayonnaise)". Similarly, since eggs, vinegar, and salt are known ingredients of mayonnaise, the food ontology uses a thick containment edge to automatically classify the first three items as in "mayonnaise contains (eggs, vinegar, salt)". On the other hand, since pickles are unknown ingredients estimated by the food analysis system 210, the food ontology uses a thin containment edge to describe pickles as in "tuna salad contains (pickles)". The node can be scrolled, indicating that one or more databases 240 of the platform 200 have specific nutritional values connected to the node items.

[0214] Figure 56 An exemplary two - dimensional graphical representation of the food ontology 1600 related to ingredient taxonomy is shown. The central "grains and cereals" node can be connected by edges to each of a plurality of child nodes (where each edge is indicated by an arrow pointing in the direction from the child node to the central node). One or more of the child nodes can represent known or unknown ingredients of the "grains and cereals" node. Examples of the child nodes (ingredients) of the "grains and cereals" node can include pseudocereals, oats, rice, barley, wheat, corn, rye grains, millet, teff, sorghum grains, and triticale. The nature of the relationship between the node and each child node can be represented on the food ontology by the length of the edge (e.g., the length of the arrow), the width or color of the edge, the distance between the node and the child node, the color / shape / size of the child node, etc. Examples of the nature of the relationship between the node and each child node can include, for example, whether or not the ingredient child node should or can be present in the food node, the source of the ingredient, substitutes for the ingredient, etc.

[0215] The food ontology can be used to infer associations (e.g., known or unknown associations, abstract associations, etc.) between foods or between food-related data (e.g., between ingredients within each food). One or more algorithms can include one or more statistical methods. In some cases, the food ontology can be represented by at least two components: (1) a taxonomy, which can be a hierarchy of various "food space" dimensions (e.g., ingredients, processing methods, serving methods, nutrient compositions, cuisines, etc.), and (2) the nodes and edges that make up each taxonomy (e.g., food item nodes). Such a representation of at least two components can be used to infer, generate, evaluate, confirm, or reject one or more associations between nodes within the taxonomy. In some cases, this food ontology structure can be used to infer common associations between nodes. In one example, in the context of a recipe, a user can input "BLT", and the food ontology can reveal a recipe for a bacon-lettuce-tomato sandwich. In another example, in the context of a meal, a user can input a given food item (or partial information about a food item), and the food ontology can predict which other foods can also be included in the meal that the user input (e.g., as part of the food item or in addition to the food item).

[0216] Alternatively or additionally, the food ontology can compare two or more taxonomies and generate a new taxonomy that includes new nodes and edges to represent previously unknown or unrecognized associations between food-related information.

[0217] One or more statistical methods can be used to generate statistical associations between foods or between data related to foods. Example embodiments for generating such statistical associations are provided herein. Given a set of recipes (i.e., the "population") and a subset of recipes (i.e., the "sample"), one or more statistical methods can identify two or more nodes in an ingredient taxonomy that are associated (e.g., "strongly associated" nodes). For example, given 20,000 recipes that include 12 recipes that contain "BLT" as part of the recipe (e.g., in the title, description, ingredients, etc.), the food ontology can use one or more of its algorithms to find nodes (e.g., ingredient nodes) that are related to the "BLT" recipes. One or more algorithms can utilize multiple steps. Each node in the ingredient taxonomy can be treated as a binary variable for the population (e.g., whether each recipe is connected or not connected to "bacon", "lettuce", and / or "tomato"). In some cases, one or more algorithms can assume that the population is large enough to represent the general population and can treat the frequency of nodes in the population as a normalized approximation of the probability that a random recipe contains the node. Subsequently, for each node, one or more algorithms can pose a null hypothesis (e.g., the sample is not associated with the node, i.e., the probability that the node appears in the sample is the same as the probability that the node appears in the population), and test the null hypothesis by calculating the chance (e.g., probability score) that the sample would contain as many instances of the node as it contains. A threshold can be defined or predetermined, and when the probability of the observed data given the null hypothesis is below the threshold, one or more algorithms can reject the null hypothesis and add the node to the list of associated recipes.

[0218] Mathematically, each node can be considered a random binary variable, where probability p = the frequency of the node in the population of recipes. If the sample (or subset sample) size is n, then the number of recipes connected to the node can be considered a binomial random variable with probability p and number of repetitions n. In some cases, if k is the actual number of recipes connected to the node, calculating the probability for a given scenario or one or more extremes can include calculating the binomial survival function for p, n, and k - 1. To accept or reject the null hypothesis, as described above, a threshold range of from about 0.01% to about 10% can be used. In some cases, the threshold can be at least about 0.01%, 0.02%, 0.03%, 0.04%, 0.05%, 0.06%, 0.07%, 0.08%, 0.09%, 0.1%, 0.2%, 0.3%, 0.4%, 0.5%, 0.6%, 0.7%, 0.8%, 0.9%, 1%, 2%, 3%, 4%, 5%, 6%, 7%, 8%, 9%, 10% or more. In some cases, the threshold can be at least about 10%, 9%, 8%, 7%, 6%, 5%, 4%, 3%, 2%, 1%, 0.9%, 0.8%, 0.7%, 0.6%, 0.5%, 0.4%, 0.3%, 0.2%, 0.1%, 0.09%, 0.08%, 0.07%, 0.06%, 0.05%, 0.04%, 0.03%, 0.02%, 0.01% or less. In one instance, the above “BLT” instance can use a threshold of 0.1%, which can be considered a more “stringent” threshold compared to a 5% threshold.

[0219] Additionally, the food ontology can utilize one or more of its statistical methods as a generalization tool to identify one or more significant statistical patterns. In some cases, two users can consume French fries (e.g., daily, every other day, once a week, once a month, etc.). One user may have a negative reaction to fried foods (e.g., experience an allergic reaction, nausea, headache, etc. after consuming fried foods), while the other user may have a negative reaction to starchy vegetables (e.g., potatoes). The knowledge that French fries can be fried and contain a starchy vegetable component can be stored in the food ontology and propagated when looking for associations (e.g., by using matrix multiplication). As such, the meals of both users that contain French fries can be connected to the nodes “friends” and / or “starchy vegetables”, allowing one or more statistical methods to generalize patterns across all meals of relevant meals across users (e.g., meals that contain other fried foods of the first user and meals that contain other starchy vegetables of the second user).

[0220] Figure 57A and 57B Shows an exemplary process of using an algorithm (e.g., one or more statistical methods) to generate statistical associations between foods or between data related to foods.Figure 57A Shows exemplary results of a "BLT" text query for identifying one or more samples (recipes) among multiple recipes and how they are associated with "BLT". In this case, a query search for "BLT" (or "blt") has identified 12 samples, where the 12 samples are associated through nodes such as bacon, tomato, sandwich, lettuce, and basil. Each of the listed associations can be indicated by a probability score of the identified nodes associated with the query search.

[0221] Figure 57B Shows an exemplary way of applying one or more statistical methods to a food ontology for predictive meal recording. The food ontology may have mapped multiple (e.g., thousands of) meals of one or more users. Each meal can be described as a list of food items, where each item can be described as a set of nodes (e.g., a set of ingredients). For nodes that at least partially describe a food item (e.g., "egg") consumed by a user, one or more statistical methods can be applied to the food ontology to identify one or more nodes that the user may have consumed as part of a meal. Such one or more nodes can be identified with a probability score of one or more nodes that the user may have consumed, and / or can be ranked by this probability score. In some cases, one or more nodes can be classified by two or more associations, such as "internal" associations (describing one or more attributes of the food item of interest, e.g., "boiled" for a boiled egg, "fried" for a fried egg, etc.) and "external" associations (e.g., describing one or more attributes of different food items that one or more users previously consumed with the egg, such as cheese, bread, etc.). Each of the listed associations can be indicated by a probability score of the identified nodes associated with the query node.

[0222] As described above, a food ontology can be used for food generalization or meal generalization. A user can generate a food log (e.g., a collection or written list of images of foods or meals consumed by the user) directly (e.g., by writing down each meal in a digital diary) or indirectly (e.g., by taking and saving pictures of meals on a mobile phone to post them on social media). In some cases, it may be beneficial to generalize a meal into one or more general meals to facilitate (e.g., automatically facilitate) the user's food recording experience and / or understand the user's common meal trends.

[0223] Metrics for comparing multiple meals (e.g., at least 2, 3, 4, 5, 6, 7, 8, 9, 10 meals or more; at most 10, 9, 8, 7, 6, 5, 4, 3, or 2 meals) can be similar to or based on metrics used to compare and evaluate similarities between primary food items. For primary food items, various methods can be utilized to define similarities between nodes in a food ontology. In some cases, the methods can emphasize specificity (e.g., "fried egg" may be more specific to "egg" compared to "chicken", and the similarity metric increases as the distance between nodes in the food ontology decreases) and / or common ancestors (e.g., "French fries" and "mashed potatoes" can share a common "potato" ancestor, and the similarity metric increases as the number of common ancestors between nodes increases). In the context of meals, measuring or identifying the similarity between two meals can include iteratively identifying the common components (e.g., food items) between the two meals, identifying the component that is more common than the other component (i.e., the closest component), "matching" the closest component and removing it from both meals. This process can be repeated until there are no remaining components in at least one of the two meals, where the meal without remaining components can be labeled as the "small meal", and the meal with remaining components can be labeled as the "large meal". Subsequently, similarity can be generated based on one or more metrics (e.g., a first percentage of common components in the smaller meal, a second percentage of common components in the larger meal, the average of the first and second percentages, the difference between the first and second percentages, the number of common components).

[0224] By using one or more of the metrics described herein or modified forms thereof, diets can be summarized by grouping meals into one or more different groups (or subgroups). One method of grouping meals can include clustering, such as hierarchical clustering, density-based spatial clustering of applications with noise (DBSCAN), variants thereof, or combinations thereof. Another method of grouping meals can include multidimensional scaling (MDS) for visually presenting and arranging meals by their similarity and developing an interface for users, supervisors, and / or computational algorithms to group meals.

[0225] Figure 58A and 58B Shows an exemplary process for grouping meals based on MDS. As Figure 58AAs shown, MDS can visually present multiple meals (represented by individual points or circles). Multiple meals can be arranged by their similarity, and the similarity can be indicated by colors 5810, 5820, 5830, 5840, 5850, 5860, and 5870 and their corresponding positions relative to each other. For example, foods indicated by color 5830 may have a wide range of similarities when presented in a group, where food 5830a arranged in the upper left of the crescent shape and food 5830b arranged in the lower right of the crescent shape may show the lowest degree of similarity to each other within the group of color 5830. For example, color 5830 may indicate multiple fast food meals, where food 5830a may represent a vegetarian meal (e.g., Thai tofu fried rice noodles), and food 5830b may represent a non-vegetarian meal (e.g., Korean beef short rib barbecue). In some cases, a single food indicated by color 5870 may represent a single meal. The graphical representation in Figure 58A can be provided to users, supervisors, and / or computational algorithms to group meals.

[0226] Figure 58B Shows the results of grouping the meals presented in Figure 58A . As shown in Figure 58B , the meals previously arranged by colors 5810, 5820, 5830, 5840, and 5850 are grouped into a new group 5880, while the meals previously indicated by colors 5860 and 5870 are grouped into a new group 5890.

[0227] For meal generalization, once multiple meals have been grouped into one or more groups (e.g., as shown in Figure 58A and 58B ), the generalized meals for each group can be identified (or generated). One method may include identifying one or more branches (nodes within each meal) that appear in all meals and selecting the one or more nodes that are most common (most prevalent, e.g., at least 30%, 40%, 50%, 60%, 70%, 80%, 90%, 99% or more) among the meals within each meal group. Alternatively or additionally, the method may include evaluating the nodes that may be common among the meals within each meal group and then generating a new meal that most represents the one or more nodes that are common among the meals (e.g., at least 30%, 40%, 50%, 60%, 70%, 80%, 90%, 99% or more). In some cases, the newly generated meals can be included in the pool of multiple meals, and such a pool can be re-evaluated for meal generalization to confirm that the newly generated meals are indeed the most representative or some of the generalized meals for the previously identified meal groups.

[0228] Importantly, the food analysis system 210 can standardize data from other nutrition trackers. Data from such databases may be unstructured, fragmented, and / or disorganized and may generally be incompatible with each other. The food analysis system 210 can (1) obtain data from other nutrition trackers, (2) convert such data into structured data having a common format, and (3) organize the structured data into multi-layered information in a food ontology. Thus, the food ontology can be used as a standardized meta-object existing on top of existing food and / or nutrition databases. For example, the food analysis system 210 can analyze and map "burritos" from each of multiple databases (e.g., MyFitnessPal, LoseIt, FatSecret, etc.) and track a user's food and / or beverage intake.

[0229] Figure 41 An exemplary network layout 4100 between the food analysis system 210 of the platform 200 and one or more nutrition trackers 4110a–4110c is shown. The one or more nutrition trackers 4110a–4110c can communicate digitally with APIs 4120a–4120c, respectively. The APIs 4120a–4120c can communicate digitally with: (1) one or more databases 4130a-4130c for storing data, and (2) software and / or applications including a GUI for receiving data from and / or sending data to a user ( Figure 41 not shown). The APIs 4120a–4120c of the one or more nutrition trackers 4110a–4110c can allow a user to record the user's food intake and provide nutritional information and calorie information about the user's food intake. The APIs 4120a–4120c can feed all the data into one or more databases 4130a–4130c, respectively. The data in the one or more databases 4130a-4130c is unstructured, fragmented, and / or disorganized. Thus, when receiving data about food from the one or more nutrition trackers 4110a–4110c, the food analysis system 210 uses one or more algorithms to (1) convert the data into structured data and (2) organize the structured data into multi-layered information in a food ontology 4140. Such structured data is standardized to the common format of the food ontology 4140. Thus, food items containing the abundance of information from one or more databases 4130a–4130c of each of the one or more nutrition trackers 4110a–4110c are mapped to the food ontology 4140, where the food ontology 4140 organizes information by constructing one or more layers.

[0230] When combined with additional health-related data and / or machine learning algorithms of platform 200, the food ontology may be useful for a number of applications. Examples of such applications may include, but are not limited to: (1) estimating nutritional values ​​of recipes and / or restaurant dishes; (2) providing food and health recommendations to a user and gaining an understanding of the user's taste profile (e.g., the user's preferences for food dishes or food types); (3) building a food log; (4) generating missing primary foods from existing packaged foods; (5) generating more accurate labels for food properties; (6) analyzing food costs; (7) modeling the effects of cooking on nutritional values ​​and estimating the degree of food processing; (8) improved image classification or computer vision classification of foods; (9) improved analysis of speech-based food logs; (10) identifying food substitutes or replacements (e.g., making a recipe gluten-free, making a recipe less salty, making a recipe "healthier," etc.); and (11) creating one or more new foods (e.g., generating new food dishes from existing foods, generating new primary foods from existing packaged foods, etc.).

[0231] Devices / Data Hubs

[0232] The device / dataset aggregator 220 can generate a personalized data network for the user between the platform 200 and the device 110 and one or more third-party databases 130. The device / dataset aggregator 220 can communicate digitally with: (i) one or more devices (e.g., a personal device such as a mobile phone) including food data, health data, and / or nutrition data; and / or (ii) one or more databases including food data, health data, and / or nutrition data. The device / dataset aggregator 220 can collect and aggregate food data, health data, or nutrition data. The device / dataset aggregator 220 can be a system for collecting and aggregating multiple datasets from multiple application programming interfaces (APIs). The multiple datasets can be provided in two or more different formats. The multiple datasets can contain multiple physiological inputs associated with the user. The multiple datasets can be collected from additional sources, such as one or more third-party databases 130 (e.g., healthcare providers). The device / dataset aggregator 220 can automatically aggregate the user's food data, biomarker data, and health data (e.g., nutrition, activity, sleep, genetics, glucose, menstrual cycle, etc.). Such data of the user can be continuously streamed by the device / dataset aggregator 220. The device / dataset aggregator 220 can be connected to one or more devices and / or one or more services and collect one or more data points per month. The device / dataset aggregator 220 can be connected to more than 100 devices and services and collect approximately 400 million or more data points per month. Regardless of its source, all incoming data can be fully integrated into the format of the device / dataset aggregator 220. All incoming data can be fully integrated into the software framework of the device / dataset aggregator 220. The software framework can be a web framework (WF) or a web application framework (WAF).

[0233] The device / dataset aggregator 220 can be a serverless system for storing the raw data from the device 110 before analysis by the food analysis system 210 or the insights and recommendations engine 230. One or more changes in the APIs will not affect the raw data stored in the device / dataset aggregator 220. No raw data will be lost in the device / dataset aggregator 220. The device 110 and one or more databases 130 and their corresponding APIs compatible with the device / dataset aggregator 220 can include mobile devices, wearable electronic devices, medical devices, point-of-care (POC) devices or kits, sensors, etc. The sensors can include glucose sensors, GPS receivers, heart rate monitors, galvanic skin response (GSR) sensors, skin temperature sensors, capacitive sensors, and metabolic sensors. Each sensor can be a discrete device with a discrete API. Each sensor can be an integrated component or function of one or more devices in the device 110.

[0234] Instances of the apparatus 110 and its corresponding data types are provided. From an Abbott glucose monitor, data on blood glucose levels can be obtained. From Fitbit, data including activity, steps, weight, and sleep can be obtained. From Jawbone, data including activity, steps, weight, and sleep can be obtained. From GoogleFit, data including activity and steps can be obtained. From Moves, data on activity can be obtained. From Runkeeper, data including activity, weight, and sleep can be obtained. Additional types of the apparatus 110 can include smart clothing with a heart rate monitor, a sensor-implemented mattress that tracks and adjusts an individual's snoring, earplugs equipped with an in-ear thermometer, an artificial intelligence-embedded toothbrush that collects brushing data (frequency, duration, areas brushed, etc.) through sensors, a smart ring that tracks an individual's biomarkers (e.g., activity, steps, sleep, heart rhythm, etc.), a wearable electrocardiogram monitor, a portable air quality tracker, a medication attachment hub designed to be placed near a user's medication tablets and alert the user of scheduled medication, an electronic cigarette or vaporizer, a smart appliance designed to counteract hand tremors and other instability conditions caused by Parkinson's disease, a pregnancy-tracking wearable device that can help a woman track and understand uterine contractions, a hearing aid, a sensor for measuring antioxidants in the skin, an implantable (e.g., by swallowing) sensor for diagnosing gastrointestinal problems or measuring food intake and / or digestion status, a device that sits in the mouth to detect sounds emitted by chewing food, a portable spectrometer for measuring the absorption spectrum of food, and a portable mass spectrometer for providing a partial chemical composition of food.

[0235] The platform 200 that includes the apparatus / data aggregator 200 can be implemented in a GUI-based software interface. The GUI-based software interface can connect the platform 200 that includes the apparatus / data aggregator 220 to a user device (e.g., a personal computer, a smartphone, etc.). The GUI-based software interface can be compatible with any operating system of the user device. In use, the GUI-based software interface can access the data in the user device or the data accessible through the user device without restriction. The GUI-based software interface can automatically collect data without relying on user input. The data in the user device can include pictures, videos, voice recordings, text, location services, etc. The data accessible through the user device can also include the data in the user's cloud storage service. The data accessible through the user device can include the data from at least one third-party application that connects the user device to one or more third-party devices (e.g., a glucose monitor, a temperature sensor, etc.).

[0236] The device / data hub 220 can be a seamless food image recorder. By using a GUI-based software interface that can be installed in the user device, the device / data hub 220 can have unrestricted access to the camera roll of the user device. Each time the user device takes an image, the convolutional network of the device / data hub 220 can analyze the image to determine whether the image contains or does not contain at least one food or beverage. The convolutional network can analyze images when the user uses applications other than the GUI-based software interface (e.g., the photo application of the user device, Instagram, etc.). If the convolutional network can identify at least one food or beverage in the image, the image is automatically aggregated in one or more databases 240 together with a timestamp and a geographical location. The stored image data can be used for analysis by the food analysis system 210 and the insights and recommendations engine 230. The analysis can include studying how the user's body metrics (e.g., blood sugar level, sleep time, number of steps, etc.) may be affected by at least one food or beverage in the image. The seamless food image recorder function of the device / data hub 220 can facilitate the tedious and ongoing process of food tracking required by the food analysis system 210. An exemplary window for the seamless food image recorder function of the GUI-based software interface is shown in Figures 17A - 17D In the user device during or after installation, the GUI-based software interface can request the user to access the camera roll of the user device.

[0237] The device / data aggregator 220 can perform food tracking through text and speech recognition analysis. By using a GUI-based software interface installed in the user device, the user can record and store at least one free text or at least one voice message about food (e.g., food the user has eaten, food the user plans to eat, food the user wants to know more about, etc.) into the device / data aggregator 220. The device / data aggregator 220 can automatically convert at least one voice message into the corresponding free text by using a third-party service (e.g., Speech2Text). The data of the stored at least one free text or at least one voice message can be used for analysis by the food analysis system 210 and the insights and recommendations engine 230. In one example, if the user types the free text "1 slice of bread plus two eggs and a cup of coffee" into the GUI-based software interface, the device / data aggregator 220 can save the data of the free text into one or more databases 240 and instruct the food analysis system 210 to perform data analysis. The food analysis system 210 can abstract and classify the nutritional information (e.g., carbohydrate or nutrient intake) of the food mentioned in the free text and map the food to a food ontology. Subsequently, the device data aggregator 220 communicating with the food analysis system 210 can inform the user of the results of the analysis. The text and speech recognition functions of the device / data aggregator 220 can facilitate the tedious and ongoing process of food tracking required by the food analysis system 210. An exemplary window for the voice recognition analysis function of the GUI-based software interface is shown in Figures 18A - 18C as follows. The GUI-based software interface window can display an example sentence structure that the user can use to record a voice message into the device / data aggregator 220, as shown in Figure 18A as follows. After automatically converting the voice message into free text, the GUI-based software interface window can display the free text to the user, as shown in Figure 18B as follows. After immediately analyzing the food-related information from the free text, the GUI-based software interface window can display the results of the analysis (e.g., food items and predicted calories and serving sizes) and also ask the user to verify or edit the accuracy of the results before saving the results in one or more databases 240, as shown in Figure 18C as follows.

[0238] The device / data aggregator 220 can communicate with a number of medical devices and healthcare databases to continuously stream and store the user's personalized data. The stored user's personalized data can be used for analysis by the insights and recommendations engine 230. The user's personalized data can include glucose levels. The device / data aggregator 220 can communicate with a glucometer. The device / data aggregator 220 can communicate with a continuous glucose monitoring (CGM) device, which is also referred to as a real-time CGM (RT-CGM) device. The combination of the CGM device and its corresponding GUI on the device and / or on the user device can determine the glucose level in the blood on a continuous basis. The CGM device can monitor the glucose level of the interstitial fluid, such as is closely related to the blood glucose level. The measurement result of the glucose level of the interstitial fluid can have an error of up to about 15%, 14%, 13%, 12%, 11%, 10%, 9%, 8%, 7%, 6%, 5% or less compared to the corresponding blood glucose level. The measurement result of the glucose level of the interstitial fluid can have a relatively short delay compared to the blood glucose level. The CGM device can remain functional during the user's daily activities, including showering, exercising, sleeping, etc. The CGM device can be used for users with type 1 diabetes or type 2 diabetes to evaluate when the user should inject insulin. The CGM device may be useful for athletes to optimize their athletic performance. The CGM device may be useful for individuals interested in tracking food intake as a way to monitor their weight loss diet. The CGM device can include any suitable continuous glucose monitoring device.

[0239] The network according to the present disclosure can include the device / data aggregator 220, the device 110, one or more third-party databases 130, and one or more databases of the platform 240. The platform 240 can be a virtual private cloud (VPC) for data storage (e.g., Amazon VPC). The network can utilize a number of independent software components, which are collectively referred to as the open-source Apache HadoopTM stack. These components can include products such as TM Cassandra TM CloudStack TM HDFS, Continum TM Cordova TM Pivot TM Spark TM Storm TM and / or ZooKeeper TM The machine learning / NLP algorithms described herein can utilize existing technology machine learning libraries, including those common libraries built on Apache Spark

[0240] Insight and Recommendation Engine

[0241] The insight and recommendation engine 230 can (1) access the food ontology in the food analysis system 210, (2) access a large amount of personal biomarker data from the device / data aggregator 220, (3) analyze how food affects the user's biomarkers, and (4) continuously generate personalized nutritional recommendations for the user. The relationship between the three components of the platform 200 is shown in Figure 20 . After analyzing and validating how food can affect the user's biomarkers, the insight and recommendation engine 230 can generate one or more personalized digital signatures unique to the user. The personalized digital signature can be an algorithm for estimating the response of a user's specific biomarker to the consumption of a specific food item. For example, the digital signature of the blood glucose (or sugar) level can take into account that the blood glucose levels of two individuals at the same time may respond differently to the consumption of the same food items (coffee, apple, and sandwich), as shown by the curves in Figure 21 . One or more personalized digital signatures can be unique for many other factors related to an individual, including gender, age, race, genetics, microbiome, religious dietary restrictions, geographical location, height, weight, and time of day, month, or year. In addition to the end users of the insight and recommendation engine 230 and other functions provided in this disclosure, other potential beneficiaries include healthcare device manufacturers, third-party big data, third-party databases, and insurance companies, as shown in Figure 22 .

[0242] The insight and recommendation engine 230 can determine the various effects of food consumption on the user's body by applying at least one predictive model to multiple datasets. The multiple datasets can include the food consumed by the user and the physiological inputs associated with the user. Such data can be obtained from multiple sources including discrete APIs. The multiple datasets can also include information from the food ontology about the food consumed by the user. Applying at least one predictive model to the multiple datasets can generate multiple personalized food and health metrics for the user.

[0243] The insights and recommendation engine 230 can include many analytical and deep learning algorithms, including statistical analysis and artificial neural networks (ANNs). An ANN can be a mathematical or computational model inspired by the structural or functional aspects of a biological neural network. An ANN can include a group of interconnected artificial neurons (units). An ANN can be an adaptive system configured to change its structure (e.g., the connections between units) based on external or internal information flowing through the network during a learning phase. ANNs can be used to model complex relationships between inputs and outputs or to find patterns in data where dependencies between inputs and outputs cannot be easily achieved. In some instances, the complex relationships can include how food affects a user's body in terms of multiple biomarkers.

[0244] As an alternative to or in addition to ANNs, the insights and recommendation engine 230 can include biomathematical prediction models using metric spaces, decision trees, and decision tree learning algorithms. A metric space can provide a "ruler" or absolute measure of how two feature vectors differ. A metric space can be used to define the "distance" between two feature vectors. A decision tree can be a tree diagram or decision model along with a supporting tool for its possible consequences. A decision tree can include one or more leaf nodes (leaves) representing the final decision. The complete path to one or more leaf nodes can represent the rules for arriving at one or more decisions, respectively. A decision tree learning algorithm can be an inductive machine learning mechanism that infers accurate predictions about future events (unknown) based on a given set of past (known) events. A decision tree learning algorithm can also provide a measure of the confidence (e.g., coverage, accuracy, and confidence interval) of a correct prediction. The minimum confidence interval of the accuracy of the tree learning algorithm can be maintained at at least about 70%, 75%, 80%, 85%, 90%, 95% or higher.

[0245] The insights and recommendation engine 230 can use the data collected and analyzed by the food analysis system 210 and the device / dataset aggregator 220 to generate at least one decision tree learning algorithm. At least one decision tree learning algorithm can be used to predict how the food previously consumed by the user can affect the user's personalized biomarkers (e.g., glucose level). At least one decision tree learning algorithm can also be used to predict how food that the user has never consumed or lifestyle events that may affect the user's biomarkers (e.g., glucose level) will.

[0246] An example of how the insights and recommendation engine 230 can analyze a collection of data aggregated by the device / dataset aggregator 220 is described in detail. The device / dataset aggregator 220 communicating with the user's CGM device can continuously record the user's blood glucose level as a function of time, as shown by the curve in Figure 23 The device / dataset aggregator 220 can also record which foods the user has consumed and their corresponding timestamps.Figure 23 It shows that the user consumed cereal and milk at 8:00 am, nuts at 9:00 am, and dates at 10:00 am. The data also shows different degrees of peaks in the blood glucose level graph. To analyze part of the data and generate a "blood glucose level" digital signature for the user, the insights and recommendations engine 230 can extract known events (such as food consumption) and their associated blood glucose levels, and classify the known events by the degree of change in the blood glucose level in response to each event. Decomposition can be performed using a biomathematical prediction model. Figures 24A - 24B It shows how six different food items can be classified into two trend groups based on their impact on an individual's blood glucose level. Figure 24A The graph shows that white bread and steamed buns (also from white wheat) can similarly and negatively affect the user's blood glucose level. A relatively large change in the blood glucose level can imply insulin-mediated fat storage in response to the corresponding food. On the other hand, Figure 24B the graph in shows that coconut ice cream, lentils, and salad have a relatively small impact on the user's blood glucose level. Although not shown in Figures 24A - 24B it, other biomarkers can be analyzed in a similar manner.

[0247] The insights and recommendations engine 230 can generate and use one or more digital signatures of the user to provide one or more recommendations to the user. The one or more recommendations can relate to food, health, or wellness. In some instances, the insights and recommendations engine 230 can suggest meal plan recommendations customized for an individual's body and its responses. The recommendations can include which specific foods to consume, where to find the specific foods (such as the name and location of a restaurant), the basic ingredients of the specific foods, how to prepare the specific foods (such as cooking methods), when to consume the specific foods (such as between 4:30 - 5:30 pm), how much to consume, which activities or steps should be performed (or avoided) after consuming the specific foods, etc. The insights and recommendations engine 230 can also track what the user likes and dislikes to eat. The insights and recommendations engine 230 can also predict which other types of foods the user will like or dislike, and use such one or more predictions to generate personalized recommendations. Personalized recommendations can result in a high user compliance rate.

[0248] When the user is using a GUI-based software interface, the insights and recommendations engine 230 can send one or more personalized messages to the user. Additionally, when the user is not using a GUI-based software interface, one or more personalized messages can be pop-up messages and emails to the user's device. The one or more personalized messages can recommend that the user consume less (or more) of one or more food items, or stop (or start) consuming one or more food items. The insights and recommendations engine 230 can send the predicted impact of one or more foods on the user's body.

[0249] The insights and recommendation engine 230 can recommend recommendations that can drive behavioral changes in the user. The behavioral changes can be preferences selected by the user, or recommendations generated by the insights and recommendation engine 230. The behavioral changes can include eating less carbohydrates to lose weight. For example, a user may often eat pizza for lunch, and the insights and recommendation engine 230 can detect that the consumption of pizza is associated with a sharp increase in the user's blood sugar level. The insights and recommendation engine 230 can identify other foods that can lower the blood sugar level when eaten with pizza. The insights and recommendation engine 230 can also identify one or more alternative food items to replace pizza.

[0250] By using a GUI-based software interface in the user device, the insights and recommendation engine 230 can receive menu inputs, track the user's geographical location using the GPS of the user device, search for nearby restaurants, and recommend different menu items available to the user. The insights and recommendation engine 230 can provide ordering tips and reasons for different menu items. An exemplary window of the GUI-based software interface for personalized recommendations of menu items is shown in Figures 25A - 25C in.

[0251] The insights and recommendation engine 230 can be used for individuals with type 1 diabetes or type 2 diabetes. An individual's blood glucose level may be affected by the foods consumed and the individual's lifestyle (e.g., physical activity, sleep, stress, etc.). If the blood glucose level is too high, the individual's body may secrete a hormone called insulin to help regulate the blood glucose by guiding fat cells to absorb glucose. Insulin can also guide other cell types to absorb blood glucose as a source of energy. For diabetes, due to impaired ability of the body to produce insulin or respond to insulin, the blood glucose level may be higher than normal. Individuals with type 1 diabetes may produce insufficient insulin in the individual's body. Individuals with type 2 diabetes may produce insufficient insulin and / or have insulin resistance in the body. Individuals with type 1 or type 2 diabetes can rely on insulin injections to control their blood glucose levels. Thus, the insights and recommendation engine 230 can (1) monitor the user's food intake, blood glucose level (continuously from a CGM device or discretely using a conventional blood glucose meter), and insulin level of users using insulin injection therapy; and (2) analyze the relationship between specific food types, insulin injections, and blood glucose level responses. The insights and recommendation engine 230 can use a GUI-based software interface in the user device to display recommendations to the user. Based on the recommendations, the user can specifically identify which food items the user's blood glucose level responds most strongly to, which food items to avoid or consume more, the optimal time intervals for insulin injections, etc. In one example, the recommendation can suggest, "When you add avocado to your sandwich, your glucose response will be more than 30% lower [happening 5 out of 6 times]." Other biomarkers that can be collected by the insights and recommendation engine 230 and associated with insulin and blood glucose levels can include exercise, stress, activity, medication, menstrual cycle, etc. The combination of food with at least one or more of the other factors can improve the quality of the recommendations generated by the insights and recommendation engine 230.

[0252] Figure 26 An exemplary window for blood glucose recording showing a GUI-based software interface is presented. Blood glucose recording can be performed using a conventional blood glucose meter or a CGM device. The window can display the food items (e.g., avocado toast) that the user has consumed. The window can display the change in the user's blood glucose level at a specific time point after consuming the food item. Additionally, the window can display a report on the user's blood glucose level profile over a period of time. The time period can capture time points before and after consuming the food (e.g., 2 hours and 3 hours before or after a meal). The insights and recommendation engine 230 can change the length of the time period and the number of time points within the time period based on the profile of food and blood glucose-related data. Based on the recommended range of blood glucose levels, the insights and recommendation engine 230 can evaluate whether each measured blood glucose level among the measured blood glucose levels is within or outside the recommended range and display the evaluation on the window.

[0253] Figure 27 Disclosed is an exemplary window for displaying recommendations based on automated blood glucose records in a GUI-based software interface. The automated blood glucose records can use a CGM device. The CGM device can communicate with the GUI-based software interface in a user device (e.g., via Bluetooth, Wi-Fi, etc.). The GUI-based software interface can be connected to and utilize all features of platform 200, which features include a food analysis system 210, a device / dataset aggregator 220, and an insights and recommendations engine 230. The window can display a graph of the user's blood glucose levels, including the most recent measurements from the CGM device. For food items that the user may be interested in consuming, the insights and recommendations engine 230 can generate insights (e.g., free text, images, graphs, etc.) on how past food items have affected the user's blood glucose levels. The insights can be displayed to the user on the window. The insights can help the user make informed decisions about the consumption of food items. If the user has a wearable insulin delivery device, the insights on the window can also inform the user of different bolus options available in the wearable insulin delivery device.

[0254] The insights and recommendations engine 230 can use one or more biomathematical models described in the present disclosure to predict the user's general biomarkers. In one example, the insights and recommendations engine 230 can predict the user's glucose metabolism. The glucose metabolism process can begin with digestion. After digestion, glucose can be absorbed into the bloodstream when it enters the small intestine. When the blood glucose level increases, the pancreas can release a hormone called insulin to control the blood glucose. Insulin can help transfer glucose into various cell types that have insulin receptors. Examples of cell types can include adipocytes (adipose tissue), muscle myocytes (muscle), and hepatocytes (liver). Thus, the user's glucose metabolism can depend on one or more factors, including but not limited to the user's glucose and insulin production levels, the blood glucose level before food consumption, the carbohydrate content in the food, the insulin level in the body, blood pressure, physical activity, the user's insulin sensitivity, the time of day, stress, illness, pregnancy, medication, etc. Thus, there can be many ways to generate a biomathematical model with one or more mathematical parameters to describe and predict the relationship between such factors and the user's glucose metabolism. In some examples, some of the factors may be more relevant to glucose metabolism than other factors. Additionally, the relevance of such factors to the user may change over time.

[0255] In one example, the Glucose Absorption and Insulin Assimilation (GAIA) model can be a biomathematical model used to describe and predict a user's glucose metabolism and its interaction with insulin. For a patient receiving insulin injections or endogenous insulin, the insulin can be injectable (exogenous) insulin. The GAIA model can use the user's historical data on food consumption and blood glucose and insulin levels to predict the glucose response. The GAIA model can use two, three, four, five, six, seven, eight, nine, ten, eleven, twelve, or more historical meals from the user, along with their corresponding glucose and insulin levels, to predict the user's glucose and insulin response to one or more new meals. FIG. 28 shows a flow chart of the GAIA model for predicting glucose and insulin interaction in the body. Figure 29 An exemplary graph showing the measured glucose levels and the GAIA-estimated blood glucose levels is shown.

[0256] The GAIA model can be defined by using the equations for glucose and insulin in the blood. In one example, the glucose model can be a differential equation of the concentration of glucose in the blood over time, as shown by the following equation:

[0257]

[0258] where is the rate of glucose absorption from food in the blood,

[0259] is the rate of endogenous glucose production by the liver,

[0260] is the rate of glucose utilization by the body, and

[0261] represents the glucose-insulin interaction in the blood.

[0262] In a healthy individual with normal blood glucose levels, the total glucose intake (utilization rate + interaction; or ) can range from 1.9 to about 2.2 . Figure 29 FIG. shows a four-arm blood glucose model modeled using the above equation.

[0263] The rate of glucose absorption from food can depend on several parameters. Such parameters can be meal-related or meal-unrelated. For example, the rate of glucose absorption from food can be determined by four meal-related parameters is described, and the parameters can represent the rate of increase or decrease in glucose absorption and the total amount of glucose absorbed after a meal is given.

[0264] A simplified model can be constructed that shows the desired glucose absorption rate for a meal during the immediate duration after a meal (e.g., less than 2 hours) or for a non-complex meal over an extended period (e.g., more than 2 hours, 3 hours, 4 hours, 5 hours, 6 hours, 7 hours, 8 hours, 9 hours, 10 hours, or more hours). If a meal contains a large amount of fat or protein, glucose absorption may be slower. Thus, calculating glucose absorption can involve multiple maxima not captured by this model. In such cases, the model can be used only during the immediate duration after the meal (e.g., less than 2 hours). The rate of endogenous glucose production (EGP) entering the bloodstream through the liver depends on several parameters. Such parameters can be meal-related or meal-unrelated. For example, the rate of EGP can be described by two meal-related parameters and I(t), where the parameters are functions of insulin in the blood and where represents the baseline endogenous glucose production in the absence of insulin, and represents the rate of decrease in endogenous glucose production with plasma insulin.

[0265] The rate of glucose utilization can depend on several parameters. Such parameters can be meal-related or meal-unrelated. For example, the rate of glucose utilization can be described as a function that grows non-linearly during the duration of a meal, and the function can depend on three meal-unrelated parameters, where and represent two asymptotes of the glucose utilization rate (at the initial time and the final time, respectively), and represents the rate of change of the utilization rate . The rate of glucose utilization may be useful for breakfast because it is expected that glucose utilization rises in the morning after an individual wakes up from sleep.

[0266] The glucose-insulin interaction in the blood can be meal-related or meal-unrelated. For example, the glucose-insulin interaction in the blood can be modeled as a meal-unrelated non-linear interaction. The glucose-insulin interaction in the blood can be a function of different parameters, such as blood glucose concentration , plasma insulin concentration , insulin sensitivity and which represents how much insulin sensitivity decreases as the blood glucose level of each patient increases. The interaction term can be approximated using a Taylor series around different blood glucose concentrations.

[0267] The concentration of plasma insulin may vary for each patient. For example, for patients with type 1 diabetes for more than one year, it can be assumed that all insulin in the body is provided by an exogenous source . The source can be an insulin pump or manual insulin injection. An insulin model for patients with type 1 diabetes can use a three-compartment model, as described by the following differential equations:

[0268]

[0269] where is the volume of insulin at the injection site,

[0270] is the volume of insulin in the interstitial (interstitial fluid),

[0271] is the plasma insulin concentration as a function of time,

[0272] is the rate of insulin provided by the source (usually an insulin pump).

[0273] The source can be represented as a superposition of a fixed basal rate, a square bolus, and a normal bolus (represented by a δ function). The rate constants for insulin transfer through each compartment are represented by the matrix M:

[0274]

[0275] where is the mass of the patient in kilograms,

[0276] is the rate constant for insulin transfer from the injection site to the interstitial fluid (1 / hour),

[0277] is the rate constant for insulin transfer from the interstitial fluid to the plasma (1 / hour),

[0278] is the rate constant for insulin loss at the site (1 / hour),

[0279] is the rate constant for the transfer of insulin from interstitial fluid outside the blood (unutilized insulin; 1 / hour),

[0280] is the rate constant for the disposition of plasma insulin (unutilized insulin; 1 / hour), and

[0281] is the effective volume of plasma per body weight (liters per kilogram).

[0282] Figure 30 A flow chart showing this spread of exogenous insulin from injection into interstitial fluid to plasma and the corresponding rate constants is shown. The flow chart also includes potential losses of insulin and their rate constants.

[0283] As shown above, by way of example only, the GAIA model can utilize 11 parameters, including 4 glucose absorption parameters ( ), 2 endogenous glucose production parameters ( ), 3 glucose utilization parameters ( ), and 2 glucose-insulin interaction parameters ( ). Some of the parameters can be meal-related and thus can differ from each other between meals. Some of the parameters can be meal-independent and thus can remain fixed during multiple meal durations. Some parameters can switch between meal-related and meal-independent from meal to meal.

[0284] The GAIA model can use at least one prediction pipeline to make at least one prediction of changes in a user's blood glucose level. In one instance, the pipeline can include the following steps: (1) identify one or more historical meals with sufficient and reliable data (e.g., historical meals with known ingredients, where glucose and insulin levels are tracked for 2 hours before and 4 hours after the meal, respectively); (2) fit the measured glucose and insulin levels of each of the one or more historical meals to the above model / equation and obtain the parameter space of all physical values and 11 parameters of the GAIA model, where the parameter space can be a set of all possible combinations of physical values and 11 parameters; (3) generate a distribution function of the parameter space; (4) repeatedly sample each combination from the parameter space by: (i) recalculating meal-related parameters with the generated meal-independent parameters; (ii) calculating the error in the fit of each parameter; (iii) calculating the total error over all meals; and (iv) repeating (i)-(iii) as long as the total error decreases; and (5) generate a personalized prediction model for the user using the finally determined parameters. Additionally, the insights and recommendations engine 230 can use the finally determined parameters and machine learning to study how other biomarkers of the user may affect meal-related parameters.

[0285] Figure 31 An exemplary fit 3100 of the GAIA model is shown. Line 3110 is the rate of change of the measured blood glucose level 3115. Line 3120 is the rate of change of the estimated blood glucose level 3125. The rate of change of the estimated blood glucose level is similar to the rate of change of the measured blood glucose level. Line 3130 is the glucose-insulin interaction in the blood 3135. Line 3140 is the rate of endogenous glucose production by the liver 3145 minus the rate of glucose utilization by the body. Line 3150 is the rate of glucose absorption from food in the blood 3155.

[0286] The insights and recommendation engine 230 can be used for passive food tracking. Factors such as physical activity and sleep can be passively and automatically tracked via wearable devices (e.g., Apple Watch, Fitbit, Samsung Gear, Samsung Galaxy Watch, Android Wear, etc.). Additionally, blood glucose levels can be passively and automatically tracked via a CGM device. On the other hand, tracking food intake may require active and frequent intervention by the user (e.g., manual recording via voice or text). This food intake tracking can be a tedious and unreliable process for data collection. By using machine learning, the insights and recommendation engine 230 can combine a set of user-specific parameters to generate a predictive model for passive food tracking. The parameters can include the user's geographical location using GPS on the user device. The parameters can include changes in the user's blood glucose levels monitored via the automatic blood glucose recording function. The parameters can include the user's historical food and / or beverage consumption and blood glucose response data, as well as the food ontology of the food analysis system 210. In one example, when a spike occurs in the user's blood glucose level, the insights and recommendation engine 230 can (1) search the user's historical food items with similar blood glucose responses (e.g., intensity and duration); (2) generate a list of available foods near the user; (3) use the GAIA model to predict the user's blood glucose response to each of the available foods in (2); (4) find food items that the user has consumed repeatedly; (5) find common food items with similar glucose profiles in steps (1) through (4); and (6) predict the common food items that the user is most likely to consume within the past 2 - 3 hours. The foregoing steps can be used to assist passive food tracking. Additional parameters for generating a predictive model for passive food tracking can include the sound generated by chewing food, the absorption spectrum of the food, and the partial chemical composition of the food.

[0287] The insights and recommendation engine 230 can predict the user's meal patterns or habits. Up to 78% of meals can be repetitive for an individual's diet itself. The insights and recommendation engine 230 can find recurring patterns in the diet based on: (1) the user's historical meal or beverage consumption data; (2) the relationships between different foods derived from the food ontology, and (3) location and / or time of day. For example, as shown in FIG. Figures 32A - 32BAs shown, a user may have the habit of consuming bananas, red tomatoes, whole wheat toast, and white rice on the first day, and broccoli rice on the second day. The next time the user consumes a sub - combination or the whole of bananas, red tomatoes, whole wheat toast, and white rice, the insights and recommendation engine 230 can predict that the next meal will be broccoli rice. The insights and recommendation engine 230 can use a GUI - based software interface to require the user to confirm or correct the prediction before recording the meal. Based on the user's response, the insights and recommendation engine 230 can confirm or improve its dining pattern prediction algorithm. In another example, the user can enter "fried eggs" in a GUI - based software interface, and the insights and recommendation engine 230 can predict that the next most likely foods to be recorded may be "bread" and "coffee" and automatically complete the user's meal as "fried eggs with sliced bread and a cup of coffee." This auto - completion ability can allow the user's clicks and / or inputs to be reduced by at least 30%, 40%, 50%, 60%, 70% or more.

[0288] Thus, if the user opens the GUI - based software interface on the user device, the insights and recommendation engine 230 can (1) use GPS to detect the time and / or the user's geographical location; (2) search for the user's repeated historical meals at similar times or geographical locations; (3) generate a list of available meals near the user; (4) find the common food items in (1)-(3); (5) predict which common food items the user is most likely to consume soon. As Figure 32B shown, the insights and recommendation engine 230 can use a GUI - based software interface to ask and / or confirm whether the prediction is correct. Based on the usage response, the insights and recommendation engine 230 can confirm or improve its dining pattern algorithm.

[0289] The insights and recommendation engine 230 can also find the food consumption patterns of each user and a broad population. The broad population can be a collection of 5, 10, 100, 10,000, 100,000 or more individual users. The insights and recommendation engine 230 can combine the detected food consumption pattern of the user with the detected food consumption pattern of the broad population to determine which population type the user may belong to.

[0290] In addition to glucose and insulin, the insights and recommendations engine 230 (referred to herein as the "engine") can be used to track and / or predict other factors that may affect food and / or be affected by food. The engine can be used to calculate antioxidant (e.g., thiol, vitamin C, etc.) levels. By using a device that can measure antioxidants at a location in the human body (e.g., the skin) either automatically or with manual intervention, the engine can generate one or more digital signatures of the user to track and predict the antioxidant response to food. The engine can be used to calculate blood pressure levels. By using a continuous blood pressure monitoring device, the engine can provide insights into the relationship between food and blood pressure, especially for users with hypertension or other cardiovascular diseases. The engine can be used to calculate digestive problems. By using implanted (semi-permanent or fully permanent) or ingestible (temporary) sensors, the engine can provide insights into the relationship between food and the condition of the digestive tract (e.g., pH, contraction intensity and / or frequency, etc.). This function can help patients with digestive problems (e.g., gastroesophageal reflux disease (GERD), irritable bowel syndrome (IBS), etc.) rule out foods that may be hindering their quality of life. The engine can be used to calculate the onset of migraines. By studying the correlation between food and migraines (e.g., from manual user input), the engine can help users with chronic migraines rule out foods that are predicted to be likely to cause migraines. The engine can be used to help users sleep. Sleep quality can be measured using wearable devices (e.g., Apple Watch, Fitbit, Samsung Gear, etc.). Sleep can be affected by food intake, but conversely, sleep also affects the user's hunger or metabolism. Therefore, the engine can find the correlation between the user's food intake and sleep quality and provide insights and recommendations accordingly. For example, the engine can use a GUI-based software interface to notify the user that "92% of the time, when you drink coffee after 4 pm, you will have poor sleep." Alternatively or additionally, the engine can analyze the relationship between food and other factors, which include but are not limited to drowsiness / fatigue, sleepiness, or cortisol levels. Any characteristic of the user's daily activities or physiology that can be measured by a wearable device or a medical device can be analyzed by the engine to continuously provide the user with a better understanding of their body and / or a healthier diet.

[0291] Figures 33A - 33C An exemplary window showing multiple features of a GUI-based software interface is shown. As Figure 33A shown, the window can display which food items (e.g., pear smoothie) the user has consumed, how many ingredients (e.g., mango, goji berry, green Anjou pear, etc.) are found or predicted to be present in the food item, a general or specific picture of the food item (e.g., imported directly from the website of the corresponding restaurant), and a continuous tracking of the user's blood glucose level and one or more insulin injections. As Figure 33BAs shown, the window can display the timeline of multiple events, including insulin injections and their dosages, daily activities (such as cycling), and food items or dishes (e.g., green vegetable salad with green vegetables, morning smoothie). As Figure 33C As shown, the window can display which foods the user frequently consumes (e.g., goji berries, mangoes) and the average blood glucose response to such food items.

[0292] Figures 34A - 34C Exemplary windows of a GUI-based software interface are shown that provide comprehensive reports via insights and recommendation engines as performed by 230. As Figure 34A As shown, the window can display a graph showing the user's blood glucose measurements throughout the day. The window can display additional details about blood glucose, including the user's weekly average blood glucose level and its standard deviation value, the percentage of glucose measurements that have been within a predetermined target range (target time), below the target range, and above the target range. The window can also display popular meal times, represented by the average number of meals consumed per hour throughout the day; and a distribution of the carbohydrates (e.g., in grams) consumed per hour throughout the day. The window can also display the average blood glucose value 2 hours after a meal (e.g., breakfast, lunch, and dinner). The window can also indicate whether the average blood glucose value 2 hours after a meal is above or below the predetermined target range of the blood glucose level. As Figure 34B As shown, the window can display more details about the analysis of meals. Meals can be grouped into breakfast, lunch, and dinner. For each group, the window can display the minute-to-minute (or other interval) change in blood glucose level during the 2-hour period after the meal, the average meal nutritional breakdown (e.g., protein, carbohydrates, fat, etc.), and an assessment of the balance of the average meal nutritional breakdown. Additionally, the analysis of meals can include the top three meals with significant changes in blood glucose level and the top three meals with the least changes in blood glucose level. As Figure 34C As shown, the window can display frequently consumed foods and their impact on blood glucose levels. In addition to foods, the window can also display the correlation between blood glucose levels and other factors. Other factors can include sleep quality, sleep duration, activity type, and the number of steps per day.

[0293] Any of the embodiments described herein (e.g., related to food analysis, food ontology, and personalized food / health / nutrition recommendations) are also applicable for use with the systems and methods for managing nutritional health described in U.S. Patent Application No. 13 / 784,845 (published as US 2014 / 0255882), which is hereby incorporated by reference in its entirety.

[0294] Calibration kit

[0295] The calibration kit can optimize platform 200 for a user's physiological responses to different foods. Optimizing platform 200 can include optimizing the functions of food analysis system 210, device / data hub 220, and insights and recommendations engine 230. Since users can respond differently to the same food and wearable and / or medical devices can have different compatibilities for different users, the calibration kit can be used to set a food baseline for all users. Generating a food baseline profile for a user can include monitoring the effects of different foods on the user's body while the user consumes one or more pre-packaged meals over a period of time. The one or more pre-packaged meals can contain known amounts of food. The monitored effects can be used to generate a food baseline profile. The calibration kit can be a modular kit. The calibration kit can include a monitoring system (e.g., glucose monitoring system, blood test, genetic test, etc.) and one or more standardized meals (also referred to as "calibration meals"). The calibration meals can include food bars, beverages, or both. Platform 200 can know and have tested all the characteristics of the calibration meals (e.g., ingredients, nutrients, processing, etc.). In some instances, the user can place the device on the body (or perform the provided monitoring tests) and consume one calibration meal / every morning. The user may be required to fast overnight (e.g., for 12 hours) the entire previous day. The device can measure the user's response to the calibration meal. The user can consume other foods throughout the day and track the foods to platform 200 via a GUI-based software interface (e.g., food tracking via text and speech recognition analysis, seamless food image recorder, etc.). After a short period of time (e.g., one week), platform 200 can use the data and predictions to set a baseline for the user. The baseline can be referred to as the user's unique personalized food "fingerprint".

[0296] Figure 35Disclosed is an exemplary calibration kit 3500. The calibration kit may include a first box 3502 that contains a CGM device; a second box 3504 that contains a DNA collection kit for DNA testing (e.g., a saliva collection kit); a third box 3506 that contains a biome collection kit for microbiome analysis (e.g., samples collected from the gut, genitalia, mouth, nose, and / or skin); and a fourth box 3508 that contains one or more calibration meals. The calibration kit may optionally include any of the above boxes, or different combinations of boxes. The calibration kit may include 1, 2, 3, 4, 5, or more calibration foods. The calibration kit may include 2, 3, 4, 5, 6, or more boxes. The calibration kit may include 1, 2, 3, 4, 5, or more monitoring systems. In some embodiments, the calibration kit may include one or more other components / devices (e.g., a blood test kit, a wearable device, or other biomarker tests / devices) that help generate the user's baseline health status. The calibration kit may also include a detailed list of instructions for easy user compliance. If needed, the calibration kit may include containers for each collection kit.

[0297] Example 1: Healthcare Provider

[0298] End users of the insights and recommendation engine 230 and other features of platform 200 may include healthcare providers. Healthcare providers may use a GUI-based software interface (e.g., a portal) to monitor and study the possible effects of different foods on a patient's body. Healthcare providers and patients may share or exchange information by connecting to platform 200 acting as a hub. In some examples, the portal may be used to monitor patients with type 2 diabetes and prediabetes. Figure 36 An exemplary window 3600 of a healthcare provider's portal is shown. On the portal, as shown in window 3600, a healthcare provider may invite new patients 3610, count the number of active patients 3620 who have accepted the invitation, and track the number of how participants record meals 3630 and / or activities 3640. A healthcare provider may select meal 3630 and / or activity 3640 images on window 3600 to connect to at least one additional window ( Figure 35 not shown in figure) and access more data and analysis.

[0299] New users (e.g., patient participants of a healthcare provider) may receive an invitation from the healthcare provider to install a GUI-based software interface (e.g., a mobile app) on a user device (e.g., a smartphone). Figure 37(Parts A - F) show exemplary windows 3710 - 3760 of a mobile application on a user device. After first launching the application (window 3710), the mobile application may ask the user to create an account (window 3720), enter basic information about the patient, including weight (window 3730), height (not shown), gender (window 3740), type of diabetes therapy if any (window 3750), and enable access to other features on the user device (window 3760). Other features may include the notification function of the user device or other mobile applications for health monitoring, motion detection, photography, etc. Enabling access to other features may automate multiple processes of platform 200 and reduce its dependence on user input.

[0300] The user may also receive a calibration kit to initiate optimization of platform 200 for the patient's physiological response. The user may use one or more devices and consume one or more calibration foods included in the calibration kit to perform an initial short program (e.g., 1 week) for baseline collection. The user may also consume and track other foods and / or beverages. Platform 200 may use the data and predictions generated from the initial short program to generate a baseline for the user. The baseline may reflect the user's physiological response to foods. Figure 38 (Parts A - C) show exemplary windows 3810, 3820, and 3830 of a mobile application on a user device for baseline data collection. The data recorded may include meals (e.g., breakfast, lunch, dinner, etc.; window 3810), daily activities (e.g., sleep, steps, etc.; not shown), and additional biomarkers (e.g., glucose level, insulin level, heart rate, etc.; window 3820). The data may be recorded either through user input or by using a tracker such as a wearable device. During baseline collection, the user may access a summary (insights) of the data collected. The summary may include the number of meals, steps walked, hours of sleep, etc.

[0301] Figure 39 (Parts A - D) show exemplary windows 3910 - 3940 of a mobile application on a user device that show a food image recording interface. The user may select a meal (e.g., breakfast, lunch, or dinner; window 3910). For accurate recording and baseline data collection, the mobile application may allow the user to record snacks. The user may be prompted to window 3920 to take a picture of the food item to be consumed (e.g., an apple). A picture of the food item may be recorded, and the mobile application may ask the user to enter a description of the food item and the meal time (window 3930). If the user records a food item for breakfast, the mobile application may check off breakfast from the list of meals to be recorded (window 3940).

[0302] After completion of baseline data collection, the user can access reports of the analysis and insights generated by the insights and recommendations engine 230. If the user links to a healthcare provider system or a health-related study, the healthcare provider or the coordinator of the health-related study can access part or all of the report. Figure 40A and 40B shows an exemplary window of a GUI-based software interface showing data about the user. In some instances, the report can focus on factors that may affect blood glucose levels: food, activity, and sleep. The report can inform the user of a predefined target glucose level range (e.g., 70 - 170 mg / dL) along with the user's average glucose level. The report can include an assessment of one or more meals based on how the user's glucose level responds to the one or more meals. The assessment can utilize a rating system (e.g., "A" for a balanced glucose response, "F" for a poor glucose response, etc.). The report can also display recommendations generated by the insights and recommendations engine 230. The recommendations can compare two food items consumed by the user and, based on the user's physiological response, suggest whether one of the two food items is a healthier option compared to the other of the two food items. In some instances, the recommendations can compare two types of bread (whole wheat bread vs. white bread) and recommend switching white bread to a whole wheat alternative. Another recommendation can compare two types of desserts (ice cream and fruit vs. dates stuffed with walnuts) and suggest that dates stuffed with walnuts may be a healthier option compared to ice cream and fruit. Different recommendations can compare two types of beverages (drink with honey vs. drink with artificial sweetener) and recommend switching the artificial sweetener to one tablespoon of honey. Additionally, the report can show the correlation between the number of steps taken by the user and the user's corresponding average blood glucose level. The report can inform the user that when the user increases the number of steps from less than 3,000 steps per day to more than 10,000 steps, the user's blood glucose level drops from 132 mg / dL to 125 mg / dL. Further, the report can show the correlation between sleep and blood glucose level. The report can inform the user that when the user increases the number of hours of sleep from less than 6 hours to more than 8 hours, the user's blood glucose level drops from 129 mg / dL to 115 mg / dL.

[0303] Example 2: Dietary Glucose Monitoring

[0304] Introduction

[0305] Each individual is unique. Each individual's body can process food (e.g., meals, beverages, etc.) in a different way from other individuals. Thus, one or more identical foods may have different effects on the biomarkers (e.g., glucose levels) of different individuals. Additionally, one or more identical foods may have different effects on each individual's biomarkers when consumed at different times (e.g., in the morning and in the afternoon).

[0306] A large number of databases and services are available to provide food advice, nutritional advice, and health advice. Examples of such databases and services include healthcare providers, food or nutrition manufacturers, restaurants, online and offline food recipes, and scientific articles. However, such available nutritional and / or dietary guidelines have not been customized for each individual and are limited in scope (e.g., type of food, biomarkers, health conditions, etc.). Whether for entertainment, beauty, medical, or other purposes, individuals inevitably have to rely on multiple sources of information every day to make food and nutrition-related decisions to improve or maintain health.

[0307] Accordingly, there is a need for systems and methods that can provide real-time feedback on the impact of food on each individual's health. There is a need for systems and methods that can: continuously collect a large amount of data from discrete sources (e.g., ingredients in a dish, nutritional information, glucose levels, blood pressure, temperature, etc.), analyze the data and reconstruct the data into a common format, evaluate and predict the correlation between the food consumed by an individual and the biomarkers, and provide personalized nutrition recommendations based on the individual's health and metabolic status at any given time.

[0308] Dietary Glucose Monitor

[0309] The ecosystem 100 includes a platform 200 that can be implemented as part of a Dietary Glucose Monitor (DGM) 101 according to some embodiments. The DGM 101 can provide personalized nutrition insights for each user based on the impact of different foods on at least the user's glucose levels. The DGM 101 can provide real-time feedback on the impact of one or more foods that a user has consumed or will consume on the user's blood glucose level. The DGM 101 can provide advice on adjusting the user's diet or lifestyle to maintain or improve the user's glucose level and thus maintain the user's health. The DGM 101 can be applicable to any user regardless of age, gender, race, health condition, etc. The DGM 101 may or may not require calibration by the user before its use.

[0310] The terms "food analysis system" and "food analysis module" (e.g., Figure 1 the food analysis system 210 shown in

[0311] The DGM 101 may include some or all of the components of the ecosystem 100, such as Figure 1 shown. In some embodiments, the DGM 101 may include a platform 200 that includes one or more of a food analysis module 210, a device / data aggregator 220, an insights and recommendations engine 230, and a device 110. The components of the platform 200 may communicate with each other. Thus, when describing the functions of the components of the platform 200, those skilled in the art will be able to understand that it can be interpreted as the function of the components alone or in combination with one or more other components of the platform 200. In one example, when describing the function of the food analysis module 201, those skilled in the art will be able to understand that it can be interpreted as the function of the food analysis module 201 alone or in combination with the device / data aggregator 220 and / or the insights and recommendations engine 230.

[0312] The DGM 101 may be applicable to many user applications. The DGM 101 may be used by users with type 1 diabetes or type 2 diabetes to evaluate when the user should inject insulin. The DGM 101 may be useful for athletes to optimize their athletic performance. The DGM 101 may be useful for individuals interested in tracking food intake as a way to monitor their weight loss diet.

[0313] Figure 59 An example of the DGM 101 is shown. The glucose level monitor 111 of the DGM 101 may be placed on or near the skin of a subject 102. The glucose level monitor 111 may communicate 121 (e.g., Bluetooth, NFC, WiFi, etc.) with the platform 200 of the DGM 101.

[0314] Figure 60 Additional details of the DGM 101 according to some embodiments are shown. The DGM 101 may include a food analysis module 210 that communicates with the glucose level monitor 111, where the food analysis module 210 is configured to (1) analyze data indicative of the food consumed by the user, and (2) determine the effect of an individual food on the user's glucose level based on changes in the user's blood glucose level as measured by the glucose level monitor. The DGM 101 may include a glucose level monitor 111.

[0315] The glucose level monitor 111 can function when placed anywhere in the body of the subject 102. The glucose level monitor 111 can communicate operatively and / or digitally with the platform 200 such that any data collected by the glucose level monitor can be transmitted in real time to one or more components of the platform (e.g., the device / data hub 220, the insights and recommendations engine 230, etc.). In some cases, the glucose level monitor 111 can be configured to transmit a set of data at predetermined time intervals (e.g., at least once every 5 minutes, 10 minutes, 20 minutes, 30 minutes, 60 minutes, 2 hours, 3 hours, 4 hours, 5 hours, 6 hours, 8 hours, 12 hours, 18 hours, 24 hours, etc.). In some cases, the glucose level monitor 111 can be configured to transmit a set of data once a set of digital-sized (e.g., byte) data reaches a predetermined threshold (e.g., 5 megabytes).

[0316] In some cases, the glucose level monitor 111 can be a "one size fits all" glucose level monitor that can be applicable to various users regardless of age, gender, height, weight, disease (e.g., type 1 vs. type 2 diabetes), etc. In some cases, the glucose level monitor 111 can be a personalized glucose level monitor for each user (e.g., a specific shape for a specific location of a specific user, a specific detection mechanism for a specific disease of the user, etc.).

[0317] Food can be primary food (e.g., fruits, vegetables, milk, eggs, meat, poultry, fish, nuts, etc.), recipes of food, packaged food, restaurant dishes, home-cooked dishes, etc. A single food can be an ingredient and / or nutrient of the primary food. In one example, the food is a dish of pad thai, and a single food can be an ingredient of pad thai, such as noodles, oil, garlic, eggs, soy sauce, lime juice, brown sugar, fish sauce, onion, basil leaves, peanuts, tofu, shrimp, etc.

[0318] The glucose level monitor 111 can measure the amount of glucose in a body fluid (e.g., interstitial fluid, blood, etc.) at a given moment. In some cases, the glucose level monitor 111 can measure the amount of glucose in a non-blood fluid as an indication of the blood glucose level. In some cases, the glucose level monitor 111 can directly measure the amount of glucose in the blood. The DGM 101 (e.g., the insights and recommendations engine 230 of the DGM 101) can use the data obtained from the glucose level monitor 111 and from the food analysis module 210 to quantify the personalized impact of food (or a single food) on an individual. The DGM 101 can detect the change in glucose level over a period of time (e.g., the slope of the glucose level curve over time) as an indicator of the change in glucose level.

[0319] The glucose level monitor can be configured to measure a range of glucose levels from about 0.1 millimoles per liter (mmol / L) to about 100 mmol / L. The glucose level measured by the glucose level monitor can be at least about 0.1 mmol / L, 0.2 mmol / L, 0.3 mmol / L, 0.4 mmol / L, 0.5 mmol / L, 1 mmol / L, 2 mmol / L, 3 mmol / L, 4 mmol / L, 5 mmol / L, 10 mmol / L, 20 mmol / L, 30 mmol / L, 40 mmol / L, 50 mmol / L, 100 mmol / L or higher. The glucose level measured by the glucose level monitor can be at most about 100 mmol / L, 50 mmol / L, 40 mmol / L, 30 mmol / L, 20 mmol / L, 10 mmol / L, 5 mmol / L, 4 mmol / L, 3 mmol / L, 2 mmol / L, 1 mmol / L, 0.5 mmol / L, 0.4 mmol / L, 0.3 mmol / L, 0.2 mmol / L, 0.1 mmol / L or higher.

[0320] The glucose level monitor can be configured to measure a range of glucose levels from about 1 milligram per deciliter (mg / dL) to about 500 mg / dL. The glucose level measured by the glucose level monitor can be at least about 1 mg / dL, 2 mg / dL, 3 mg / dL, 4 mg / dL, 5 mg / dL, 10 mg / dL, 20 mg / dL, 30 mg / dL, 40 mg / dL, 50 mg / dL, 100 mg / dL, 200 mg / dL, 300 mg / dL, 400 mg / dL, 500 mg / dL or higher. The glucose level measured by the glucose level monitor can be at most about 500 mg / dL, 400 mg / dL, 300 mg / dL, 200 mg / dL, 100 mg / dL, 50 mg / dL, 40 mg / dL, 30 mg / dL, 20 mg / dL, 10 mg / dL, 5 mg / dL, 4 mg / dL, 3 mg / dL, 2 mg / dL, 1 mg / dL or lower.

[0321] There may be multiple ways to quantify the blood glucose response of a single meal into a single meal: (1) incremental area under the curve (iAUC); (2) percentage within range (% within range); (3) peak; (4) total change. First, the iAUC can be calculated as the area between the glucose curve at the start of the meal and the glucose above the baseline. The unit of the iAUC can be mg·hr / dL (milligram-hour per deciliter). The iAUC can allow a continuous range of values. The iAUC can sample the entire glucose level curve over time. Second, the % within range can be the percentage of the glucose level during the time after the meal that is within a predefined range. In some cases, the predefined range can be provided or suggested by a physician for a user or a group of users. Third, the peak can be the difference between the "peak" and the baseline glucose level. In some cases, the baseline glucose level can be the average of the user's glucose levels during a period when the user has not consumed any food or any significant amount of food. Fourth, the total change can be defined as the sum of the absolute differences between all consecutive values.

[0322] The platform 200 of the DGM 101 can be implemented using one or more GUIs ( Figure 3 not shown in the figure) to enable the user to select and adopt the features of the three components 210, 220, and 230. The GUI can be presented on a display screen of the user device. Thus, the DGM 101 (e.g., the food analysis module 210) can communicate with the user device that displays the GUI. Examples of user devices can include one or more of the devices 110, such as the mobile device 114 or the wearable device 112. One or more components of the DGM 101 can communicate with each other (e.g., share data). In some cases, the insights and recommendations engine 230 can (1) use the information about the food consumed by the user from the food analysis module 210 and the information about the user's biosignals from the device / dataset aggregator 220, and (2) analyze any correlations (e.g., causal relationships) between the food consumed by the user and the biosignals.

[0323] The food analysis module 210 can be configured to analyze data indicating the food consumed by the user and other data indicating the food not consumed by the user. Such data indicating the food can be structured or unstructured when the food analysis module 210 initially receives (e.g., collects) the food. The food analysis module 210 can be capable of mapping such unstructured (e.g., different formats) data into structured (e.g., the same format) data. The food analysis module 210 can represent such structured data related to all foods as a map of foods (i.e., a network of foods, a food ontology).

[0324] Data indicating the food consumed by the user may include one or more images of the food consumed by the user, one or more videos of the food, one or more voice logs of the food, one or more text logs of the food, and one or more barcodes on each package of the food. Thus, the food analysis module 210 may be configured to predict (or determine) one or more ingredients within the food using information from the food ontology to construct and / or analyze data (e.g., one or more images, one or more videos, one or more voice logs, one or more text logs, one or more barcodes, etc.). The food analysis module 210 may also be further configured to use information from the food ontology to determine one or more nutrients (e.g., calories, total fat, sodium, etc.) and the amount thereof estimated from one or more predicted (or determined) ingredients.

[0325] The food analysis module 210 may communicate with the mobile device 114 or the wearable device 112. The mobile device 114 or the wearable device 112 may be configured to partially collect data indicating the food consumed by the user (e.g., one or more images, one or more videos, one or more voice logs, one or more text logs, one or more barcodes, etc.). The food analysis module 210 may be configured to abstract and / or analyze information from one or more captured images of food or food packaging, one or more foods, one or more voice logs, one or more text logs, or one or more barcodes related to the food, for example, by using the food ontology. The food analysis module 210 may also be further configured to update the food ontology based on such data obtained from one or more user devices of the user and other users of the DGM 101 (or ecosystem 100). The food analysis module 210 may further communicate with a smart home device 118 (e.g., a smart refrigerator). In one example, the food analysis module 210 may communicate with a smart refrigerator. The smart refrigerator may be configured to track the movement of food in and out of the refrigerator to track or predict the user's food consumption. The smart refrigerator may track the movement of food by using sensors (e.g., cameras) that capture images or videos of the food, or read barcodes of prepackaged food or meals.

[0326] One or more GUIs of DGM 101 may be configured to visually annotate data of the food consumed by the user with the name of each food item present in the prediction data and the ingredients within each food item (e.g., one or more images, one or more videos, one or more voice logs, one or more text logs, one or more barcodes, etc.). In some cases, one or more GUIs of DGM 101 (i.e., one or more graphic modules of DGM 101) may be configured to visually annotate one or more images of the food consumed by the user with the name of each food item and / or the ingredients within each food item.

[0327] The glucose level monitor 111 may include a blood glucose monitor, a continuous glucose monitor (CGM), a flash glucose monitor (FGM), or a glucose-sensing bioprosthesis. The glucose level monitor 111 may be configured to measure the user's glucose level in an analyte (e.g., body fluid), the analyte including blood, interstitial fluid, sweat, tears, and / or saliva. The glucose level monitor 111 may be placed on or near the user's skin, or within the user's body (e.g., under the skin). In some cases, a portion of the glucose level monitor 111 may be in fluid communication with the user's analyte to measure the glucose level in the analyte. In some cases, the glucose level monitor 111 may be a non-invasive device (e.g., a non-invasive medical device) capable of measuring the user's glucose level without piercing the user's skin and without withdrawing blood from the user's skin. In one example, the glucose level monitor 111 may direct electromagnetic radiation (e.g., radio waves, microwaves, infrared light, visible light, ultraviolet (UV) light) through the user's skin and towards the user's body fluid. Thereafter, the glucose level monitor 111 may read at least a portion of the directed electromagnetic radiation transmitted or reflected by the user's body fluid, the transmitted or reflected readings indicating the glucose level of the body fluid. The electromagnetic radiation may include one or more wavelengths (e.g., a laser having one wavelength). Similarly, the glucose level monitor 111 may use an electrical signal to non-invasively detect the user's glucose level.

[0328] The device / data hub 220 can communicate with the CGM to continuously stream and store the user's personalized data regarding glucose levels. The stored personalized data of the user can be used for analysis by the insights and recommendations engine 230. The measurement results of the CGM of interstitial fluid can have an error of up to about 15%, 14%, 13%, 12%, 11%, 10%, 9%, 8%, 7%, 6%, 5% or less compared to the corresponding blood glucose level. The measurement results of the glucose level in interstitial fluid can have a relatively shorter delay compared to the blood glucose level. The CGM can remain operative during the user's daily activities, which include showering, exercising, sleeping, etc. The glucose level monitor 111 can include any suitable continuous glucose monitoring device.

[0329] The device / data hub 220 can communicate with the FGM. The FGM can be a wearable or a small sensor on the user's skin. A part of the FGM can be seated under the skin to be in fluid communication with interstitial fluid. The FGM can determine the glucose level in interstitial fluid as closely related to the blood glucose level. The FGM can continuously or intermittently record the user's glucose level (e.g., once every 5 minutes, 10 minutes, 15 minutes, 20 minutes, 30 minutes, 60 minutes, more frequently, less frequently, etc.), and the user can access the data including the collected glucose levels of the user by scanning the FGM with a separate scanner (e.g., a specific monitor or a user device such as a mobile device).

[0330] The user can wear the glucose monitor (e.g., CGM or FGM) for about 10 days to 30 days. The user can wear the glucose monitor for at least about 5 days, 10 days, 15 days, 20 days, 25 days, 30 days, 40 days, 50 days or more. The user can wear the glucose monitor for at most about 50 days, 40 days, 30 days, 25 days, 20 days, 15 days, 10 days, 5 days or fewer days. After use, the glucose monitor can be removed and replaced with a new one. The new glucose monitor may or may not need to be calibrated for the user.

[0331] The glucose level monitor 111 can communicate with the user device to transmit data indicating the user's glucose level. The communication can include electromagnetic radiation, sound, Institute of Electrical and Electronics Engineers (IEEE) 802.11 wireless local area network (WLAN) standard, Bluetooth, Zigbee, Z-Wave, near field communication (NFC), magnetic secure transmission (MST), functional modifications thereof, or a combination thereof. In some cases, the glucose level monitor 111 can be hardwired to the user device.

[0332] The glucose level monitor 111 may measure the user's glucose level at a frequency of at least about once every 0.5 minutes, 1 minute, 2 minutes, 3 minutes, 4 minutes, 5 minutes, 10 minutes, 15 minutes, 20 minutes, 30 minutes, 60 minutes, 90 minutes, 120 minutes, or longer. The glucose level monitor 111 may measure the user's glucose level at a frequency of at most about once every 120 minutes, 90 minutes, 60 minutes, 30 minutes, 20 minutes, 15 minutes, 10 minutes, 5 minutes, 4 minutes, 3 minutes, 2 minutes, 1 minute, 0.5 minutes, or shorter. The glucose level monitor 111 may measure one or more of the user's glucose levels in an irregular pattern. In some cases, the glucose level monitor 111 may be adaptive. In some cases, the glucose level monitor 111 may be configured to monitor the user's glucose level at a first rate, and when the glucose level monitor 111 detects a sharp change (e.g., an increase or decrease) in the glucose level (e.g., by comparing it to the user's glucose level history, a predetermined reference glucose level, or a predetermined reference glucose level range, etc.), the glucose level monitor 111 may monitor the user's glucose level at a second rate higher than the first rate. This transformation of the rate of glucose level detection may be automatic (e.g., without manual intervention or indication at the moment), or user-assisted (e.g., the glucose level monitor 111 may prompt the user via a user device to approve the change in the rate of glucose detection).

[0333] The glucose level monitor 111 may further be capable of measuring additional biomarker levels of the user. The glucose level monitor 111 may be capable of measuring at least about 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, or more additional biomarker levels of the user. The glucose level monitor 111 may be capable of measuring at most about 10, 9, 8, 7, 6, 5, 4, 3, 2, or 1 additional biomarker level of the user. In this case, the food analysis module may be further configured to (1) analyze data indicative of the food consumed by the user, and (2) determine the effect of an individual food on the additional biomarker levels of the user based on changes in the additional biomarker levels of the user as measured by the glucose level monitor.

[0334] Additional biomarkers may include genes, peptides, proteins, lipids, metabolites, etc. Examples of additional biomarkers include insulin, lactate, ketones, antioxidants, neurotransmitters, amines, vitamins, cholesterol, carbohydrates, alcohols, amino acids, nucleic acids, etc. Additional biomarkers may further include additional biological characteristics of the user, such as body temperature, heart rate, sweat, or body movement.

[0335] Alternatively or additionally, the food analysis module can further communicate 126 with a medical device 116 (such as Figure 60 shown), which is configured to measure additional biomarkers within the user's body. The medical device 116 may be capable of measuring at least about 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 or more additional biomarker levels of the user. The medical device 116 may be capable of measuring at most about 10, 9, 8, 7, 6, 5, 4, 3, 2 or 1 additional biomarker levels of the user. In this case, the food analysis module can be further configured to (1) analyze data indicative of the food consumed by the user, and (2) determine the effect of an individual food on the additional biomarker levels of the user based on changes in the additional biomarker levels of the user as measured by the medical device. The medical device 116 can include a heart rate monitor, a blood pressure monitor, a sweat sensor, a galvanic skin response (GSR) sensor, an electrocardiogram (ECG) monitor, a bioelectrical impedance analysis (BIA) monitor (e.g., a body fat monitor), a functional modification thereof, or a combination thereof. The medical device 116 can include a mood monitor, e.g., a pressure monitor placed on or near the user's skin and configured to measure skin temperature, skin conductance, and / or arterial pulse wave as an indication of the user's stress level.

[0336] The medical device 116 can communicate with the user device to transmit data indicative of the user's additional biomarkers. The communication can include electromagnetic radiation, sound, Institute of Electrical and Electronics Engineers (IEEE) 802.11 wireless local area network (WLAN) standard, Bluetooth, Zigbee, Z-Wave, near field communication (NFC), magnetic secure transmission (MST), a functional modification thereof, or a combination thereof. In some cases, the medical device 116 can be hardwired to the user device.

[0337] Medical device 116 may measure the user's additional biomarker levels at a frequency of at least about once every 0.5 minutes, 1 minute, 2 minutes, 3 minutes, 4 minutes, 5 minutes, 10 minutes, 15 minutes, 20 minutes, 30 minutes, 60 minutes, 90 minutes, 120 minutes, or longer. Medical device 116 may measure the user's additional biomarker levels at a frequency of at most about once every 120 minutes, 90 minutes, 60 minutes, 30 minutes, 20 minutes, 15 minutes, 10 minutes, 5 minutes, 4 minutes, 3 minutes, 2 minutes, 1 minute, 0.5 minutes, or shorter. Medical device 116 may measure the user's additional biomarker levels in an irregular pattern. In some cases, medical device 116 may be adaptive. In some cases, medical device 116 may be configured to monitor the user's additional biomarker levels at a first rate, and in the presence of a sharp change (e.g., an increase or decrease) in the glucose level (e.g., by comparing it to the user's glucose level history, a predetermined reference glucose level, or a predetermined reference glucose level range, etc.) or an additional biomarker level (in a manner similar to the way of the glucose level), medical device 116 may begin to monitor the user's additional biomarker levels at a second rate higher than the first rate. This transformation of the rate of additional biomarker level detection may be automatic (e.g., without manual intervention or indication instantaneously), or user-assisted (e.g., medical device 116 may prompt the user via the user device to approve the change in the rate of glucose detection).

[0338] In some cases, the wearable device 112 (as shown in Figure 1 ) may be used in addition to or in place of medical device 116 to monitor the user's additional biomarker levels. Examples of wearable device 112 include but are not limited to Apple Watch, Fitbit, Samsung Gear, Android Wear devices, etc.

[0339] The food analysis module 210 may be further configured to determine the correlation between the user's glucose level and the user's additional biomarker levels. In one example, the food analysis module may determine how a change in the glucose level from the consumption of food affects the user's lactate level (e.g., in the blood) or the user's body temperature.

[0340] Due to the relationship between glucose and insulin, a dietary glucose monitor can also indirectly measure insulin by using a glucose-insulin model to calculate insulin levels. The glucose-insulin model can be a biomathematical model (such as the GAIA model described above), a machine learning model, or any other suitable model. Since insulin is regarded as the main regulator of fat storage in the body, this allows the use of a dietary glucose monitor as an aid in weight loss. Although glucose is a relatively good surrogate for insulin, the relationship between them may not be trivial (see, for example, Holt SHA, Brand Miller JC, Petocz P., "The insulin index of foods: the insulin demand generated by 1000-kJ portions of common foods.", Am. J. Clin. Nutr. 66, 1264-1276 (1997)), and a suitable model may be required to derive the estimated insulin levels. Once the insulin levels are estimated, the dietary glucose monitor can thus provide insights and recommendations on which foods, combinations of foods, or lifestyle events affect an individual's weight, thereby providing a personalized weight loss plan based on biomarker measurements.

[0341] The food analysis module 210 (and / or the insights and recommendations engine 230 in communication with the food analysis module 210) can be further configured to provide one or more recommendations to the user based on the impact of different foods on the user's glucose levels to manage the user's glucose levels. The food analysis module 210 can be further configured to provide one or more recommendations to the user based on the impact of different foods on the user's other biomarker levels to manage the user's other biomarker levels. One or more recommendations can include changes to the user's diet. For example, the user may have diabetes, and the user's physician may have recommended a specialized diet.

[0342] One or more recommendations may include at least one food to consume or avoid, a combination of two or more foods to consume or avoid, or a time to consume or avoid at least one food. In some cases, the food analysis module 210 may detect a decrease or maintenance of the user's glucose level when two specific foods are consumed together (e.g., simultaneously as two items or as ingredients in the same dish) or one after the other, and recommend the user consume such a food combination. In some cases, the food analysis module 210 may detect a sudden increase in the user's glucose level when two specific foods are consumed together (e.g., simultaneously as two items or as ingredients in the same dish) or one after the other, and recommend the user avoid consuming such a food combination. In some cases, the food analysis module 210 may detect that when a food (e.g., a donut, ice cream, pasta, etc.) is consumed late at night (e.g., after 9 p.m.), the user's glucose level increases more sharply compared to when the same food is consumed earlier in the day (e.g., before 6 p.m.). In this case, the food analysis module 210 may recommend the user avoid consuming such a food after a certain time (e.g., 6 p.m.).

[0343] Refer to Figure 60, the food analysis module 210 can be further configured to direct the delivery of one or more foods to the user. The food analysis module 210 can communicate with one or more food delivery or meal planning services 250, such as HelloFresh, Green Chef, EveryPlate, Sun Basket, Home Chef, BlueApron, BistroMD, Freshly, NutriSystem, Plated, Purple Carrot, GreenBlender, PeachDish, Chef'd, Terra's Kitchen, FreshDirect, Daily Harvest, Instacart, Uber Eats, etc. The food analysis module 210 can recommend meals provided by one or more food delivery or meal planning services 250 that are consistent with one or more of its recommendations, the recommendations including at least one food to consume or avoid, a combination of two or more foods to consume or avoid, or at least a time to consume or avoid at least one food. The food analysis module 210 can automatically direct one or more food delivery or meal planning services 250 to deliver this meal (and / or supplements) to the user with or without user confirmation. The user can provide prior consent such that the food analysis module 210 can automatically direct the ordering and delivery of the recommended food or meal. One or more food delivery or meal planning services 250 can provide personalized and curated grocery lists (e.g., by automatically generating a grocery list on Instacart), recipes, and suggestions. The recipes provided can be existing recipes or new recipes generated for the user by the systems described herein. The food analysis module 210 can direct the delivery of about 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 or more foods (e.g., one). The food analysis module 210 can direct the delivery of 1, 2, 3 or more meals per day. The food analysis module 210 can direct the delivery of foods at least about 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 or more times per week. The food analysis module 210 can direct the delivery of foods up to about 10, 9, 8, 7, 6, 5, 4, 3, 2 or 1 time per week. The food analysis module 210 can update or change the preferences for the foods being ordered and delivered based on the user's preferences, ratings of the foods, or responses in terms of glucose levels or other biomarker levels.In addition, the food analysis module 210 can communicate with delivery services such as the United States Postal Service (USPS), FedEx, United Parcel Service (UPS), DHL, Google Express, Amazon, etc., to track and update the user regarding the delivery of one or more foods to the user.

[0344] The food analysis module 210 can be further configured to recommend recipes for meals or diet plans to the user, where the recipes reflect one or more recommendations provided to the user. The food analysis module 210 can communicate with one or more sources (such as websites, online or offline publications, etc.) to recommend such recipes. Examples of one or more sources can include but are not limited to cookpad, allrecipes, chefkoch, dianping, marmiton, foodnetwork, russianfood, geniuskitchen, bbcgoodfood, thekitchn, cuisineaz, epicurious, seriouseats, 1000.menu, yummly, foodandwine, bonappetit, taste, allrecipes, eater, tasty, etc.

[0345] One or more recommendations provided by the food analysis module 210 can further include changes to the user's lifestyle. In some cases, based on the user's history or the compiled history of two or more users, the food analysis module 210 can suggest changes to the user's lifestyle that can directly or indirectly provide changes or maintenance of the user's glucose level or other biomarker levels. The user's lifestyle can include exercise, walking, standing, sitting, stress management, meditation, medication, dietary supplements, sexual activity, or sleep. The food analysis module 210 can communicate with one or more user devices (such as the glucose level monitor 111, wearable device 112, mobile device 114, or medical device 116, etc.) to determine the relationship between one or more of the following groups: the user's lifestyle, the consumption of one or more foods or one or more individual foods, or the user's glucose level or other biomarker levels. The food analysis module 210 can suggest changes to the user's lifestyle that have been shown or predicted to (e.g., by a healthcare professional or through the use of machine learning algorithms) help maintain or change the user's glucose level or other biomarker levels.

[0346] One or more GUIs of DGM 101 can be configured to visually annotate data of foods having ingredients within each food item, ratings or rankings of each food item, fan insights of each food item, and / or dietary recommendations (e.g., one or more images, one or more videos, or one or more screenshots thereof, transcriptions of one or more voice logs, one or more text logs, etc.). The rating or ranking can indicate the impact of the food on the user's glucose level and / or other biomarker levels (e.g., positive or negative).

[0347] The rating can be a numerical rating, letter rating, alphanumeric rating, percentage, graphical system, etc. In some cases, the rating can be provided on a certain scale, where a higher rating can have a more positive impact on the user's glucose level compared to a lower rating. In some cases, the rating can be on a scale from 0 percent (%) to 100%, with a higher percentage value indicating a more positive impact on the user's glucose or other biomarker levels. The rating can be at least about 0%, 1%, 2%, 3%, 4%, 5%, 6%, 7%, 8%, 9%, 10%, 20%, 30%, 40%, 50%, 60%, 70%, 80%, 90%, 99% or higher. The rating can be at most about 100%, 90%, 80%, 70%, 60%, 50%, 40%, 30%, 20%, 10%, 9%, 8%, 7%, 6%, 5%, 4%, 3%, 2%, 1% or less. In some cases, the rating can be based on another numerical scale, such as a scale from 0 to 10 (e.g., 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, and 10 in order of positive impact). In some cases, the rating can be based on a graphical scale, such as a scale from 1 star to 5 stars (e.g., 1 star, 2 stars, 3 stars, 4 stars, and 5 stars in order of positive impact). In some cases, the rating can be a letter rating system, such as one or more of D-, D, D+, C-, C, C+, B-, B, B+, A-, A, and A+.

[0348] Each rating in the rating can indicate the measured or predicted impact of the corresponding food item or ingredient on the user's glucose level or other biomarker level. In some cases, the DGM 101 can determine when the user eats or has consumed one or more foods (e.g., through user input or based on the user's geographical location). The DGM 101 can abstract information about one or more foods (e.g., ingredients, nutrition, etc.) based on the data of one or more foods. Thereafter, the DGM 101 can provide a rating for one or more foods based on its past impact on the user's or user population's glucose level or other biomarker level. Alternatively or additionally, the DGM 101 can use real-time data of the user's glucose level or other biomarker level when consuming one or more foods or around that time, and then provide a rating for one or more foods based on its impact on the glucose level or other biomarker level.

[0349] A positive impact on the user's glucose level can include an increase or decrease in the user's glucose level. For some users, the positive impact of a food can be an increase in the glucose level or other biomarker level. For some users, the positive impact of a food can be a decrease in the glucose level or other biomarker level. For some users, the negative impact of a food can be an increase in the glucose level or other biomarker level. For some users, the negative impact of a food can be a decrease in the glucose level or other biomarker level. For some users, the positive impact of a food can be to maintain the glucose level or other biomarker level within a predefined range of the user. For some users, the negative impact of a food can be to maintain the glucose level or other biomarker level within a predefined range of the user. This predefined range can be user-defined (e.g., the user's personal goal) or recommended by a medical professional or nutritionist.

[0350] The food analysis module 210 may be configured to determine the impact of a single food on a user's glucose level in real time, periodically, or at one or more predetermined time points. The food analysis module 210 may determine the impact of a single food on the user's glucose level or another biomarker level in real time (e.g., simultaneously) and / or over a period of time around the time of consuming the single food (or a food including the single food). The food analysis module 210 may determine the impact of a single food on the user's glucose level or another biomarker level by analyzing previously collected data every 1 hour, 2 hours, 3 hours, 4 hours, 6 hours, 8 hours, 12 hours, 24 hours, 2 days, 3 days, 4 days, 5 days, 6 days, 1 week, 2 weeks, 3 weeks, 4 weeks, 1 month, etc. The food analysis module 210 may determine the impact of a single food on the user's glucose level or another biomarker level at one or more predetermined time points, such as, for example: 6:00 am (e.g., such that the food analysis module 210 may provide a recommendation before the user consumes breakfast), 11:00 am (e.g., such that the food analysis module 210 may provide a recommendation before the user consumes lunch), 5:00 pm (e.g., such that the food analysis module 210 may provide a recommendation before the user consumes dinner), 10:00 pm (e.g., such that the food analysis module 210 may provide a summary of the correlation between the consumed food and the glucose or other biomarker level before the user goes to sleep, etc.).

[0351] Figure 61A and 61B shows examples of visual annotations via one or more GUIs of the DGM 101 according to some embodiments. Referring Figure 61A , a user may access one or more GUIs of the DGM 101 via a user device (e.g., the mobile device 114). The user may use one or more GUIs of the DGM 101 to capture an image 6105 of a dish to be consumed (e.g., a salad). In some cases, the user may use one or more other GUIs (e.g., a camera application) of the mobile device 114 to capture the image 6105 of the dish to be consumed. In some cases, the user may obtain the image 6105 of the dish from other sources, such as, for example, a restaurant's website or one or more SNS networks (e.g., Yelp). In some cases, a video rather than an image 6105 of the dish to be consumed may be captured, and the DGM 101 may be configured to generate at least the image 6105 of the dish based on this video. One or more GUIs of the DGM 101 may scan 6110 the image 6105 to identify one or more food items from the image 6105 (e.g., individual foods or ingredients in the dish). The names of one or more food items of the image 6105 may be visually annotated by the DGM 101 on one or more GUIs of the DGM 101. Referring Figure 61B, the DGM 101 can analyze one or more food items 6115 and provide one or more ratings 6120 (or rankings) for the one or more food items 6115. The one or more ratings 6120 can indicate the effect of the one or more food items 6115 on the user's glucose level and / or additional biomarker levels. In one example, the one or more ratings 6120 for the one or more food items 6115 can be a letter system, where the highest rating A+ indicates the least increase in the user's glucose level and lower ratings (e.g., B, C, D, etc.) indicate a higher increase in the user's glucose level than rating A+. Additionally, the DGM 101 can predict the food the user is consuming based on one or more food items predicted (or identified) from the image 6105 and then display one or more predictions 6125 (e.g., Quinoa salad, French shrimp salad, Caesar salad, etc.) for the dish on one or more GUIs. In some cases, the food prediction 6125 feature can allow the user to provide the name of the dish.

[0352] Figure 62A and 62B shows an exemplary window of an analysis report showing the user's food data of a GUI-based software interface according to some embodiments. Referring to Figure 62A, the analysis report can focus on factors that may affect blood glucose levels: food, activity, and sleep. The report can be based on a predefined target glucose level range for the user (e.g., 70 mg / dL - 170 mg / dL). The report can include an assessment of one or more meals based on how the user's glucose level responds to the one or more meals. The assessment can utilize a rating system (e.g., "A" for a balanced glucose response, "F" for a poor glucose response, etc.). The report can also display recommendations generated by the insights and recommendations engine 230. The recommendations can compare two food items (consumed by the user) and, based on the user's physiological response, recommend whether one of the two food items is a healthier option compared to the other of the two food items. In some instances, the recommendations can compare two types of fruits (nectarines vs. grapes) and also recommend pairing either fruit with protein or fat. Another recommendation can compare two types of snacks (edamame vs. potato chips) and suggest that a small energy bar may be a healthier snack option. Different recommendations can compare two types of beverages (juice vs. cola) and recommend avoiding sugary drinks. Additionally, the report can show the correlation between the number of steps taken by the user and the user's corresponding average blood glucose level. The report can inform the user that when the user increases the number of steps from less than 3,000 steps per day to more than 10,000 steps, the user's blood glucose level drops from 132 mg / dL to 125 mg / dL. Furthermore, the report can show the correlation between sleep and blood glucose level. The report can inform the user that when the user increases the number of hours of sleep from less than 6 hours to more than 8 hours, the user's blood glucose level drops from 129 mg / dL to 115 mg / dL. Refer to Figure 62B , another analysis report can include a report on the user's average peak glucose level over a set period of time (e.g., 1 day, 2 days, 3 days, 4 days, 5 days, 6 days, 7 days, 8 days, 9 days, 10 days, or longer). In one instance, the analysis report can inform the user that, based on 1 week of data collection, on average in a day, the user's glucose level peaks at 172 mg / dL at 7:35 am. Additionally, the analysis report can inform the user of one or more dishes (e.g., dishes with an "A" rating) that the user has consumed with a balanced glucose response over a given period of time (e.g., 1 week). The analysis report can also inform the user of one or more dishes (e.g., dishes with an "F" rating) that the user has consumed with a poor glucose response over a given period of time (e.g., 1 week).

[0353] Refer to Figure 60, the DGM 101 can be a tool for connecting two or more users 260. The DGM 101 can be a tool for peer support in group therapy, support groups, diet and / or health management. Such group therapy, support groups or peer support can help promote or maintain behavior changes to achieve better glucose level or other biomarker level management. The food analysis module 210 can be further configured to initiate a group consisting of two or more users based on a comparison. Even if two or more users are not designated as a predefined group, the food analysis module 210 can still initiate groups of two or more users based on different factors, such as gender, age range, race, geographical location, working hours, occupation, condition or disease, etc. Additionally, the food analysis module 210 can be further configured to generate one or more subgroups within a group.

[0354] The DGM 101 can be further configured to allow a user to share any information collected and / or generated by the DGM 101 for the user with other people who may or may not be users of the DGM 101. The DGM 101 can be configured to allow a user to share information with a CDE, dietitian, nutritionist, physician, nurse or sports coach. The DGM 101 can be configured to allow a user to share information with friends or family members. Examples of information collected and / or generated by the DGM 101 for a user can include an analysis of the following: the food consumed by the user, the user's glucose or other biomarker levels, or the correlation and / or prediction between the food consumed and the user's glucose or other biomarker levels.

[0355] The food analysis module 210 can be further configured to compare the impact of a single food on glucose levels between two or more users. Users of the DGM 101 can use the same platform 200, and thus, the platform 200 including the food analysis module 210 can correlate and compare data related to two or more users. The food analysis module 210 can correlate and compare the data (such as glucose levels or other biomarker levels) of two or more users in the following aspects: the same gender, age range (such as 10 - 20 years old, 20 - 30 years old, 30 - 40 years old, 40 - 50 years old, 50 - 60 years old, 60 - 70 years old, 70 - 80 years old, etc.), race, geographical location (such as by continent, country, city, town, school, etc.), working hours (such as day shift, night shift, etc.), occupation (such as athlete, doctor, student, restaurant staff, stay-at-home parent, office clerk, mail delivery person, etc.), condition or disease (such as healthy, type 1 or type 2 diabetes, high blood pressure or low blood pressure, partial or complete paralysis, history of stroke, viral infection, cancer, sleep disorder, mood disorder, etc.).

[0356] The food analysis module 210 can correlate and compare data (e.g., glucose levels or other biomarker levels) of two or more users of a predefined group. A healthcare provider can access (with restricted access) the DGM 101 synchronized with two or more users. The healthcare provider can assign two or more users to a group, e.g., a group for studying a specific food or condition. For example, the healthcare provider can generate a group of patients with type 1 diabetes within the DGM 101 and use the DGM 101 to track their glucose levels (e.g., via the glucose level monitors 111 of each user in the group) and / or provide recommendations to the users in the group. Additionally, users in the group can access (with restricted access rights) each other's information, and this can, in some cases, promote competition and motivate users to follow the recommended diet.

[0357] Whether in a predefined group or a group generated by the food analysis module 210, the food analysis module 210 is further configured to allow a group of two or more users to communicate with each other via a GUI on the user device associated with each user (e.g., via the GUI of the DGM 101). Two or more users can communicate with each other via the GUI on each other's mobile devices 114 or wearable devices 112. Communication between groups of two or more users can include information shared via a graphical user interface, the information including text, images, videos, and / or voice recordings.

[0358] Referring Figure 60 , the DGM 101 can be further configured to connect a user to a healthcare or fitness expert 270. The DGM 101 (e.g., the food analysis module 210 and / or the insights and recommendations engine 230) can be configured to identify a healthcare or fitness expert 270 for a user based on the impact of a single food on the user's glucose level or other biomarker. The healthcare or fitness expert 270 can include a certified diabetes educator (CDE), dietitian, nutritionist, physician, nurse, or sports coach. The healthcare or fitness expert 270 can provide telemedicine (e.g., diagnosing or treating a health condition or disease via telecommunications), nutritional advice (e.g., regarding glucose management and its relationship to diet and other lifestyles), and guidance (e.g., exercise routines).

[0359] Figure 63An exemplary window showing a video between a user and another person in a GUI-based software interface is presented. DGM 101 can provide a GUI-based software interface for the user to talk to one or more other users of DGM 101. One or more other users may have a diet plan and / or a health condition (such as type 2 diabetes) similar to that of the user. One or more other users may have carried out the diet plan that the user is currently on, and one or more other users may provide support and encouragement to the user. DGM 101 can provide a GUI-based software interface for the user to talk to a healthcare or fitness expert 270. In some cases, on-site feedback from the healthcare or fitness expert 270 may be more effective in terms of the user's compliance with the diet and diet goals.

[0360] Referring Figure 60 , DGM 101 can be further configured to remind the user about one or more tests 290 related to the user's health. One or more tests 290 can be related to the user's diet, glucose level, additional biomarker levels, diseases, and / or other conditions. In some cases, one or more tests 290 can be biannual or quarterly tests of the A1C test. The A1C test can be a blood test that can provide information about the average level of the user's blood glucose over a period of time (e.g., the past 3 months). The A1C test can be used to diagnose type 2 diabetes and / or prediabetes. The A1C test can also be used for diabetes management. The A1C test can also be referred to as hemoglobin A1C, HbA1c, glycated hemoglobin, or glycosylated hemoglobin test. The A1C test results can be reported as a percentage (e.g., the higher the percentage, the higher the user's blood glucose level). In one example, a healthy A1C level can be less than 10% (e.g., 5.7%). In some cases, DGM 101 can schedule one or more tests 290. DGM 101 can be operably linked to a database for one or more tests 290 to receive or retrieve the results of the user's one or more tests 290 for further analysis.

[0361] Referring Figure 60, the DGM 101 can be further configured to connect the user to the virtual health assistant 280. The virtual health assistant 280 can communicate with one or more components of the DGM 101 (e.g., the food analysis module 210 and / or the insights and recommendations engine 230). The virtual health assistant 280 can be configured to automatically generate one or more recommendations and provide them to the user. The virtual health assistant 280 can be configured to communicate with the user via the user device (e.g., the mobile device 114). The virtual health assistant 280 and the user can communicate via one or more GUIs of the DGM 101 on the user device. The virtual health assistant 280 can use artificial intelligence including one or more machine learning algorithms to (1) understand questions or comments from the user; (2) identify one or more recommendations based on the impact of a single food on the user's glucose level; and (3) generate one or more recommendations and provide them to the user. The virtual health assistant 280 can use artificial intelligence including one or more machine learning algorithms to (1) understand questions or comments from the user; (2) identify one or more recommendations based on the impact of a single food on the user's glucose level and / or additional biomarker levels; and (3) generate one or more recommendations and provide them to the user. One or more machine learning algorithms can include natural language processing (NLP), optical character recognition (OCR) capabilities, computer vision systems, or statistical models.

[0362] One or more machine learning algorithms can include one or more training data sets, the one or more training data sets including example questions or comments from the user regarding food, individual foods, the user's glucose level, the user's additional biomarker levels, the user's lifestyle, any predictions thereof, or any correlations therebetween. The one or more training data sets can allow the one or more machine learning algorithms to learn multiple parameters to generate one or more models (e.g., mathematical models, classifiers) that can be used to distinguish or differentiate different aspects of the user's questions or comments, thereby providing appropriate answers or comments to return to the user.

[0363] In one example, a user can ask the virtual health assistant 280 (e.g., by sending a voice message or a text message) "How is my glucose level?", and the virtual health assistant 280 can provide a summary of the user's glucose level, such as: (1) "After 2:30 p.m., when you consume coffee and a muffin, your glucose level will drop"; (2) "Your glucose level peaks after consuming spaghetti and meatballs"; (3) "In the past two months, it has been noted that when you have dinner after 8 p.m., your glucose level peaks"; (4) "Running or swimming is more effective than weightlifting in managing glucose levels"; or (5) "There has been an abnormal trend in your blood glucose level in the past two days. Please see an endocrinologist as soon as possible. Dr. Christine Emdy is a highly rated endocrinologist in your insurance network and is available for an appointment this week." The virtual health assistant 280 can communicate with healthcare providers and the user's healthcare insurance information to automatically make an appointment (e.g., with a nurse, a general practitioner, a specialist, a dentist, an optometrist, etc.) with or without the user's consent.

[0364] One or more recommendations made by the DMG 101 (e.g., by the food analysis module 210) can be automatically generated by the virtual health assistant 280 and provided to the user. The virtual health assistant 280 can provide one or more recommendations to the user at least once every 1 hour, 2 hours, 3 hours, 4 hours, 6 hours, 12 hours, 24 hours, 2 days, 3 days, 4 days, 5 days, 6 days, 1 week, 2 weeks, 3 weeks, 4 weeks, 2 months or longer. The virtual health assistant 280 can provide one or more recommendations to the user at most once every 2 months, 4 weeks, 3 weeks, 2 weeks, 1 week, 6 days, 5 days, 4 days, 3 days, 2 days, 25 hours, 12 hours, 6 hours, 4 hours, 3 hours, 2 hours, 1 hour or less.

[0365] A user can initiate communication with the virtual health assistant 280 by providing (e.g., via voice or text message) a question (e.g., regarding the user's diet, glucose, or other biomarker levels, etc.). The virtual health assistant 280 can provide an answer to the user's question in at least about 1 second, 2 seconds, 3 seconds, 4 seconds, 5 seconds, 6 seconds, 7 seconds, 8 seconds, 9 seconds, 10 seconds, 20 seconds, 30 seconds, 40 seconds, 50 seconds, 1 minute, 2 minutes, 3 minutes, 4 minutes, 5 minutes, 10 minutes, 20 minutes, 30 minutes, 40 minutes, 50 minutes, 1 hour, 2 hours, 3 hours, 4 hours, 5 hours, or longer. The virtual health assistant 280 can provide an answer to the user's question in at most about 5 hours, 4 hours, 3 hours, 2 hours, 1 hour, 50 minutes, 40 minutes, 30 minutes, 20 minutes, 10 minutes, 9 minutes, 8 minutes, 7 minutes, 6 minutes, 5 minutes, 4 minutes, 3 minutes, 2 minutes, 1 minute, 50 seconds, 40 seconds, 30 seconds, 20 seconds, 10 seconds, 9 seconds, 8 seconds, 7 seconds, 6 seconds, 4 seconds, 3 seconds, 2 seconds, 1 second, or shorter. In some cases, the virtual health assistant 280 can provide an answer to the user's question one or more times. The virtual health assistant 280 can provide at least about 1 time, 2 times, 3 times, 4 times, 5 times, 6 times, 7 times, 8 times, 9 times, 10 times, or more answers.

[0366] In some cases, the virtual health assistant 280 can be configured to ask the user one or more questions after receiving the user's question (e.g., the user's original question). The virtual health assistant 280 can ask one or more questions to receive more information about the user's question. Information about the user's question can help the virtual health assistant 280 better understand the user's question and provide relevant answers to the question.

[0367] Figure 64Displays a window of a GUI-based software interface that shows message exchange (e.g., a conversation) between a user and a coach. The coach can be an expert (e.g., a nurse, a physician, a nutritionist, etc.) who has unrestricted or restricted access to the user's data and its analysis through the DGM 101 (e.g., through the insights and recommendations engine 230). Alternatively or additionally, the coach can be an artificial intelligence algorithm (e.g., a chatbot) of the platform 200 operatively coupled to the DGM 101. The artificial intelligence algorithm can communicate with the user using one or more of the machine learning algorithms described above. The conversation between the coach and the user can be initiated by the user. Alternatively, the conversation between the coach and the user can be initiated by the coach. The coach can notify the user of one or more recommendations generated by the DGM 101 (e.g., by the insights and recommendations engine 230). In some cases, the user may be more receptive (more positive, more encouraged, etc.) to one or more recommendations when notified or instructed through a conversation with the coach rather than an analysis report generated by the DGM 101. In some instances, the coach can notify the user whether the user has shown a balanced glucose response or not. In some instances, the coach and notify the user of any foods and / or times that indicate an adverse glucose response. The coach can also provide one or more recommendations regarding one or more additional biomarker levels of the user. The coach can also ask the user questions that the DGM 101 may need to further analyze the user's data.

[0368] The DGM 101 can further include one or more prepackaged meals containing known amounts of food, wherein the food analysis module 210 can be further configured to (1) monitor the effect of the food in the one or more prepackaged meals on the user's glucose level and / or additional biomarker levels after the user consumes the one or more prepackaged meals for a period of time; and (2) generate a glucose and / or additional biomarker baseline profile of the user based on the monitored effect. The prepackaged meals can be provided to the user in the form of a calibration kit, which can be used to optimize the DGM 101 (e.g., the platform 200 of the DGM 101) for the user's physiological response to different foods.

[0369] The optimized DGM 101 (e.g., platform 200 of DGM 101) may include an optimized food analysis module 210, a device / data hub 220, and an insights and recommendations engine 230, as well as the functionality of at least the glucose level monitor 111 that communicates with these components. Since users can react differently to the same food and the glucose level monitor 111 and medical device 116 can have different compatibilities for different users, a calibration kit can be used to set a food baseline for one or more users. Generating a food baseline profile for a user may include monitoring the impact of different foods (or a single food) on the user's body when the user consumes one or more pre-packaged meals over a period of time. One or more pre-packaged meals may contain known amounts of food. The monitored impacts can be used to generate a food baseline profile. The calibration kit can be a modular kit. The calibration kit may include a monitoring system (e.g., glucose level monitor 111, medical device 116, blood test kit, genetic test, etc.) and one or more standardized meals (also referred to as "calibration meals"). Calibration meals can include food bars, beverages, or both. The platform 200 of DGM 101 can know and have tested all the characteristics of the calibration meals (e.g., ingredients, nutrients, processing, etc.). In some instances, the user may place a device (e.g., glucose level monitor 111) on the body (or perform a provided monitoring test, e.g., blood test kit) and consume one calibration meal every morning. The user may be required to fast overnight (e.g., for 12 hours). The device can measure the user's reaction to the calibration meal. The user can consume other foods throughout the day and track the foods to the platform 200 via a GUI-based software interface (e.g., food tracking via text and speech recognition analysis, seamless food image recorder, etc.). After a short period (e.g., one week), the platform 200 can use the data and predictions to set a baseline for the user. The baseline can be referred to as the user's unique personalized food "fingerprint".

[0370] Figure 65Shows an exemplary calibration kit 6500. The calibration kit can include a box 6502 containing a glucose level monitor 111, and an additional box 6504 containing one or more calibration meals. The calibration kit can include a different box 6506 containing a DNA collection kit (e.g., a saliva collection kit) for DNA testing. The calibration kit can include another different box 6508 containing a biome collection kit for microbiome analysis (e.g., samples collected from the gut, genitalia, mouth, nose, and / or skin). The calibration kit can optionally include any of the above boxes, or different combinations of boxes. The calibration kit can include 1, 2, 3, 4, 5, or more calibration foods. The calibration kit can include 2, 3, 4, 5, 6, or more boxes. The calibration kit can include 1, 2, 3, 4, 5, or more monitoring systems. In some embodiments, the calibration kit can include one or more other components / devices (e.g., medical device 116, wearable device, or other biomarker test / device) that help generate a user's baseline health status. The calibration kit can also include a detailed list of instructions for easy user compliance. If needed, the calibration kit can include a container for each collection kit.

[0371] Computer system

[0372] The present disclosure provides a computer system programmed to implement the methods of the present disclosure. Figure 66 Illustrates a computer system 6601 programmed or otherwise configured to operate a DGM 101. The computer system 6601 can regulate various aspects of the DGM 101 of the present disclosure, e.g., platform 200 including food analysis module 210, device / data hub 220, and insights and recommendations engine 230. The computer system 6601 can be a user's electronic device or a computer system remotely located relative to the electronic device. The electronic device can be a mobile electronic device.

[0373] The computer system 6601 includes a central processing unit (CPU, also referred to herein as "processor" and "computer processor") 6605, which can be a single-core or multi-core processor, or multiple processors for parallel processing. The computer system 6601 also includes a memory or memory location 6610 (e.g., random access memory, read-only memory, flash memory), an electronic storage unit 6615 (e.g., hard disk), a communication interface 6620 for communicating with one or more other systems (e.g., network adapter), and peripheral devices 6625 such as caches, other memories, data storage areas, and / or electronic display adapters. The memory 6610, storage unit 6615, interface 6620, and peripheral devices 6625 communicate with the CPU 6605 via a communication bus (solid lines), such as a motherboard. The storage unit 6615 can be a data storage unit (or data repository) for storing data. The computer system 6601 can be operably coupled to a computer network ("network") 6630 via the communication interface 6620. The network 6630 can be the Internet, the Internet and / or an extranet, or an intranet and / or extranet that communicates with the Internet. The network 6630 is in some cases a telecommunications network and / or a data network. The network 6630 can include one or more computer servers that can implement distributed computing, such as cloud computing. The network 6630 can in some cases implement a peer-to-peer network via the computer system 6601, which can enable devices to be coupled to the computer system 6601 to act as clients or servers.

[0374] The CPU 6605 can execute a series of machine-readable instructions, which can be embodied in a program or software. The instructions can be stored in a memory location such as the memory 6610. The instructions can relate to the CPU 6605, which can then be programmed or otherwise configured to implement the methods of the present disclosure. Examples of operations performed by the CPU 6605 can include fetching, decoding, executing, and writing back.

[0375] The CPU 6605 can be a circuit, such as part of an integrated circuit. One or more other components of the system 6601 can be included in the circuit. In some cases, the circuit is an application-specific integrated circuit (ASIC).

[0376] The storage unit 6615 can store files, such as drivers, libraries, and saved programs. The storage unit 6615 can store user data, e.g., user preferences and user programs. The computer system 6601 can in some cases include one or more additional data storage units located external to the computer system 6601, such as on a remote server that communicates with the computer system 6601 via an intranet or the Internet.

[0377] The computer system 6601 can communicate with one or more remote computer systems via a network 6630. For example, the computer system 6601 can communicate with a user's remote computer system. Examples of remote computer systems include personal computers (e.g., portable PCs), tablets or tablet PCs (e.g., Apple® iPad, Samsung® Galaxy Tab), telephones, smart phones (e.g., Apple® iPhone, Android-enabled devices, Blackberry®), or personal digital assistants. A user can access the computer system 6601 via the network 6630. Other examples of remote computer systems include the glucose level monitor 111 and the medical device 116.

[0378] The methods described herein can be implemented by machine (e.g., computer processor) executable code stored on an electronic storage location of the computer system 6601 (e.g., on the memory 6610 or the electronic storage unit 6615). The machine executable or machine readable code can be provided in the form of software. During use, the code can be executed by the processor 6605. In some cases, the code can be retrieved from the storage unit 6615 and stored on the memory 6610 for ready access by the processor 6605. In certain instances, the electronic storage unit 6615 can be excluded, and the machine executable instructions are stored on the memory 6610.

[0379] The code can be pre-compiled and configured for use with a machine having a processor adapted to execute the code, or can be compiled during runtime. The code can be supplied in a programming language, and the programming language can be selected to enable the code to be executed in a pre-compiled or as-compiled manner.

[0380] Aspects of the systems and methods provided herein, such as server 6601, may be embodied in programming. Various aspects of the technology may be regarded as a "product" or "article" typically in the form of machine (or processor) executable code and / or associated data that is executed on or embodied in one type of machine-readable medium. The machine executable code may be stored in an electronic storage unit such as a memory (e.g., read-only memory, random access memory, flash memory) or a hard disk. A "storage" type medium may include any or all of the tangible memory of a computer, processor, etc., or associated modules thereof, such as various semiconductor memories, tape drives, hard disk drives, etc., that may provide non-transitory storage for software programming at any time. All or part of the software may sometimes be communicated via the Internet or various other telecommunications networks. Such communication may, for example, enable the loading of software from one computer or processor into another, such as from an administrative server or a host computer into the computer platform of an application server. Thus, another type of medium that may carry software elements includes light waves, radio waves, and electromagnetic waves used via wired and optical landline networks as well as across physical interfaces between local devices over various air links. Physical elements that carry such waves, such as wired or wireless links, optical links, etc., may also be regarded as media that carry software. As used herein, unless restricted to non-transitory, tangible "storage" media, terms such as computer or machine "readable media" refer to any medium that participates in providing instructions to a processor for execution.

[0381] Thus, a machine-readable medium (such as computer-executable code) can take many forms, including but not limited to tangible storage media, carrier media, or physical transmission media. Non-volatile storage media includes, for example, optical discs or magnetic disks, such as any storage device in any one or more storage devices in a computer, such as can be used to implement a database shown in the drawings. Volatile storage media includes dynamic memory, such as the main memory of such a computer platform. Tangible transmission media includes coaxial cables; copper wire and fiber optics, including the wires that comprise a bus within a computer system. Carrier transmission media can take the form of electrical or electromagnetic signals, or acoustic or light waves, such as those generated during radio frequency (RF) and infrared (IR) data communications. Thus, common forms of computer-readable media include, for example: floppy disks, flexible disks, hard disks, magnetic tape, any other magnetic media, CD-ROM, DVD or DVD-ROM, any other optical media, punch cards, paper tape, any other physical storage media with hole patterns, RAM, ROM, PROM, and EPROM, flash-EPROM, any other memory chip or cartridge, a carrier wave carrying data or instructions, a cable or link carrying such a carrier wave, or any other medium from which a computer can read program code and / or data. Many of these forms of computer-readable media can involve carrying one or more sequences of one or more instructions to a processor for execution.

[0382] Computer system 6601 may include or communicate with an electronic display 6635, the electronic display including a user interface (UI) 6640 for providing, for example, food image ratings, analysis reports, and / or one or more recommendations. Examples of UIs include, but are not limited to, graphical user interfaces (GUIs) and web-based user interfaces.

[0383] Remarks

[0384] While the preferred embodiments of the present disclosure have been shown and described herein, it will be apparent to those of ordinary skill in the art that such embodiments are provided by way of example only. Numerous variations, changes, and substitutions will now occur to those skilled in the art without departing from the present disclosure. It should be understood that various alternatives to the embodiments described herein may be employed. It is intended that the following claims define the scope of the present disclosure and thereby cover the methods and structures within the scope of these claims and their equivalents.

Claims

1. A system for mapping food, comprising: One or more processors; And One or more processor-readable media storing instructions which, when executed by the one or more processors, cause the execution of: Obtaining user input, the user input including a text input indicating quantitative information for a food item; Determining nutritional information for the food item based on the quantitative information for the food item, wherein the quantitative information for the food item is different from the nutritional information for the food item; and Generating a glucose management recommendation based on the nutritional information.

2. The system according to claim 1, wherein, The nutritional information includes the carbohydrate content of the food item.

3. The system according to claim 1, wherein The quantitative information includes the quantity, serving size, or both of the food item.

4. The system according to claim 1, wherein The food item is to be consumed by a user of a wearable insulin delivery device.

5. The system according to claim 1, wherein, The glucose management recommendation includes an insulin delivery recommendation.

6. The system according to claim 1, wherein The glucose management recommendation includes a dietary recommendation.

7. The system according to claim 1, wherein The glucose management recommendation includes a lifestyle recommendation.

8. A processor-implemented method for mapping food, comprising: Obtaining user input, the user input including a text input indicating quantitative information for a food item; Determining nutritional information for the food item based on the quantitative information for the food item, wherein the quantitative information for the food item is different from the nutritional information for the food item; and Generating a glucose management recommendation based on the nutritional information.

9. The method according to claim 8, wherein The nutritional information includes the carbohydrate content of the food item.

10. The method according to claim 8, wherein, The quantitative information includes the quantity, serving size, or both of the food item.

11. The method according to claim 8, wherein, The food item is to be consumed by a user of a wearable insulin delivery device.

12. The method according to claim 8, wherein The glucose management recommendation includes an insulin delivery recommendation.

13. A system for mapping food, comprising: One or more processors; And One or more processor-readable storage media storing instructions which, when executed by the one or more processors, cause the execution of: Obtaining user input, the user input including a text input indicating quantitative information for a food item; Determining nutritional information for the food item based on the quantitative information for the food item, wherein the quantitative information for the food item is different from the nutritional information for the food item; and Generating an insulin delivery recommendation based on the nutritional information.

14. The system according to claim 13, wherein, The nutritional information includes the carbohydrate content of the food item.

15. The system according to claim 13, wherein Generating the insulin delivery recommendation includes accessing an equation that models the relationship between carbohydrates and insulin.

16. A processor-implemented method for mapping food, comprising: Obtaining user input, the user input including a text input indicating quantitative information for a food item; Determining nutritional information for the food item based on the quantitative information for the food item, wherein the quantitative information for the food item is different from the nutritional information for the food item; and Generating an insulin delivery recommendation based on the nutritional information.

17. A processor-implemented method for mapping food, comprising: Determining a user's consumption of a first food item; Receive data indicating the user's glucose level; Determine the effect of the first food item on the user's glucose level based on a change in the user's glucose level; And Store the effect of the first food item on the user's glucose level in a database that includes data indicating the effect of individual food items on the user's glucose level.

18. The processor-implemented method according to claim 17, wherein, Determining the user's consumption of the first food item includes analyzing (1) one or more images of one or more food items consumed by the user, (2) one or more videos of one or more food items consumed by the user, (3) one or more voice logs of one or more food items consumed by the user, (4) one or more text logs of one or more food items consumed by the user, (5) one or more barcodes associated with one or more food items consumed by the user, (6) one or more labels associated with one or more food items consumed by the user, or a combination thereof.

19. The processor-implemented method according to claim 17, wherein, Determining the user's consumption of the first food item includes: Determining the quantity of the first food item; Determining the serving size of the first food item; Determining one or more ingredients within the first food item; Determining the nutritional information of the first food item; Or a combination thereof.

20. The processor-implemented method according to claim 19, wherein, Determining the one or more ingredients or the nutritional information includes determining the one or more ingredients or the nutritional information based on the food ontology of the first food item.

Images