Food placemat with integrated camera, microcontroller and weighing platform for automatic estimation of meal composition in terms of calories, macros and micronutrients

Abstract

The present invention relates to an innovative food placemat (1) equipped with an integrated camera (3), an advanced microcontroller, and a weighing platform (2) designed to revolutionize the daily nutritional intake monitoring process. By using artificial vision algorithms and data analysis, the placemat is capable of automatically estimating meal composition in terms of calories, macros, and micronutrients. The user places the food on the placemat (4), the system starts capturing images of the meal and processes the information to provide an accurate estimate of total calories and nutrients present. This enables more precise dietary control and facilitates meal planning based on individual nutritional goals. The invention represents a significant advancement in nutrition management, contributing to promoting a healthier lifestyle through simple and effective food intake monitoring.

Description

[0001] FOOD PLACEMAT WITH INTEGRATED CAMERA, MICROCONTROLLER AND WEIGHING PLATFORM FOR AUTOMATIC ESTIMATION OF MEAL COMPOSITION IN TERMS OF CALORIES, MACROS AND MICRONUTRIENTS

[0002] DESCRIPTION

[0003] Field of the invention

[0004] This patent application relates to the field of technologies related to estimating the nutritional intake of meals.

[0005] Background of the invention

[0006] The current state of the art in the field of technologies related to estimating the nutritional intake of meals includes several innovations and significant developments. Some of these include:

[0007] Mobile Apps and Software: There are numerous mobile applications and software solutions that allow users to manually record the foods consumed (including scanning the barcode on food items) and obtain information about their nutritional intake. However, these solutions require manual interaction and can often be prone to data entry errors, especially when it comes to portion estimation.

[0008] Food Labelling: Food product labelling has improved in providing detailed information about the nutritional content of foods, by including lists of calorie and macronutrient thereof. However, even in this case, the sizes of an individual portion are not taken into account.

[0009] Advanced Imaging Technologies: Imaging techniques such as mass spectrometry and Raman or infrared spectroscopy have been used to analyze the chemical composition of foods. However, these techniques often require specialized equipment and are not suitable for household use.

[0010] Artificial Vision: Artificial vision techniques are used to identify the type of meal consumed through the camera of the user’s smartphone. However, also in this case, the technology requires constant manual interaction and suffers from errors in calorie calculation, mainly due to only estimating the weight of the meal rather than relying on a real measure thereof.

[0011] For example, the publication W02020086795 describes a platform assembly for monitoring food consumed from one or more dishes supported by the platform assembly includes a housing defining an interior and having an upper surface for supporting the one or more dishes. One or more weight assemblies are disposed in the interior of the housing and each weight assembly includes a weight sensor configured to detect a weight of one dish of the one or more dishes and to provide a weight signal corresponding to the detected weight of the one dish. A difference between the weight taken at a first time during a meal and at a second time, after the first time, during the meal being generally indicative of an amount of food consumed during the meal from the one or more dishes.

[0012] Another example is given by JP4838383 that describes a portable scale allowed to be easily carried by a person often taking a lunch or the like out because of diet or health management and capable of accurately measuring dietary intake weight, and to provide a management system such as a cellular phone interlocking with the portable scale.

[0013] Summary of the invention

[0014] Considering the state of the art described above, the proposed food placemat according to the invention, equipped with a camera for automatic estimation of nutritional intake represents a significant advancement over the current state of the art. By integrating artificial vision and data analysis into an accessible and user-friendly device, this technology promises a practical and accurate solution for real-time monitoring of the nutritional intake of meals consumed.

[0015] This innovative system offers numerous advantages. Firstly, it greatly simplifies the process of monitoring nutritional intake and maintaining a detailed food diary, solving a number of technical challenges that often hinder individuals from achieving their dietary goals. The use of external tools such as nutritional tables or recording applications requires time and effort, which discourage people from adopting more conscious eating habits. The food placemat equipped with a camera, a microcontroller, and a weighing platform according to the present invention makes the process of monitoring nutritional intake immediate and seamless, encouraging adherence to the goal of eating healthier. The device’s intuitive interface makes it easy to use and accessible to everyone, thereby helping to overcome technological barriers that could discourage some users.

[0016] A common problem encountered by people that try to monitor their food intake is the manual and laborious calculations required to estimate the calories and nutrients in meals. The accuracy of such calculations is often compromised by human errors in interpreting the nutritional information of individual ingredients. The placemat with a camera and a microcontroller according to the present invention eliminates this complication by providing automatic and precise estimates of calories and nutrients based on a comprehensive visual analysis of the meal. This solves the challenge of accurately calculating nutritional intake, enabling users to obtain reliable data and, therefore, make informed dietary decisions. Portion sizes vary widely, and accurately measuring small or irregular amounts can be challenging. The technology of the food placemat according to the present invention solves this problem by using multiple techniques, including capturing a detailed image set of the meal and analyzing each component, to accurately estimate the amounts consumed. This makes it easier to adjust the nutritional intake to specific goals, such as weight loss or maintaining a healthy eating style.

[0017] In conclusion, the proposed system addresses the technical challenges of accurately monitoring nutritional intake by simplifying the creation of a detailed food diary. By significantly reducing calculation errors, the system according to the present invention provides precise estimates of the amount of food consumed and makes the monitoring process more practical and accessible to the user. This, in turn, promotes more conscious dietary choices and a healthier lifestyle.

[0018] The food placemat system (1) according to the invention is equipped with a camera (3), a microcontroller, and a weighing platform (2) designed to transform the process of monitoring the nutritional intake of meals. This advanced system utilizes artificial vision technologies and data analysis to provide automatic and accurate estimates of the calories, macros, and micronutrients contained in meals.

[0019] The device comprises a weighing platform with an integrated camera and a microcontroller that ensures wireless connectivity to the internet or directly to a smartphone. The user places the food on the surface of the placemat (Fig. 3), activating the data acquisition process. The camera captures detailed images during the meal consumption, which are then instantly processed by the microcontroller, either locally, if the acquired information is processed locally, or on dedicated external processors through a wireless connection. The results of this processing are saved in the user’s food diary, which can be accessed through a dedicated smartphone application (Android or iOS) or through a dedicated website.

[0020] Thanks to advanced image and dataset analysis algorithms, combined with the detected weight, the system accurately identifies and quantifies the meal’s constituent elements, including their portions, the ingredients, and their relative quantities. Using a large database containing the ingredients and the proportions of the most common recipes and the nutritional information thereof, the system calculates the calories, the micro and macro nutrients, and the vitamins present in the meal. The results are recorded within the user’s food diary. This diary can be edited, integrated, and modified as desired. In the implementation shown in Fig. 4 of the food placemat according to the present invention, this information is also displayed in real-time on a small screen (5) and recorded in the user’s food diary.

[0021] Brief description of the drawings

[0022] In these drawings:

[0023] Fig. 1 shows the food placemat (1), the weighing platform (2), and the meal detection camera (3) in an open configuration, with the meal detection camera (3) in raised position relative to the placemat’s surface.

[0024] Fig. 2 shows the food placemat (1), the weighing platform (2), and the meal detection camera (3) in a closed configuration, where the camera (3) is placed parallel to the placemat’s surface so as to occupy less space.

[0025] Fig. 3 shows a meal (4) placed on the on the weighing platform (2) of the food placemat (1) with the meal detection camera (3) in a raised position relative to the placemat’s surface.

[0026] Fig. 4 shows a different implementation of the food placemat (1) of Figure 1 , including a screen (5), for example a touch-screen, for real-time display of meals added to the food diary (software application), along with additional information. Additionally, physical buttons (6) are included to allow for a more comprehensive interaction (possible even with the implementation of a touch-screen) by enabling the user to initiate the meal detection and manually control other features of the system, such as the calibration phase. Detailed description of the invention

[0027] Once the user lifts the camera (3 in Fig. 2) to its final position (3 in Fig. 1), the placemat transitions from the off mode to the standby mode. At this point, as soon as the weighing platform (2) detects a significant weight oscillation caused by a meal (4) placed on the weighing platform (2), the camera (3) begins to capture images. Different strategies or a combination thereof are used to evaluate portions and accurate quantities of ingredients.

[0028] Strategy 1 : A meal composed of a multitude of ingredients (such as, for example, a rich mixed salad) can be assembled directly on the placemat. Through the analysis of frames recorded instant by instant associated with the meal’s weight, it is possible to accurately estimate the proportions and quantities of the various ingredients.

[0029] Strategy 2: In cases where the meal has already been assembled beforehand or if the meal consists of a more complex recipe, the artificial vision system is able to identify the specific recipe in question (e.g., spaghetti carbonara) or make a more approximate estimation of the various components (e.g., a steak with vegetables next to it) through image segmentation.

[0030] Ongoing Monitoring and Adjustments

[0031] To address different possible scenarios as the meal progresses, the system continuously monitors changes by capturing new images and weight data. A Planner, based on a Large Language Model (LLM), compares the textual descriptions (through Visual Language Models, VLMs) of successive images and their weights to detect any differences. Based on these differences, the LLM may decide to perform various actions such as segmentation, re-segmentation, classification, reclassification, or adjustment of the weight distribution if significant changes are detected.

[0032] The LLM updates the classification and weight assignment based on the observed changes. Upon detecting the end of the meal (e.g., removal of the plate from the platform or a closing of the camera), the LLM performs a final analysis. The system may re-run segmentation and classification to ensure all remaining items are correctly identified. Non-food items (e.g., a glass or an utensil) detected during the meal are excluded from the final food weight calculation. The final weights and classifications of the remaining food items are updated, and the system generates a comprehensive report detailing a clear summary of the meal’s consumption, including the total amount of food consumed.

[0033] Algorithm Steps

[0034] The modified vision algorithm is controlled by a Planner based on an LLM that decides which actions to compute given the input image and weight data. The LLM can decide to perform a variety of actions described below:

[0035] Initial Data Capture and Analysis

[0036] • Action: The system captures an initial image of the platform and records the total weight of the items placed on it. The LLM planner calls a Vision Language Model (VLM) to convert the image into a textual description and utilizes a segmentation algorithm (e.g., Segment Anything Model, SAM, Foodvisor etc.) to identify all the items on the placemat, including non-food items.

[0037] Image Segmentation and Classification

[0038] • Action: The segmentation algorithm (e.g., SAM, possibly fine-tuned on food datasets) divides the image into distinct regions, each corresponding to a different food item or non-food item (e.g., chicken, glass, fork, etc.).

[0039] Weight Association

[0040] • Action: The system calculates the initial weight of the entire platform and associates portions of this weight with each classified food and non-food item based on their segmented areas or values known from previous interactions. The weight of the food items is distributed across the identified classes (e.g., chicken 150g, rice 200g).

[0041] Difference Analysis

[0042] • Action: This analysis compares two successive images to detect any significant changes. It is particularly useful when a relevant or sudden weight change occurs to understand what has happened.

[0043] Example Scenario: Meal Consumption Process

[0044] Step 1 : Initial Setup

[0045] • Scenario: The user places a plate with chicken, rice, and broccoli on the platform. The platform captures an image and records the initial weight as 900g.

[0046] Step 2: Initial Analysis

[0047] • Action: The LLM planner receives the initial image and outputs from the VLM and SAM models, identifying three distinct food items (chicken, rice, broccoli) and three non- food items (plate, fork, glass).

[0048] Step 3: Weight Assignment

[0049] • Action: After segmentation and classification, the weights are assigned to the different segments in proportion to their segment area or based on pre-determined values from previous interactions (e.g., plate weight) and density values stored in a dedicated database:

[0050] Segment 1 : Classified as chicken, 200g.

[0051] Segment 2: Classified as rice, 150g.

[0052] Segment 3: Classified as broccoli, 150g.

[0053] Segment 4: Classified as Ceramic Plate, 300g (estimate based on previous uses).

[0054] Segment 5: Classified as Fork, 50g (estimate based on previous uses).

[0055] Segment 6: Classified as Glass, 50g (estimate based on previous uses).

[0056] Step 4: Eating Process

[0057] • Scenario: The user eats some of the chicken and rice. The system detects a weight reduction of 150g and captures a new image.

[0058] • Action: The LLM runs a difference analysis between the initial and current images, detecting that the chicken and rice segments have changed. The LLM updates the classification and weight distribution:

[0059] Chicken, 100g.

[0060] Rice, 100g.

[0061] Broccoli, 150g (unchanged).

[0062] Ceramic Plate, 300g (unchanged).

[0063] Fork, 50g (unchanged).

[0064] Glass, 50g (unchanged).

[0065] Step 5: Adding a Non-Food Item

[0066] • Scenario: The user fills the glass with water, increasing the weight by 200g.

[0067] • Action: The system recognizes this change and, upon analysis, determines that the glass is not food (based on the image difference highlighting the glass as the only change) and exclude it from the food weight calculation (optionally, it can be included in a dietary diary). Step 6: Utensil Interaction

[0068] • Scenario: The user places a spoon on the platform.

[0069] • Action: The system detects a small weight increase and a new object in the image. The LLM triggers the difference algorithm and SAM, identifying the new item as a spoon, which is then excluded from food classification and weight association.

[0070] Step 7: Final Consumption

[0071] • Scenario: The user removes the remaining food from the platform, causing a sudden weight change.

[0072] • Action: The system recognizes this as the end of the meal, analysis the image just before the change, and detects that only a small portion of broccoli is left. The system records the final weight (700g) and performs a final segmentation and classification, confirming that only broccoli, the plate, fork, spoon, and the full glass remain. It then adjusts the final weight assignments accordingly.

[0073] Final Output

[0074] • Action: The system outputs a report showing the consumption of food items and their weights, excluding non-food items (e.g., fork, glass of water). The report includes a summary of the meal’s nutritional content:

[0075] Consumed chicken: 200g.

[0076] Consumed rice: 150g.

[0077] Consumed broccoli: 100g (50g left).

[0078] Total Calories: 761 kcal, including:

[0079] ■ Carbohydrates: 109g.

[0080] ■ Protein: 64g.

[0081] ■ Fats: 7g.

[0082] Advantages

[0083] • Dynamic Adaptation: The LLM-driven approach allows the system to dynamically adapt to changes during the meal, ensuring accurate tracking, recognition, and personalization.

[0084] • Versatility: The algorithm handles various scenarios, from simple meals to complex ones with multiple items and interactions. • Precision: By integrating weight data with image analysis, the system provides a precise estimation of food consumption.

[0085] • Personalization: The LLM approach accounts for personalization by storing and retrieving previous successful interactions with users. It learns particular habits or recipes and uses this knowledge for better planning and action choices.

[0086] Furthermore, this analysis of the frames associated with their respective weights allows for an easy plate tare removal and the exclusion from the food diary recording of any leftovers or uneaten meal portions.

[0087] In fact, by analyzing the frame taken just before the end of the meal, easily detectable by the weighing platform no longer registering a weight, it is possible to subtract the weight of the plate or container used and any uneaten or inedible portions, such as bones. In any case, users can access their food diary to adjust the proportions, but with the certainty and constraint of a precisely measured overall weight.

[0088] Let’s consider the example of a slightly more complex meal, such as spaghetti carbonara. The system initially detects a weight of 700 grams, correctly identifying the presence of spaghetti carbonara. At the end of the meal, not all of it has been consumed, and in the frame just before the removal of the meal from the placemat, a recorded weight of 450 grams is detected. In this case, the system will record in the food diary, which can be accessed and modified through the user’s personal area on a PC (website) or through the dedicated application, 250 grams of spaghetti carbonara.

[0089] The system has access to a database containing a basic recipe for spaghetti carbonara, including its ingredients. Specifically, we have: Raw spaghetti 320 g (800 g cooked), Guanciale 150 g, 6 Egg yolks (medium-sized) 108 g, Pecorino Romano cheese 50 g, Black pepper as needed, for a total weight of 1108 grams. The associated nutritional value is 2720 kCal, consisting of macronutrients: Carbohydrates 344 g, Proteins 108 g, Fats 104 g, and various micronutrients such as Cholesterol, Sodium, Potassium, Saturated and Unsaturated Fats, Fiber, Sugar, etc.

[0090] Through a simple proportion calculation, the system can assign macronutrients and kCal to the meal. In this case, we would have 614 kCal with 78 g of Carbohydrates, 24 g of Proteins, and 23 g of Fats. Furthermore, to the system allows the user to customize the basic recipe by entering one’s own variations and proportions used. The system will automatically update the calculation based on the correct readings and / or measurements and remember the recipe for future assignments of spaghetti carbonara meals.

[0091] The system can be put into a continuous learning state and personalized, as the captured images can be used along with corrections to train an alternative personalized system that is better at recognizing the user’s own cooking style and the most common individual recipes. Such alternative personalized system has also an improved container tare removal accuracy, especially if the same container is consistently used for meal consumption.

[0092] The system’s design includes an internal battery for portability and waterproofing, making it washable.

[0093] The potential application of this system is broad and diverse. It can be used by individuals following specific diets, athletes, health professionals, nutritionists and patients with diabetes. Moreover, it can be adopted in foodservice facilities, corporate cafeterias, company canteens and hospitals, improving food quality control and facilitating the provision of personalized meals.

[0094] In conclusion, this innovative food placemat with a camera, a microcontroller, and weighing platform represents an advanced solution for the automatic estimation of meal nutritional intake. The technology combines the accuracy of artificial vision and data analysis with the convenience of daily use, helping to promote more conscious eating habits and a healthier lifestyle.

Claims

CLAIMS1. A food placemat (1) equipped with an integrated camera (3), a microcontroller and a weighing platform (2), characterized by the fact that:- said camera (3) is capable of capturing multiple images of objects placed on the weighing platform (2) and transmitting such images to said microcontroller;- said weighing platform (2) is capable of acquiring the weight of objects and transmitting said data to said microcontroller;- said microcontroller is capable of collecting the images transmitted by the camera (3) and the weight data transmitted by weighing platform (2) and to process them, directly or externally transmitting them to a dedicated external processor through wireless methods, wherein said data processing is performed by using modified artificial vision algorithms capable of processing video images and meal weights for the purpose of classifying foods and assigning them corresponding quantities and nutritional characteristics.

2. A food placemat (1) according to claim 1 , characterized by the fact that the data processed by the microcontroller or the dedicated external processor is recorded on a virtual food diary accessible and modifiable through a user interface on a PC (website) or smartphone (dedicated application).

3. A food placemat (1) according to claim 1 , characterized by a different implementation that requires the addition of a screen (5), which may also be touchscreen, capable of providing real-time information about the processing and classification performed, physical buttons (6) or the afore- mentioned touch-screen, in the case of touch-screen functionality, enhance the classification and data acquisition process through manual input.

4. A food placemat (1) according to claim from 1 to 3, comprising an internal battery.

5. A food placemat (1) according to claim from 1 to 4 characterized by the fact to be waterproof, making it washable.

6. A proceeding for monitoring the consumption of food implemented by the food placemat according to one of the previous claims comprising the step of:A. capturing an initial image of the platform and records the total weight of the itemsplaced on it;B. processing the image to identify all the items on the placemat, including non-food items;C. processing the image to divide it into distinct regions, each corresponding to a different food item or non-food item;D. associating weight portions of the total weight of the items to each classified food and non-food item;E. when a weight change occurs, capturing a successive image;F. repeating the steps from B. to D. to process the successive image of the platform; G. comparing the initial and successive image to detect any significant changes;H. providing a report showing the consumption of food items and their weights, excluding non-food items, and a summary of the meal’s nutritional content.

Images