system

Abstract

Provide a system. A food image acquisition means; means for analyzing the meal image to identify the type and amount of food; means for calculating calories based on the identified foods; a means for acquiring exercise data; means for calculating calories burned based on the exercise data; A means for calculating a calorie balance based on calorie intake and calorie expenditure; means for providing dietary and exercise advice to a user based on said calorie balance.

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In today's society, there is a demand for tools that enable effective health management and dieting. Accurately understanding calorie intake and calorie expenditure is particularly important for maintaining a proper balance between diet and exercise. However, manually recording and managing this data is time-consuming and often lacks accuracy. Furthermore, typical calorie calculation tools have difficulty understanding a user's specific dietary and exercise data in real time. This poses a challenge, as users are unable to receive appropriate dietary advice, making it difficult to achieve their goals. [Means for solving the problem]

[0005] The present invention provides a system including a means for acquiring food images, a means for analyzing the food images to identify the types and amounts of food, a means for calculating calories based on the identified foods, a means for acquiring exercise data, a means for calculating calories burned based on the exercise data, a means for calculating a calorie balance based on calories ingested and calories burned, and a means for providing a user with diet and exercise advice based on the calorie balance. This system enables a user to automatically and accurately manage calories ingested and burned and receive appropriate diet advice. As a result, health management and diet goals can be efficiently achieved.

[0006] A "food image" is an image file of a photograph of food that a user takes.

[0007] "Analysis" refers to the process of mechanically processing food images and exercise data and determining their content.

[0008] "Food type" refers to the specific classification of food contained in the food image, and includes, for example, bread, omelet, salad, and the like.

[0009] "Quantity" refers to the specific quantity of a food item, such as the physical size or weight of the identified food item.

[0010] A "calorie" is a unit of energy that a living organism takes in or expends through food and exercise.

[0011] "Exercise data" is information about the exercise performed by the user, and includes the number of steps, heart rate, exercise time, and the like.

[0012] "Calorie balance" refers to the difference between calories ingested and calories expended, and is also called energy balance.

[0013] "Advice" refers to guidance or policy recommendations given to the user, and specifically refers to specific advice regarding diet and exercise.

[0014] A "system" refers to a set of devices and software that operate in combination with multiple means to assist a user in managing their health.

[0015] "Means" refers to a device, method, or combination thereof used to accomplish a particular result. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] The present invention relates to a system that combines images of food consumed by a user with exercise data to manage the daily calorie balance and provide appropriate diet advice. This system includes means for acquiring food images, analyzing images, calculating calories, acquiring exercise data, calculating calories burned, calculating calorie balance, and providing advice.

[0038] System configuration

[0039] 1. Acquiring food images

[0040] Device (smart glasses): When the user eats, it takes pictures of the meal. When the user presses the start button, the photo of the meal is automatically taken.

[0041] 2. Analysis of food images

[0042] Server: The image data of the photographed meal is sent to the server, which uses image analysis algorithms to identify the meal contents (e.g., bread, omelet, salad) and the amount of each.

[0043] 3. Calorie Counting

[0044] Server: Retrieves calorie information corresponding to each identified food category from the database and calculates the total calories. For example, if bread is 250 kcal, omelet is 150 kcal, and salad is 50 kcal, the total will be calculated as 450 kcal.

[0045] Terminal (smartphone): Notifies the user of the calculation results and saves them as a meal history.

[0046] 4. Acquisition of exercise data

[0047] Device (smartwatch): When a user exercises, it automatically records exercise data such as the number of steps, heart rate, and exercise time. For example, if a user runs for 30 minutes, the data will be recorded.

[0048] Server: Exercise data is periodically sent to the server and stored for analysis.

[0049] 5. Calorie Calculation

[0050] Server: Calculates the calories burned based on the exercise data sent. For example, it calculates that 230 kcal is burned by running for 30 minutes.

[0051] Terminal (smartphone): Notifies the user of the calculation results.

[0052] Specific examples

[0053] Acquiring food information and calculating calories

[0054] User: I eat bread and an omelet for breakfast.

[0055] Terminal (smart glasses): Takes a photo of the meal and sends it to the server.

[0056] Server: Performs image analysis to distinguish between bread and omelet. Calculates the calories as 250 kcal for bread and 150 kcal for omelet, for a total of 400 kcal.

[0057] Device (smartphone): Display the total as 400 kcal to the user and save the data.

[0058] Acquiring exercise information and calculating calories burned

[0059] User: Go for a 30-minute run in the morning.

[0060] Device (smartwatch): Records running exercise data and sends it to the server.

[0061] Server: Calculate the calories burned by running and it is 230 kcal.

[0062] Calculates calorie balance and provides advice

[0063] Server: Based on the calories ingested (400 kcal from breakfast) and calories expended (230 kcal from running), calculate your calorie balance (400 kcal - 230 kcal = 170 kcal).

[0064] Server: Analyzes whether the calorie balance is appropriate for the user's diet goals. If necessary, provides guidance on reducing calorie intake at the next meal.

[0065] Device (smartphone): Display the advice to the user, "Keep your next meal under 200 kcal."

[0066] In this way, users can accurately manage their daily calorie intake and expenditure and receive appropriate advice on maintaining a healthy weight. This system is expected to dramatically improve the success rate of dieting and health management.

[0067] The processing flow will be explained below.

[0068] Step 1:

[0069] Device (smart glasses): Before the user starts eating, they take a photo of their meal. When the user presses a button, the smart glasses automatically take a photo of the meal.

[0070] Step 2:

[0071] Terminal (smart glasses): The captured food images are sent to the server. The image data is often compressed before being transferred.

[0072] Step 3:

[0073] Server: Prepares the received food images for analysis, specifically by pre-processing the images to apply image recognition algorithms.

[0074] Step 4:

[0075] Server: Uses image recognition algorithms to identify the type and quantity of food in a meal image. For example, identify if an image contains bread, an omelet, or a salad.

[0076] Step 5:

[0077] Server: Retrieve calorie information for each identified food type from the database. For example, bread is 250 kcal, omelette is 150 kcal, and salad is 50 kcal.

[0078] Step 6:

[0079] Server: Add up the calories from each food item to calculate your total calorie intake. In this example, we'll use bread, omelet, and salad as a total of 450 kcal.

[0080] Step 7:

[0081] Device (smartphone): The calculated total calories of 450 kcal are notified to the user's smartphone and saved in the meal history.

[0082] Step 8:

[0083] Device (smartwatch): Records exercise data in real time when the user exercises. For example, it records the number of steps, heart rate, and exercise time during a 30-minute run.

[0084] Step 9:

[0085] Device (smartwatch): Periodically sends recorded exercise data to the server.

[0086] Step 10:

[0087] Server: Based on the received exercise data, calculates the calories burned using exercise intensity, time, and the user's physical data (weight, age, etc.). For example, it calculates that 230 kcal is burned by running for 30 minutes.

[0088] Step 11:

[0089] Terminal (smartphone): Notifies the user of the calculated calorie consumption of 230 kcal.

[0090] Step 12:

[0091] Server: Combines the data of calorie intake (450 kcal) and calorie expenditure (230 kcal) to calculate the daily calorie balance. Here, the calorie balance is calculated as 450 kcal - 230 kcal = 220 kcal.

[0092] Step 13:

[0093] Server: Evaluates whether the calorie balance is appropriate for the user's diet goal. For example, if the user has set a daily calorie goal of 0 kcal, this 220 kcal is outside the goal.

[0094] Step 14:

[0095] Server: Generates specific advice about the next meal and exercise based on the calorie balance assessment. For example, it provides advice such as "keep your calorie intake at your next meal to 200 kcal or less."

[0096] Step 15:

[0097] Device (smartphone): Provides advice to the user and instructs them to reflect this in their daily diet and exercise.

[0098] This allows users to accurately manage their calorie intake and expenditure and maintain a healthy lifestyle.

[0099] Example 1

[0100] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0101] Currently, many people struggle to balance dietary calorie management and exercise as part of their health management. Conventional methods require users to manually input meal contents and calculate calories themselves, which is time-consuming and prone to errors. In addition, the collection and analysis of exercise data is not centrally managed, often preventing users from receiving appropriate diet and health guidance. Therefore, there is a need for a system that allows users to easily manage their daily calorie intake and expenditure and maintain healthy lifestyle habits.

[0102] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0103] In this invention, the server includes means for analyzing food images to identify the type and amount of food, means for calculating calories based on the identified food, means for calculating calories burned based on exercise data, means for calculating a calorie balance based on calories ingested and calories burned, and means for providing the user with advice on diet and exercise based on the calorie balance. This allows the user to automatically and accurately grasp the calorie intake from food and the calories burned from exercise, and by receiving appropriate advice based on this, it becomes possible to efficiently maintain a balance between health management and calorie balance.

[0104] A "meal image" is image data captured to record the contents of a meal that a user eats.

[0105] A "terminal" is an electronic device used by a user, such as smart glasses, a smart watch, or a smartphone.

[0106] The "server" is an information processing device that analyzes image data and exercise data, calculates calories, and provides advice.

[0107] "Type and amount of food" refers to the specific food name and portion size identified by analyzing the meal image.

[0108] A "calorie" is a unit that represents the amount of energy contained in food.

[0109] "Exercise data" refers to data such as the type of exercise performed by the user, the number of steps taken, heart rate, and exercise time.

[0110] "Calories burned" is the amount of energy consumed through exercise, calculated based on exercise data.

[0111] "Calories ingested" refers to the amount of energy a user takes in through food.

[0112] "Calorie balance" is the number obtained by subtracting calories consumed from calories consumed.

[0113] "Providing advice" refers to giving instructions or advice to the user regarding diet and exercise based on calorie balance.

[0114] A "natural language processing model" is one of the artificial intelligence techniques used to analyze food images and exercise data.

[0115] "Biometric data" refers to the user's physical information (e.g., height, weight, age, gender, etc.).

[0116] This invention relates to a system that combines images of the food a user eats with exercise data to manage their daily calorie intake and expenditure and provide appropriate diet advice. This system enables users to efficiently manage their daily calorie intake and expenditure and aim to maintain a healthy weight.

[0117] The main hardware components of the system include a device for acquiring food images (e.g., smart glasses), a device for acquiring exercise data (e.g., a smart watch), a device for notifying the user of the analysis results (e.g., a smartphone), and a server for performing data analysis.

[0118] First, the user takes an image of the meal using the smart glasses. When the user presses the start button on the smart glasses, the camera captures the image and sends the image data wirelessly to the server. Next, the server uses an image analysis algorithm (e.g., TENSORFLOW (registered trademark)) to analyze the captured image of the meal. This analysis identifies the contents of the meal and the type and amount of each food item. The analysis results are stored in a database, and the server calculates the total calories based on the calorie information of each food item. For example, if bread is 250 kcal, an omelet is 150 kcal, and a salad is 50 kcal, the total calories are calculated as 450 kcal.

[0119] The calculated calorie information is sent from the server to the smartphone and notified to the user. Using the smartphone application, the user can check the calorie intake information and save it as a meal history.

[0120] Furthermore, when a user exercises using a smartwatch, the smartwatch automatically records exercise data such as the number of steps, heart rate, and exercise time. For example, if a user runs for 30 minutes, the data is recorded. The recorded exercise data is sent to a server at regular intervals. Based on the exercise data, the server calculates the calories burned according to the type and intensity of exercise. For example, it calculates that 230 kcal is burned in a 30-minute run. This calorie burn information is also notified from the server to the smartphone, where it is displayed to the user and saved as exercise history.

[0121] Finally, the server calculates the calorie balance based on the calorie intake and calorie expenditure information. For example, if the calorie intake for breakfast is 400 kcal and the calorie expenditure through exercise is 230 kcal, the calorie balance is 400 kcal - 230 kcal = 170 kcal. The server analyzes whether the calorie balance is appropriate for the user's diet goal and generates advice regarding the next meal or exercise as necessary. For example, advice such as "Keep your next meal to 200 kcal or less" is sent from the server to the user's smartphone and notified.

[0122] This specific system allows users to accurately manage their daily calorie intake and expenditure, and receive appropriate advice based on that, enabling them to efficiently manage their health.

[0123] Prompt Sentence Examples

[0124] "I had bread and an omelet for breakfast, so please count the calories."

[0125] "I ran for 30 minutes this morning. How many calories did I burn?"

[0126] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0127] Step 1:

[0128] Acquiring food images

[0129] Device (smart glasses): When the user starts eating, the device presses the start button to take an image of the meal. The image is captured by the smart glasses' camera and saved in the device's internal memory. The image is then sent to the server via wireless communication.

[0130] Input: The action of the user pressing the start button on the device.

[0131] Output: Food image data sent to the server.

[0132] Step 2:

[0133] Food image analysis

[0134] Server: The server processes the received image data using an analysis algorithm (e.g., TensorFlow) to determine the meal contents and portions. This analysis includes the type of food (e.g., bread, omelet, salad) and the portion size of each.

[0135] Input: Food image data sent from the device.

[0136] Output: Data on the type and quantity of food analyzed.

[0137] Step 3:

[0138] Calorie Counting

[0139] Server: Based on the analysis results, the calorie information for each food category is retrieved from a database (e.g., MySQL (registered trademark)) and the total calories are calculated. For example, if bread is 250 kcal, an omelet is 150 kcal, and a salad is 50 kcal, the total is calculated as 450 kcal.

[0140] Input: Data on the type and quantity of food parsed.

[0141] Output: Numeric data showing total calories.

[0142] Step 4:

[0143] Calorie information notification

[0144] Terminal (smartphone): The server sends the calculated total calorie information to the terminal and notifies the user. The smartphone application displays the calorie information and saves it in a database as a meal history.

[0145] Input: Numeric data of total calories sent from the server.

[0146] Output: Calorie notification to user and storage in food history database.

[0147] Step 5:

[0148] Acquisition of exercise data

[0149] Device (smartwatch): When the user starts exercising, the device automatically records exercise data such as the number of steps, heart rate, and exercise time. After the exercise is finished, the data is periodically sent to the server.

[0150] Input: User's athletic activity.

[0151] Output: Exercise data sent to the server.

[0152] Step 6:

[0153] Calorie Calculation

[0154] Server: Based on the received exercise data, calculates the calories burned according to the type and intensity of exercise. For example, it calculates that 230 kcal is burned in 30 minutes of running.

[0155] Input: Submitted exercise data.

[0156] Output: Numerical data showing calories burned.

[0157] Step 7:

[0158] Calorie consumption information notification

[0159] Device (smartphone): The server sends the calculated calorie consumption information to the device and notifies the user. The smartphone application displays the calorie consumption information and saves it in a database as exercise history.

[0160] Input: Numerical data of calories burned sent from the server.

[0161] Output: Notify user of calories burned and save to exercise history database.

[0162] Step 8:

[0163] Calculating calorie balance

[0164] Server: Calculates calorie balance using data on calories in and calories out. For example, if your calorie intake is 400 kcal and your calorie expenditure is 230 kcal, your calorie balance will be 170 kcal.

[0165] Input: Numerical data of calories ingested and calories burned.

[0166] Output: Numerical data showing calorie balance.

[0167] Step 9:

[0168] Providing advice

[0169] Server: Based on the calorie balance, the server generates advice based on the user's diet goals. For example, the server may instruct the user to limit their next meal to 200 kcal or less. The advice is sent from the server to the smartphone.

[0170] Input: Calorie balance data.

[0171] Output: Advice notice to user.

[0172] Step 10:

[0173] User Notification and Data Retention

[0174] Terminal (smartphone): Advice is displayed, and once the user confirms it, the record is saved in the database.

[0175] Input: Advice notification sent by the server.

[0176] Output: Advice notification to the user and record saved in the database.

[0177] (Application example 1)

[0178] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0179] In modern society, individual health management is extremely important, but it is not easy to properly manage daily calorie intake and expenditure and maintain a balanced diet. Furthermore, few systems allow users to easily understand their dietary habits and receive appropriate advice based on that information. Furthermore, there is no integration of food delivery services that allow users to immediately order suggested meals. This leads to people living their daily lives without adequate health management, which increases long-term health risks.

[0180] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0181] In this invention, the server includes a means for acquiring food images, a means for analyzing the food images to identify the type and amount of food, a means for calculating calories based on the identified food, a means for acquiring exercise data, a means for calculating calories burned based on the exercise data, a means for calculating a calorie balance based on calories ingested and calories burned, a means for providing the user with dietary and exercise advice based on the calorie balance, and a means for suggesting and ordering a meal menu suitable for a food delivery service based on the advice. This allows users to manage their own dietary and exercise habits in real time and receive appropriate suggestions for maintaining a healthy lifestyle. Furthermore, the suggested meals can be easily ordered, greatly improving user convenience.

[0182] The "means for acquiring food images" is a device that has the function of taking an image of a meal when the user eats and transmitting the image to the system.

[0183] The "means for analyzing a meal image to identify the type and amount of food" is an algorithm or program for analyzing a captured meal image and identifying the type and amount of food contained in the image.

[0184] The "means for calculating calories based on identified foods" refers to a database or calculation mechanism that has the function of calculating the calories of a food based on the type and amount of the analyzed food.

[0185] "Means for acquiring exercise data" refers to a device that has the function of recording data related to the exercise performed by the user (e.g., number of steps, exercise time, heart rate, etc.) and transmitting that data to the system.

[0186] The "means for calculating calories burned based on exercise data" is an algorithm or program that calculates the calories burned by the user based on the acquired exercise data.

[0187] The "means for calculating the calorie balance based on the calorie intake and the calorie expenditure" is a calculation mechanism having a function of calculating the difference between the calories taken in and the calories consumed, and calculating the calorie balance for that day.

[0188] The "means for providing dietary and exercise advice to a user based on calorie balance" is an algorithm or program for suggesting the next meal or recommending exercise to a user based on calorie balance.

[0189] The "means for proposing a meal menu suitable for a food delivery service and ordering that menu" is a system for proposing a meal menu suitable for a user based on the user's calorie balance and ordering that menu through a food delivery service.

[0190] The present invention relates to a system that combines images of meals consumed by a user with exercise data to manage daily calorie intake and provide appropriate meal menus through a food delivery service.

[0191] System configuration details

[0192] 1. Acquiring food images

[0193] Device (smartphone or smart glasses): Takes pictures of the meal while the user is eating. For example, a user takes a photo of their breakfast with their smartphone and uploads it to the application.

[0194] 2. Analysis of food images

[0195] Server: The image data of the photographed meal is sent to the server, which uses an image analysis algorithm (e.g., a machine learning model such as CNN) to identify the meal contents (e.g., bread, omelet, salad) and the amount of each.

[0196] 3. Calorie Counting

[0197] Server: Retrieves calorie information corresponding to each identified food item from the database and calculates the total calories. For example, if bread is 250 kcal, omelet is 150 kcal, and salad is 50 kcal, the total is calculated as 450 kcal.

[0198] 4. Acquisition of exercise data

[0199] Device (smartwatch): When a user exercises, data such as the number of steps, heart rate, and exercise time is automatically recorded. For example, if a user runs for 30 minutes in the morning, the data is recorded.

[0200] Server: Exercise data is periodically sent to the server and stored.

[0201] 5. Calorie Calculation

[0202] Server: Based on the exercise data sent, the server uses machine learning algorithms to calculate the calories burned. For example, it calculates that 30 minutes of running will burn 230 kcal.

[0203] 6. Calculate calorie balance and provide advice

[0204] Server: Calculates the calorie balance based on the identified calorie intake and expenditure (e.g., intake: 450 kcal - expenditure: 230 kcal = 220 kcal).

[0205] Server: Analyzes whether this calorie balance is appropriate for the user's diet goals and health management standards, and provides advice on reducing or increasing calorie intake at the next meal.

[0206] 7. Food delivery service integration

[0207] Server: Based on the generated advice (e.g., "Keep your next meal under 200 kcal"), it proposes a meal menu suitable for the food delivery service and allows the user to order that menu. In this case, it uses a generative AI model to automatically suggest the optimal menu.

[0208] Specific examples

[0209] A user eats bread and an omelet for breakfast, takes a photo of it with their smartphone, and uploads it to the application. The image is analyzed by the server, and the bread is determined to have 250 kcal and the omelet to have 150 kcal. If the user runs for 30 minutes in the morning, 230 kcal will be burned.

[0210] The server uses this data to calculate calorie balance and suggests ways to reduce calories in your next meal, allowing you to quickly order a salad or snack from a food delivery service.

[0211] Prompt Sentence Examples

[0212] A user has uploaded an image of bread and an omelet that they had for breakfast. Calculate the total calories of this meal and calculate their daily calorie balance using their exercise data as input. Then, suggest their next meal based on their calorie balance and order that menu for delivery.

[0213] This system allows users to manage their calorie balance in real time and receive support to maintain a healthy lifestyle, while also significantly improving user convenience.

[0214] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0215] Step 1:

[0216] Acquiring food images

[0217] Input: An image taken by the user while eating.

[0218] How it works: A user takes a picture of their meal using their smartphone or smart glasses and uploads the image to the application.

[0219] Output: Photographed food image data.

[0220] Step 2:

[0221] Food image analysis

[0222] Input: Food image data sent to the server.

[0223] How it works: The server uses an image analysis algorithm (e.g., a machine learning model such as CNN) to identify the types and amounts of food in the image.

[0224] Output: Data on the type and quantity of food identified.

[0225] Step 3:

[0226] Calorie Counting

[0227] Input: Identified food type and quantity data.

[0228] Operation: Based on this food data, the server retrieves the calorie information for each food from the database and calculates the total calories by adding up the calories for each food.

[0229] Output: Total calorie intake data.

[0230] Step 4:

[0231] Acquisition of exercise data

[0232] Input: User's exercise data (e.g., number of steps, heart rate, exercise duration, etc.).

[0233] How it works: A user uses a device such as a smartwatch to record exercise data and periodically sends that data to a server.

[0234] Output: Recording of exercise data.

[0235] Step 5:

[0236] Calorie Calculation

[0237] Input: Exercise data sent to the server.

[0238] How it works: The server uses machine learning algorithms to calculate calories burned based on your exercise data.

[0239] Output: Calories burned data.

[0240] Step 6:

[0241] Calculates calorie balance and provides advice

[0242] Input: Total calorie intake and calorie expenditure data.

[0243] How it works: The server calculates the calorie balance based on the calories ingested and calories burned, and then uses a generative AI model to create advice about the next meal and exercise based on that balance.

[0244] Output: Calorie balance calculation results and advice data.

[0245] Step 7:

[0246] Proposal and ordering for food delivery services

[0247] Input: Generated advice data.

[0248] Operation: The server suggests an appropriate meal menu based on the user's calorie balance, and provides the user with the ability to order that menu from a food delivery service.

[0249] Output: Suggested meal menu and ordering data.

[0250] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0251] The present invention relates to a system that manages daily calorie intake and expenditure by combining images of the food a user consumes, exercise data, and emotional data of the user, and provides appropriate diet advice. This system includes means for acquiring food images, analyzing images, calculating calories, acquiring exercise data, calculating calorie expenditure, an emotional engine, calculating calorie intake and expenditure, and providing advice.

[0252] System configuration

[0253] 1. Acquiring food images

[0254] Device (smart glasses): When the user eats, it takes pictures of the meal. When the user presses the start button, the photo of the meal is automatically taken.

[0255] 2. Analysis of food images

[0256] Server: The image data of the photographed meal is sent to the server, which uses image analysis algorithms to identify the meal contents (e.g., bread, omelet, salad) and the amount of each.

[0257] 3. Calorie Counting

[0258] Server: Retrieves calorie information corresponding to each identified food category from the database and calculates the total calories. For example, if bread is 250 kcal, omelet is 150 kcal, and salad is 50 kcal, the total will be calculated as 450 kcal.

[0259] Terminal (smartphone): Notifies the user of the calculation results and saves them as a meal history.

[0260] 4. Acquisition of exercise data

[0261] Device (smartwatch): When a user exercises, it automatically records exercise data such as the number of steps, heart rate, and exercise time. For example, if a user runs for 30 minutes, the data will be recorded.

[0262] Server: Exercise data is periodically sent to the server and stored for analysis.

[0263] 5. Calorie Calculation

[0264] Server: Based on the exercise data sent, the server calculates the calories burned using the exercise intensity, duration, and the user's physical data (weight, age, etc.). For example, it calculates that 230 kcal is burned by running for 30 minutes.

[0265] Terminal (smartphone): Notifies the user of the calculation results.

[0266] 6. Introducing the Emotion Engine

[0267] Device (smartphone): Recognizes the user's emotions while eating or exercising. The emotion engine uses the camera and microphone to analyze changes in the user's facial expressions and voice to identify emotions.

[0268] Server: The emotion data obtained from the emotion engine is sent to the server.

[0269] Specific examples

[0270] Acquiring food information and calculating calories

[0271] User: I eat bread and an omelet for breakfast.

[0272] Terminal (smart glasses): Takes a photo of the meal and sends it to the server.

[0273] Server: Performs image analysis to distinguish between bread and omelet. Calculates the calories as 250 kcal for bread and 150 kcal for omelet, for a total of 400 kcal.

[0274] Device (smartphone): Display the total as 400 kcal to the user and save the data.

[0275] Acquiring exercise information and calculating calories burned

[0276] User: Go for a 30-minute run in the morning.

[0277] Device (smartwatch): Records running exercise data and sends it to the server.

[0278] Server: Calculate the calories burned by running and it is 230 kcal.

[0279] Emotion recognition and advice provision

[0280] User: Feels certain emotions (e.g., satisfaction, fatigue, stress) after eating or during exercise.

[0281] Device (smartphone): Recognizes user emotions using a camera and microphone.

[0282] Server: The recognized emotion data is sent to the server.

[0283] Server: Calculates calorie balance based on calorie intake, calorie expenditure, and emotional data, and generates advice. For example, if a user is feeling stressed, it provides advice on diet and exercise to help them relax.

[0284] Calculates calorie balance and provides advice

[0285] Server: Combines the data of calorie intake (400 kcal) and calorie expenditure (230 kcal) to calculate the daily calorie balance. Here, the calorie balance is calculated as 400 kcal - 230 kcal = 170 kcal.

[0286] Server: Evaluates whether the calorie balance is appropriate for the user's diet goal. If necessary, generates specific advice about the next meal or exercise. For example, it provides advice such as "Keep your calorie intake to 200 kcal or less at your next meal."

[0287] Device (smartphone): Provides advice to the user and instructs them to reflect this in their daily diet and exercise.

[0288] This allows users to accurately manage their calorie intake and expenditure and receive appropriate advice based on their emotional state, making the system a powerful support tool for maintaining a healthy lifestyle.

[0289] The processing flow will be explained below.

[0290] Step 1:

[0291] Device (smart glasses): Before the user starts eating, they take a photo of the meal. When the user presses the start button, the smart glasses automatically take a photo of the meal.

[0292] Step 2:

[0293] Terminal (smart glasses): The captured food images are sent to the server. The image data is often compressed before being transferred.

[0294] Step 3:

[0295] Server: Prepares the received food images for analysis, specifically by preprocessing the images (noise removal, resolution adjustment, etc.) to apply image recognition algorithms.

[0296] Step 4:

[0297] Server: Uses image recognition algorithms to identify the type and quantity of food in a meal image. For example, identify if an image contains bread, an omelet, or a salad.

[0298] Step 5:

[0299] Server: Retrieve calorie information for each identified food type from the database. For example, bread is 250 kcal, omelette is 150 kcal, and salad is 50 kcal.

[0300] Step 6:

[0301] Server: Add up the calories from each food item to calculate your total calorie intake. In this example, we'll use bread, omelet, and salad as a total of 450 kcal.

[0302] Step 7:

[0303] Device (smartphone): The calculated total calories of 450 kcal are notified to the user's smartphone and saved in the meal history.

[0304] Step 8:

[0305] Device (smartwatch): Records exercise data in real time when the user exercises. For example, it records the number of steps, heart rate, and exercise time during a 30-minute run.

[0306] Step 9:

[0307] Device (smartwatch): Periodically sends recorded exercise data to the server.

[0308] Step 10:

[0309] Server: Based on the received exercise data, calculates the calories burned using exercise intensity, time, and the user's physical data (weight, age, etc.). For example, it calculates that 230 kcal is burned by running for 30 minutes.

[0310] Step 11:

[0311] Terminal (smartphone): Notifies the user of the calculated calorie consumption of 230 kcal.

[0312] Step 12:

[0313] Server: Combines the data of calorie intake (450 kcal) and calorie expenditure (230 kcal) to calculate the daily calorie balance. Here, the calorie balance is calculated as 450 kcal - 230 kcal = 220 kcal.

[0314] Step 13:

[0315] Device (smartphone): When the user eats or exercises, the emotion engine uses the camera and microphone to recognize the user's emotions.

[0316] Step 14:

[0317] Server: Emotion data obtained from the emotion engine is sent to the server. For example, it is recognized that the user is feeling stressed.

[0318] Step 15:

[0319] Server: Adapts advice content based on calorie balance and emotional data. For example, it provides advice on relaxation-enhancing diets and exercise to a user who is feeling stressed.

[0320] Step 16:

[0321] Device (smartphone): Notifies the user of the adapted advice. For example, display advice such as "Keep your calorie intake to 200 kcal or less at your next meal and do yoga to relax."

[0322] This series of processes allows users to accurately manage their calorie intake and expenditure, and receive appropriate advice based on their emotional state, making it easier to maintain a healthy lifestyle.

[0323] Example 2

[0324] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0325] Conventional calorie balance management systems collect and analyze users' dietary and exercise data, but no system provides advice that takes into account the user's emotional state. Therefore, our goal is to make it easier for users to maintain a healthier lifestyle by providing more personalized advice based on the user's emotional state in addition to dietary and exercise data.

[0326] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0327] In this invention, the server includes means for analyzing meal images to identify the type and amount of food, means for calculating calories based on the identified food, means for calculating calories burned based on exercise data, means for calculating calorie balance based on calories ingested and calories burned, means for recognizing and analyzing the user's emotions, and means for providing appropriate advice to the user based on the emotion data, thereby making it possible to provide detailed and personalized health advice according to the user's emotional state.

[0328] The term "means for acquiring food images" refers to a device or method for taking an image of the food a user eats and recording that image.

[0329] "Means for analyzing meal images to identify food types and quantities" refers to an algorithm or program for analyzing captured meal images and identifying the type and quantity of each food item in the image.

[0330] "Means for calculating calories" refers to a method or program for calculating total calorie intake by adding up the calories of each food based on the type and amount of food analyzed.

[0331] "Means for acquiring exercise data" refers to a device or method for recording data related to the exercise performed by a user (e.g., number of steps, heart rate, exercise time, etc.) and saving that data.

[0332] "Means for calculating calories burned" refers to a program or algorithm for calculating the amount of energy consumed by a user's exercise based on the acquired exercise data.

[0333] "Means for calculating calorie balance" refers to a method or program for calculating daily calorie balance based on calories consumed and calories burned.

[0334] "Means for providing dietary and exercise advice to a user" refers to a device or method that provides specific instructions or suggestions to a user regarding their next meal or exercise based on the calculated calorie balance.

[0335] "Means for recognizing and analyzing user emotions" refers to programs or algorithms that identify a user's emotional state using a camera or microphone and analyze that data.

[0336] "Means for providing appropriate advice to a user based on emotional data" refers to a device or method for providing more personalized health advice based on calorie intake, calorie expenditure, and calorie balance, taking into account the user's emotional state.

[0337] This invention is a system that manages daily calorie balance and provides appropriate diet advice by combining images of the food a user consumes, exercise data, and the user's emotional data. This system includes means for acquiring food images, analyzing images, calculating calories, acquiring exercise data, calculating calories burned, an emotional engine, calculating calorie balance, and providing advice.

[0338] Specifically, when a user starts eating, they use the device (smart glasses) to take a photo of their meal. When the user presses the start button, the smart glasses take two to three photos in succession and record the images. The captured image data is then temporarily processed using edge computing, compressed, and noise-removed before being sent to the server.

[0339] The server analyzes the received food image data. During this process, an image analysis algorithm (e.g., a deep learning model using TensorFlow) runs to identify the meal contents (bread, omelet, salad, etc.) and the amount of each. The results of the identification are saved along with the identification confidence, and the one with the highest confidence is adopted.

[0340] Next, the server retrieves calorie information corresponding to each identified food category from a database (for example, a dietary database such as the USDA or the Japanese Food Composition Tables). Based on the analyzed foods, the calories of each food are added up to calculate the total calorie intake. The calculation result is notified to the device (smartphone), and a message is displayed to the user saying, "The total calories for breakfast are 450 kcal." This data is also saved on the device as a meal history.

[0341] When a user exercises, the device (smartwatch) automatically records exercise data such as the number of steps, heart rate, and exercise time. The smartwatch's sensors collect data in real time and send it to a server every 10 minutes. The server stores the received exercise data for analysis and classifies it by exercise type. For example, this includes exercise data for running, walking, cycling, etc.

[0342] The server calculates the calories burned based on the received exercise data. It calculates the MET (Metabolic Equivalent of Task) value based on the type, intensity, and duration of exercise, as well as the user's physical data (weight, age). For example, it calculates that 230 kcal will be burned for a 30-minute run using a pre-set MET value (8 METs). The calculation result is sent to the device (smartphone), and a message is displayed saying, "You burned 230 kcal running."

[0343] Furthermore, when the user eats or exercises, the device (smartphone) uses a camera and microphone to recognize the user's emotions. Facial recognition software (e.g., Microsoft® Face API or Google® Cloud Vision) and voice analysis software are used to identify emotions. The server receives the recognized emotion data and stores it as time-series data. If a specific emotion (such as satisfaction, fatigue, or stress) is recognized frequently, an alert is generated.

[0344] Finally, the server calculates the daily calorie balance based on the calorie intake, calorie expenditure, and emotional data. If the calorie intake is 450 kcal and the calorie expenditure is 230 kcal, the calorie balance is calculated as (450 kcal - 230 kcal = 220 kcal). Based on this calorie balance, advice is generated according to the user's diet goals and appropriate advice is provided to the user. For example, a specific prompt such as "Aim for 200 kcal or less at your next meal" is generated.

[0345] Examples of prompts for using the system include:

[0346] "Calculate calorie balance and generate appropriate diet advice based on today's calorie intake and expenditure, as well as the user's emotional data."

[0347] This allows users to accurately manage their calorie intake and expenditure and receive personalized advice based on their emotional state, making the system a powerful support tool for maintaining a healthy lifestyle.

[0348] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0349] Step 1:

[0350] Input: A picture of the meal when the user starts eating

[0351] Specific operation: When a user starts eating, they use the device (smart glasses) to take a photo of their meal. When the user presses the start button, the smart glasses take two to three photos in succession. The captured image data is temporarily processed using edge computing, where it is compressed and noise-removed.

[0352] Output: The processed meal image data is sent to the server.

[0353] Step 2:

[0354] Input: Food image data sent to the server

[0355] Specific operation: The server analyzes the received food image data. An image analysis algorithm (e.g., a deep learning model using TensorFlow) runs to identify the meal contents (bread, omelet, salad, etc.) and the amount of each. The identification results are saved along with the identification confidence score, and the one with the highest confidence score is adopted.

[0356] Output: Identified food type and quantity data

[0357] Step 3:

[0358] Input: Identified food type and quantity

[0359] Specific operation: The server retrieves calorie information corresponding to each identified food category from a database (e.g., a dietary database such as the USDA or the Japanese Food Composition Table). Based on the analyzed foods, the server calculates the total calorie intake by adding up the calories of each food.

[0360] Output: Total calorie intake data

[0361] Step 4:

[0362] Input: Total calorie intake data

[0363] Specific operation: The server notifies the device (smartphone) of the calculation results. The device displays a message to the user saying, "The total calories for breakfast is 450 kcal." This data is also saved on the device as a meal history.

[0364] Output: Total calorie intake information notified to the user

[0365] Step 5:

[0366] Input: Exercise data when the user exercises (number of steps, heart rate, exercise time, etc.)

[0367] How it works: When a user exercises, the device (smartwatch) automatically records exercise data. The smartwatch's sensors collect data in real time and send it to the server every 10 minutes.

[0368] Output: Exercise data sent to the server

[0369] Step 6:

[0370] Input: Exercise data sent to the server

[0371] Specific operation: The server calculates the calorie consumption based on the received exercise data. It calculates the MET (Metabolic Equivalent of Task) value based on the type, intensity, and duration of exercise, as well as the user's physical data (weight, age). For example, it calculates that 230 kcal will be consumed for a 30-minute run using a pre-set MET value (8 METs).

[0372] Output: Calculated calorie consumption data

[0373] Step 7:

[0374] Input: Calorie consumption data

[0375] Specific operation: The server notifies the terminal (smartphone) of the calculation result. The terminal displays a message to the user saying, "You burned 230 kcal by running."

[0376] Output: Calorie consumption information notified to the user

[0377] Step 8:

[0378] Input: Data showing the user's emotional state (facial expression images and voice data)

[0379] Specific operation: When the user eats or exercises, the device (smartphone) uses a camera and microphone to recognize the user's emotions, using facial recognition software and voice analysis software to identify emotions (satisfaction, fatigue, stress, etc.).

[0380] Output: Recognized emotion data

[0381] Step 9:

[0382] Input: Recognized emotion data

[0383] Specific operation: The server receives the recognized emotion data and generates appropriate advice based on that data. For example, if the user is feeling stressed, advice such as "do some light exercise to relax" or "eat a low-calorie diet" will be provided.

[0384] Output: Advice generated based on emotional state

[0385] Step 10:

[0386] Input: Calories ingested, calories burned, emotional data

[0387] Specific operation: The server calculates the daily calorie balance based on calorie intake, calorie expenditure, and emotional data. Specifically, if calorie intake is 450 kcal and calorie expenditure is 230 kcal, the server calculates the calorie balance (450 kcal - 230 kcal = 220 kcal). As a result, it generates advice based on the diet goal.

[0388] Output: Calorie balance and diet advice data

[0389] In this way, each processing step of this system performs specific data processing and calculation based on input data, and supports health management by providing the results to the user.

[0390] (Application example 2)

[0391] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0392] Many people today lead busy lifestyles, making it difficult to manage their health and diet. Therefore, it is necessary to provide appropriate advice based on the user's emotional state while managing calorie intake through diet and exercise. However, current systems have the challenge of not only managing dietary and exercise data comprehensively, but also incorporating emotional data to provide optimal advice to users.

[0393] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0394] In this invention, the server includes means for acquiring food images, means for analyzing the food images to identify the types and amounts of food, means for calculating calories based on the identified foods, means for acquiring exercise data, means for calculating calories burned based on the exercise data, means for calculating a calorie balance based on calories ingested and calories burned, means for acquiring user emotion data, means for evaluating the user's condition based on the emotion data, means for providing the user with advice regarding diet and exercise based on the calorie balance and emotion data, and means for notifying the user of the advice. This makes it possible to accurately provide appropriate advice based on the user's emotional state while managing calorie intake and calorie expenditure through exercise.

[0395] The "means for acquiring food images" refers to a device that has the function of taking an image of the food that the user eats and saving it as image data or transmitting it to a server.

[0396] The "means for analyzing the food image to identify the type and amount of food" is software or hardware that includes an algorithm for analyzing the captured food image and identifying the type and amount of food in the image.

[0397] A "calorie calculation means" is a means for calculating the calories of an entire meal based on the types and amounts of food identified and using calorie information stored in a database.

[0398] A "means for acquiring exercise data" is a device that records data about the exercise performed by the user (e.g., number of steps, exercise time, heart rate, etc.) and stores or transmits the data to a server.

[0399] The "means for calculating calories burned" is a means for calculating calories burned through exercise by the user based on the acquired exercise data.

[0400] A "means for calculating calorie balance" is a means for calculating daily calorie balance based on calories ingested and calories burned through exercise.

[0401] The "means for acquiring user emotional data" refers to a means for recognizing the emotional state of the user from their facial expressions and voice using sensors such as a camera and a microphone.

[0402] The "means for assessing the user's state" is software or hardware that includes an algorithm for analyzing the acquired emotional data and assessing the user's emotional state.

[0403] The "means for providing diet and exercise advice" is a means for generating and providing optimal diet and exercise advice to a user based on calorie balance and emotion data.

[0404] The "means for notifying the user of the advice" refers to a device or software having an interface for notifying the user of the generated advice.

[0405] The purpose of this invention is to build a system that not only manages the calorie intake of a user and the calorie consumption through exercise, but also provides more appropriate diet advice by utilizing the user's emotional data. This system is configured using the following hardware and software.

[0406] Hardware Configuration

[0407] Smartphone: Obtains the user's emotional data and provides advice. It also displays the total calories of the meal and the calorie balance.

[0408] Smart glasses: When the user eats a meal, they take a picture of the meal and send the image data to a server.

[0409] Smartwatch: Records exercise data (e.g., number of steps, heart rate, and exercise time) when the user exercises and sends it to a server.

[0410] Server: Analyzes food images, calculates calories, calculates calorie expenditure, analyzes emotional data, and generates advice.

[0411] Software Configuration

[0412] Image analysis algorithm: Runs on the server and analyzes the food images sent from the smart glasses to identify the type and quantity of food.

[0413] Calorie calculation engine: Based on the image analysis results, calculates the calories of each food item and calculates the total calories.

[0414] Exercise data analysis engine: Calculates calories burned based on exercise data sent from the smartwatch.

[0415] Emotion recognition engine: Uses the smartphone's camera and microphone to recognize the user's emotional state.

[0416] Advice generation engine: Integrates total calorie intake, calorie expenditure, and emotional data to generate dietary and exercise advice tailored to the user's condition.

[0417] System Operation

[0418] 1. Acquisition and analysis of meal images

[0419] As a user wearing the smart glasses eats, the glasses take images of the meal and send the image data to a server, which uses image analysis algorithms to identify the meal contents and determine the type and quantity of each food item.

[0420] 2. Calorie Counting

[0421] The server calculates the calorie count of each food item based on the type and amount of food identified, and then calculates the total calorie count using a calorie calculation engine. The calculation results are sent to the smartphone and notified to the user.

[0422] 3. Acquisition of exercise data and calculation of calories burned

[0423] When a user wears a smartwatch and exercises, the exercise data is recorded on the smartwatch and periodically sent to a server. The server uses an exercise data analysis engine to calculate the calories burned through exercise and sends the results to a smartphone.

[0424] 4. Acquisition and analysis of emotion data

[0425] Smartphones use cameras and microphones to recognize the user's emotional state. An emotion recognition engine analyzes this and generates emotional data. The generated emotional data is sent to a server.

[0426] 5. Advice Generation

[0427] The server integrates the total calorie intake, calorie expenditure, and emotion data, and generates appropriate diet and exercise advice for the user using an advice generation engine. The generated advice is sent to the user's smartphone and notified.

[0428] Specific examples

[0429] When a user orders lunch through a food delivery app, the following process occurs:

[0430] Prompt Sentence Examples

[0431] I ordered ramen for lunch today. Please check the calorie information and provide appropriate advice based on my exercise data and calorie intake. My current feeling is recognized as "a little tired."

[0432] Specifically, the process is as follows: When a user orders ramen, the app displays the ramen's calorie information (for example, 700 kcal), and calculates the current calorie balance, taking into account the 200 kcal consumed in the previous exercise data. Emotion recognition determines that the user is "slightly tired," and since the calories consumed are sufficient, advice is provided such as "It would be a good idea to order a low-calorie side dish or a protein drink at the same time."

[0433] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0434] Step 1:

[0435] Acquiring food images

[0436] When a user wearing smart glasses eats a meal, the smart glasses take an image of the meal and send the image data to a server. The input is an image of the meal, and the output is the image data of the meal stored on the server. Specifically, when the user starts eating, the camera in the smart glasses automatically activates and takes an image.

[0437] Step 2:

[0438] Food image analysis

[0439] The server uses an image analysis algorithm to analyze the food image data sent from the smart glasses and identify the type and quantity of food. The input is the food image data, and the output is the identified food type and quantity data. Specifically, the server identifies the foods in the image and labels each food category.

[0440] Step 3:

[0441] Calorie Counting

[0442] The server uses a calorie calculation engine to calculate the calories of each food based on the type and amount of the identified food, and then calculates the total calories. The input is data on the type and amount of food, and the output is the total calorie value. Specifically, the server retrieves the calorie information of each food from the database and calculates the total calories based on that information.

[0443] Step 4:

[0444] Calorie information notification

[0445] The server sends the calculated total calorie information to the smartphone and notifies the user. The input is the total calorie value, and the output is the calorie information displayed on the smartphone. Specifically, the smartphone receives the notification and displays the calorie information on the user interface.

[0446] Step 5:

[0447] Acquisition of exercise data

[0448] When a user wears a smartwatch and exercises, the smartwatch records exercise data (e.g., number of steps, heart rate, and exercise time) and periodically transmits it to a server. The input is raw exercise data, and the output is exercise data stored on the server. Specifically, the smartwatch monitors the user's exercise in real time and records and transmits the data.

[0449] Step 6:

[0450] Calorie Calculation

[0451] The server uses an exercise data analysis engine to calculate calories burned based on the exercise data sent from the smartwatch. The input is exercise data, and the output is the calorie burned value. Specifically, the server calculates calories burned based on the exercise intensity, duration, and the user's physical data.

[0452] Step 7:

[0453] Acquiring emotion data

[0454] Smartphones use cameras and microphones to recognize the user's emotional state. The input is raw emotional data (images and audio), and the output is recognized emotional data. Specifically, the smartphone analyzes facial expressions and tone of voice to identify and record the user's emotional state.

[0455] Step 8:

[0456] Emotional Data Analysis

[0457] The server uses an emotion recognition engine to evaluate the user's emotional state based on the emotion data sent from the smartphone. The input is the emotion data, and the output is the evaluated emotional state. Specifically, the server analyzes the emotion data and labels the user's emotional state.

[0458] Step 9:

[0459] Advice Generation

[0460] The server uses an advice generation engine to generate appropriate dietary and exercise advice for the user based on the total calorie intake, calorie expenditure, and emotional data. The input is the calorie intake, calorie expenditure, and emotional data, and the output is the generated advice. Specifically, the server integrates this data and generates advice that is optimal for the user's situation.

[0461] Step 10:

[0462] Advice Notification

[0463] The server sends the generated advice to the smartphone and notifies the user. The input is the generated advice, and the output is the advice displayed on the smartphone. Specifically, the smartphone receives the notification and displays the advice on its user interface.

[0464] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0465] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0466] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0467] [Second embodiment]

[0468] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0469] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0470] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0471] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0472] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0473] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0474] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0475] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0476] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0477] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0478] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0479] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0480] The present invention relates to a system that combines images of food consumed by a user with exercise data to manage the daily calorie balance and provide appropriate diet advice. This system includes means for acquiring food images, analyzing images, calculating calories, acquiring exercise data, calculating calories burned, calculating calorie balance, and providing advice.

[0481] System configuration

[0482] 1. Acquiring food images

[0483] Device (smart glasses): When the user eats, it takes pictures of the meal. When the user presses the start button, the photo of the meal is automatically taken.

[0484] 2. Analysis of food images

[0485] Server: The image data of the photographed meal is sent to the server, which uses image analysis algorithms to identify the meal contents (e.g., bread, omelet, salad) and the amount of each.

[0486] 3. Calorie Counting

[0487] Server: Retrieves calorie information corresponding to each identified food category from the database and calculates the total calories. For example, if bread is 250 kcal, omelet is 150 kcal, and salad is 50 kcal, the total will be calculated as 450 kcal.

[0488] Terminal (smartphone): Notifies the user of the calculation results and saves them as a meal history.

[0489] 4. Acquisition of exercise data

[0490] Device (smartwatch): When a user exercises, it automatically records exercise data such as the number of steps, heart rate, and exercise time. For example, if a user runs for 30 minutes, the data will be recorded.

[0491] Server: Exercise data is periodically sent to the server and stored for analysis.

[0492] 5. Calorie Calculation

[0493] Server: Calculates the calories burned based on the exercise data sent. For example, it calculates that 230 kcal is burned by running for 30 minutes.

[0494] Terminal (smartphone): Notifies the user of the calculation results.

[0495] Specific examples

[0496] Acquiring food information and calculating calories

[0497] User: I eat bread and an omelet for breakfast.

[0498] Terminal (smart glasses): Takes a photo of the meal and sends it to the server.

[0499] Server: Performs image analysis to distinguish between bread and omelet. Calculates the calories as 250 kcal for bread and 150 kcal for omelet, for a total of 400 kcal.

[0500] Device (smartphone): Display the total as 400 kcal to the user and save the data.

[0501] Acquiring exercise information and calculating calories burned

[0502] User: Go for a 30-minute run in the morning.

[0503] Device (smartwatch): Records running exercise data and sends it to the server.

[0504] Server: Calculate the calories burned by running and it is 230 kcal.

[0505] Calculates calorie balance and provides advice

[0506] Server: Based on the calories ingested (400 kcal from breakfast) and calories expended (230 kcal from running), calculate your calorie balance (400 kcal - 230 kcal = 170 kcal).

[0507] Server: Analyzes whether the calorie balance is appropriate for the user's diet goals. If necessary, provides guidance on reducing calorie intake at the next meal.

[0508] Device (smartphone): Display the advice to the user, "Keep your next meal under 200 kcal."

[0509] In this way, users can accurately manage their daily calorie intake and expenditure and receive appropriate advice on maintaining a healthy weight. This system is expected to dramatically improve the success rate of dieting and health management.

[0510] The processing flow will be explained below.

[0511] Step 1:

[0512] Device (smart glasses): Before the user starts eating, they take a photo of their meal. When the user presses a button, the smart glasses automatically take a photo of the meal.

[0513] Step 2:

[0514] Terminal (smart glasses): The captured food images are sent to the server. The image data is often compressed before being transferred.

[0515] Step 3:

[0516] Server: Prepares the received food images for analysis, specifically by pre-processing the images to apply image recognition algorithms.

[0517] Step 4:

[0518] Server: Uses image recognition algorithms to identify the type and quantity of food in a meal image. For example, identify if an image contains bread, an omelet, or a salad.

[0519] Step 5:

[0520] Server: Retrieve calorie information for each identified food type from the database. For example, bread is 250 kcal, omelette is 150 kcal, and salad is 50 kcal.

[0521] Step 6:

[0522] Server: Add up the calories from each food item to calculate your total calorie intake. In this example, we'll use bread, omelet, and salad as a total of 450 kcal.

[0523] Step 7:

[0524] Device (smartphone): The calculated total calories of 450 kcal are notified to the user's smartphone and saved in the meal history.

[0525] Step 8:

[0526] Device (smartwatch): Records exercise data in real time when the user exercises. For example, it records the number of steps, heart rate, and exercise time during a 30-minute run.

[0527] Step 9:

[0528] Device (smartwatch): Periodically sends recorded exercise data to the server.

[0529] Step 10:

[0530] Server: Based on the received exercise data, calculates the calories burned using exercise intensity, time, and the user's physical data (weight, age, etc.). For example, it calculates that 230 kcal is burned by running for 30 minutes.

[0531] Step 11:

[0532] Terminal (smartphone): Notifies the user of the calculated calorie consumption of 230 kcal.

[0533] Step 12:

[0534] Server: Combines the data of calorie intake (450 kcal) and calorie expenditure (230 kcal) to calculate the daily calorie balance. Here, the calorie balance is calculated as 450 kcal - 230 kcal = 220 kcal.

[0535] Step 13:

[0536] Server: Evaluates whether the calorie balance is appropriate for the user's diet goal. For example, if the user has set a daily calorie goal of 0 kcal, this 220 kcal is outside the goal.

[0537] Step 14:

[0538] Server: Generates specific advice about the next meal and exercise based on the calorie balance assessment. For example, it provides advice such as "keep your calorie intake at your next meal to 200 kcal or less."

[0539] Step 15:

[0540] Device (smartphone): Provides advice to the user and instructs them to reflect this in their daily diet and exercise.

[0541] This allows users to accurately manage their calorie intake and expenditure and maintain a healthy lifestyle.

[0542] Example 1

[0543] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0544] Currently, many people struggle to balance dietary calorie management and exercise as part of their health management. Conventional methods require users to manually input meal contents and calculate calories themselves, which is time-consuming and prone to errors. In addition, the collection and analysis of exercise data is not centrally managed, often preventing users from receiving appropriate diet and health guidance. Therefore, there is a need for a system that allows users to easily manage their daily calorie intake and expenditure and maintain healthy lifestyle habits.

[0545] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0546] In this invention, the server includes means for analyzing food images to identify the type and amount of food, means for calculating calories based on the identified food, means for calculating calories burned based on exercise data, means for calculating a calorie balance based on calories ingested and calories burned, and means for providing the user with advice on diet and exercise based on the calorie balance. This allows the user to automatically and accurately grasp the calorie intake from food and the calories burned from exercise, and by receiving appropriate advice based on this, it becomes possible to efficiently maintain a balance between health management and calorie balance.

[0547] A "meal image" is image data captured to record the contents of a meal that a user eats.

[0548] A "terminal" is an electronic device used by a user, such as smart glasses, a smart watch, or a smartphone.

[0549] The "server" is an information processing device that analyzes image data and exercise data, calculates calories, and provides advice.

[0550] "Type and amount of food" refers to the specific food name and portion size identified by analyzing the meal image.

[0551] A "calorie" is a unit that represents the amount of energy contained in food.

[0552] "Exercise data" refers to data such as the type of exercise performed by the user, the number of steps taken, heart rate, and exercise time.

[0553] "Calories burned" is the amount of energy consumed through exercise, calculated based on exercise data.

[0554] "Calories ingested" refers to the amount of energy a user takes in through food.

[0555] "Calorie balance" is the number obtained by subtracting calories consumed from calories consumed.

[0556] "Providing advice" refers to giving instructions or advice to the user regarding diet and exercise based on calorie balance.

[0557] A "natural language processing model" is one of the artificial intelligence techniques used to analyze food images and exercise data.

[0558] "Biometric data" refers to the user's physical information (e.g., height, weight, age, gender, etc.).

[0559] This invention relates to a system that combines images of the food a user eats with exercise data to manage their daily calorie intake and expenditure and provide appropriate diet advice. This system enables users to efficiently manage their daily calorie intake and expenditure and aim to maintain a healthy weight.

[0560] The main hardware components of the system include a device for acquiring food images (e.g., smart glasses), a device for acquiring exercise data (e.g., a smart watch), a device for notifying the user of the analysis results (e.g., a smartphone), and a server for performing data analysis.

[0561] First, the user takes an image of their meal using the smart glasses. When the user presses the start button on the smart glasses, the camera captures the image and sends the image data wirelessly to the server. Next, the server uses an image analysis algorithm (e.g., TensorFlow) to analyze the captured image of the meal. This analysis identifies the contents of the meal and the type and amount of each food item. The analysis results are stored in a database, and the server calculates the total calories based on the calorie information for each food item. For example, if bread is 250 kcal, an omelet is 150 kcal, and a salad is 50 kcal, the total will be calculated as 450 kcal.

[0562] The calculated calorie information is sent from the server to the smartphone and notified to the user. Using the smartphone application, the user can check the calorie intake information and save it as a meal history.

[0563] Furthermore, when a user exercises using a smartwatch, the smartwatch automatically records exercise data such as the number of steps, heart rate, and exercise time. For example, if a user runs for 30 minutes, the data is recorded. The recorded exercise data is sent to a server at regular intervals. Based on the exercise data, the server calculates the calories burned according to the type and intensity of exercise. For example, it calculates that 230 kcal is burned in a 30-minute run. This calorie burn information is also notified from the server to the smartphone, where it is displayed to the user and saved as exercise history.

[0564] Finally, the server calculates the calorie balance based on the calorie intake and calorie expenditure information. For example, if the calorie intake for breakfast is 400 kcal and the calorie expenditure through exercise is 230 kcal, the calorie balance is 400 kcal - 230 kcal = 170 kcal. The server analyzes whether the calorie balance is appropriate for the user's diet goal and generates advice regarding the next meal or exercise as necessary. For example, advice such as "Keep your next meal to 200 kcal or less" is sent from the server to the user's smartphone and notified.

[0565] This specific system allows users to accurately manage their daily calorie intake and expenditure, and receive appropriate advice based on that, enabling them to efficiently manage their health.

[0566] Prompt Sentence Examples

[0567] "I had bread and an omelet for breakfast, so please count the calories."

[0568] "I ran for 30 minutes this morning. How many calories did I burn?"

[0569] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0570] Step 1:

[0571] Acquiring food images

[0572] Device (smart glasses): When the user starts eating, the device presses the start button to take an image of the meal. The image is captured by the smart glasses' camera and saved in the device's internal memory. The image is then sent to the server via wireless communication.

[0573] Input: The action of the user pressing the start button on the device.

[0574] Output: Food image data sent to the server.

[0575] Step 2:

[0576] Food image analysis

[0577] Server: The server processes the received image data using an analysis algorithm (e.g., TensorFlow) to determine the meal contents and portions. This analysis includes the type of food (e.g., bread, omelet, salad) and the portion size of each.

[0578] Input: Food image data sent from the device.

[0579] Output: Data on the type and quantity of food analyzed.

[0580] Step 3:

[0581] Calorie Counting

[0582] Server: Based on the analysis results, the calorie information for each food category is retrieved from a database (e.g., MySQL) and the total calories are calculated. For example, if bread is 250 kcal, an omelet is 150 kcal, and a salad is 50 kcal, the total is calculated as 450 kcal.

[0583] Input: Data on the type and quantity of food parsed.

[0584] Output: Numeric data showing total calories.

[0585] Step 4:

[0586] Calorie information notification

[0587] Terminal (smartphone): The server sends the calculated total calorie information to the terminal and notifies the user. The smartphone application displays the calorie information and saves it in a database as a meal history.

[0588] Input: Numeric data of total calories sent from the server.

[0589] Output: Calorie notification to user and storage in food history database.

[0590] Step 5:

[0591] Acquisition of exercise data

[0592] Device (smartwatch): When the user starts exercising, the device automatically records exercise data such as the number of steps, heart rate, and exercise time. After the exercise is finished, the data is periodically sent to the server.

[0593] Input: User's athletic activity.

[0594] Output: Exercise data sent to the server.

[0595] Step 6:

[0596] Calorie Calculation

[0597] Server: Based on the received exercise data, calculates the calories burned according to the type and intensity of exercise. For example, it calculates that 230 kcal is burned in 30 minutes of running.

[0598] Input: Submitted exercise data.

[0599] Output: Numerical data showing calories burned.

[0600] Step 7:

[0601] Calorie consumption information notification

[0602] Device (smartphone): The server sends the calculated calorie consumption information to the device and notifies the user. The smartphone application displays the calorie consumption information and saves it in a database as exercise history.

[0603] Input: Numerical data of calories burned sent from the server.

[0604] Output: Notify user of calories burned and save to exercise history database.

[0605] Step 8:

[0606] Calculating calorie balance

[0607] Server: Calculates calorie balance using data on calories in and calories out. For example, if your calorie intake is 400 kcal and your calorie expenditure is 230 kcal, your calorie balance will be 170 kcal.

[0608] Input: Numerical data of calories ingested and calories burned.

[0609] Output: Numerical data showing calorie balance.

[0610] Step 9:

[0611] Providing advice

[0612] Server: Based on the calorie balance, the server generates advice based on the user's diet goals. For example, the server may instruct the user to limit their next meal to 200 kcal or less. The advice is sent from the server to the smartphone.

[0613] Input: Calorie balance data.

[0614] Output: Advice notice to user.

[0615] Step 10:

[0616] User Notification and Data Retention

[0617] Terminal (smartphone): Advice is displayed, and once the user confirms it, the record is saved in the database.

[0618] Input: Advice notification sent by the server.

[0619] Output: Advice notification to the user and record saved in the database.

[0620] (Application example 1)

[0621] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0622] In modern society, individual health management is extremely important, but it is not easy to properly manage daily calorie intake and expenditure and maintain a balanced diet. Furthermore, few systems allow users to easily understand their dietary habits and receive appropriate advice based on that information. Furthermore, there is no integration of food delivery services that allow users to immediately order suggested meals. This leads to people living their daily lives without adequate health management, which increases long-term health risks.

[0623] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0624] In this invention, the server includes a means for acquiring food images, a means for analyzing the food images to identify the type and amount of food, a means for calculating calories based on the identified food, a means for acquiring exercise data, a means for calculating calories burned based on the exercise data, a means for calculating a calorie balance based on calories ingested and calories burned, a means for providing the user with dietary and exercise advice based on the calorie balance, and a means for suggesting and ordering a meal menu suitable for a food delivery service based on the advice. This allows users to manage their own dietary and exercise habits in real time and receive appropriate suggestions for maintaining a healthy lifestyle. Furthermore, the suggested meals can be easily ordered, greatly improving user convenience.

[0625] The "means for acquiring food images" is a device that has the function of taking an image of a meal when the user eats and transmitting the image to the system.

[0626] The "means for analyzing a meal image to identify the type and amount of food" is an algorithm or program for analyzing a captured meal image and identifying the type and amount of food contained in the image.

[0627] The "means for calculating calories based on identified foods" refers to a database or calculation mechanism that has the function of calculating the calories of a food based on the type and amount of the analyzed food.

[0628] "Means for acquiring exercise data" refers to a device that has the function of recording data related to the exercise performed by the user (e.g., number of steps, exercise time, heart rate, etc.) and transmitting that data to the system.

[0629] The "means for calculating calories burned based on exercise data" is an algorithm or program that calculates the calories burned by the user based on the acquired exercise data.

[0630] The "means for calculating the calorie balance based on the calorie intake and the calorie expenditure" is a calculation mechanism having a function of calculating the difference between the calories taken in and the calories consumed, and calculating the calorie balance for that day.

[0631] The "means for providing dietary and exercise advice to a user based on calorie balance" is an algorithm or program for suggesting the next meal or recommending exercise to a user based on calorie balance.

[0632] The "means for proposing a meal menu suitable for a food delivery service and ordering that menu" is a system for proposing a meal menu suitable for a user based on the user's calorie balance and ordering that menu through a food delivery service.

[0633] The present invention relates to a system that combines images of meals consumed by a user with exercise data to manage daily calorie intake and provide appropriate meal menus through a food delivery service.

[0634] System configuration details

[0635] 1. Acquiring food images

[0636] Device (smartphone or smart glasses): Takes pictures of the meal while the user is eating. For example, a user takes a photo of their breakfast with their smartphone and uploads it to the application.

[0637] 2. Analysis of food images

[0638] Server: The image data of the photographed meal is sent to the server, which uses an image analysis algorithm (e.g., a machine learning model such as CNN) to identify the meal contents (e.g., bread, omelet, salad) and the amount of each.

[0639] 3. Calorie Counting

[0640] Server: Retrieves calorie information corresponding to each identified food item from the database and calculates the total calories. For example, if bread is 250 kcal, omelet is 150 kcal, and salad is 50 kcal, the total is calculated as 450 kcal.

[0641] 4. Acquisition of exercise data

[0642] Device (smartwatch): When a user exercises, data such as the number of steps, heart rate, and exercise time is automatically recorded. For example, if a user runs for 30 minutes in the morning, the data is recorded.

[0643] Server: Exercise data is periodically sent to the server and stored.

[0644] 5. Calorie Calculation

[0645] Server: Based on the exercise data sent, the server uses machine learning algorithms to calculate the calories burned. For example, it calculates that 30 minutes of running will burn 230 kcal.

[0646] 6. Calculate calorie balance and provide advice

[0647] Server: Calculates the calorie balance based on the identified calorie intake and expenditure (e.g., intake: 450 kcal - expenditure: 230 kcal = 220 kcal).

[0648] Server: Analyzes whether this calorie balance is appropriate for the user's diet goals and health management standards, and provides advice on reducing or increasing calorie intake at the next meal.

[0649] 7. Food delivery service integration

[0650] Server: Based on the generated advice (e.g., "Keep your next meal under 200 kcal"), it proposes a meal menu suitable for the food delivery service and allows the user to order that menu. In this case, it uses a generative AI model to automatically suggest the optimal menu.

[0651] Specific examples

[0652] A user eats bread and an omelet for breakfast, takes a photo of it with their smartphone, and uploads it to the application. The image is analyzed by the server, and the bread is determined to have 250 kcal and the omelet to have 150 kcal. If the user runs for 30 minutes in the morning, 230 kcal will be burned.

[0653] The server uses this data to calculate calorie balance and suggests ways to reduce calories in your next meal, allowing you to quickly order a salad or snack from a food delivery service.

[0654] Prompt Sentence Examples

[0655] A user has uploaded an image of bread and an omelet that they had for breakfast. Calculate the total calories of this meal and calculate their daily calorie balance using their exercise data as input. Then, suggest their next meal based on their calorie balance and order that menu for delivery.

[0656] This system allows users to manage their calorie balance in real time and receive support to maintain a healthy lifestyle, while also significantly improving user convenience.

[0657] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0658] Step 1:

[0659] Acquiring food images

[0660] Input: An image taken by the user while eating.

[0661] How it works: A user takes a picture of their meal using their smartphone or smart glasses and uploads the image to the application.

[0662] Output: Photographed food image data.

[0663] Step 2:

[0664] Food image analysis

[0665] Input: Food image data sent to the server.

[0666] How it works: The server uses an image analysis algorithm (e.g., a machine learning model such as CNN) to identify the types and amounts of food in the image.

[0667] Output: Data on the type and quantity of food identified.

[0668] Step 3:

[0669] Calorie Counting

[0670] Input: Identified food type and quantity data.

[0671] Operation: Based on this food data, the server retrieves the calorie information for each food from the database and calculates the total calories by adding up the calories for each food.

[0672] Output: Total calorie intake data.

[0673] Step 4:

[0674] Acquisition of exercise data

[0675] Input: User's exercise data (e.g., number of steps, heart rate, exercise duration, etc.).

[0676] How it works: A user uses a device such as a smartwatch to record exercise data and periodically sends that data to a server.

[0677] Output: Recording of exercise data.

[0678] Step 5:

[0679] Calorie Calculation

[0680] Input: Exercise data sent to the server.

[0681] How it works: The server uses machine learning algorithms to calculate calories burned based on your exercise data.

[0682] Output: Calories burned data.

[0683] Step 6:

[0684] Calculates calorie balance and provides advice

[0685] Input: Total calorie intake and calorie expenditure data.

[0686] How it works: The server calculates the calorie balance based on the calories ingested and calories burned, and then uses a generative AI model to create advice about the next meal and exercise based on that balance.

[0687] Output: Calorie balance calculation results and advice data.

[0688] Step 7:

[0689] Proposal and ordering for food delivery services

[0690] Input: Generated advice data.

[0691] Operation: The server suggests an appropriate meal menu based on the user's calorie balance, and provides the user with the ability to order that menu from a food delivery service.

[0692] Output: Suggested meal menu and ordering data.

[0693] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0694] The present invention relates to a system that manages daily calorie intake and expenditure by combining images of the food a user consumes, exercise data, and emotional data of the user, and provides appropriate diet advice. This system includes means for acquiring food images, analyzing images, calculating calories, acquiring exercise data, calculating calorie expenditure, an emotional engine, calculating calorie intake and expenditure, and providing advice.

[0695] System configuration

[0696] 1. Acquiring food images

[0697] Device (smart glasses): When the user eats, it takes pictures of the meal. When the user presses the start button, the photo of the meal is automatically taken.

[0698] 2. Analysis of food images

[0699] Server: The image data of the photographed meal is sent to the server, which uses image analysis algorithms to identify the meal contents (e.g., bread, omelet, salad) and the amount of each.

[0700] 3. Calorie Counting

[0701] Server: Retrieves calorie information corresponding to each identified food category from the database and calculates the total calories. For example, if bread is 250 kcal, omelet is 150 kcal, and salad is 50 kcal, the total will be calculated as 450 kcal.

[0702] Terminal (smartphone): Notifies the user of the calculation results and saves them as a meal history.

[0703] 4. Acquisition of exercise data

[0704] Device (smartwatch): When a user exercises, it automatically records exercise data such as the number of steps, heart rate, and exercise time. For example, if a user runs for 30 minutes, the data will be recorded.

[0705] Server: Exercise data is periodically sent to the server and stored for analysis.

[0706] 5. Calorie Calculation

[0707] Server: Based on the exercise data sent, the server calculates the calories burned using the exercise intensity, duration, and the user's physical data (weight, age, etc.). For example, it calculates that 230 kcal is burned by running for 30 minutes.

[0708] Terminal (smartphone): Notifies the user of the calculation results.

[0709] 6. Introducing the Emotion Engine

[0710] Device (smartphone): Recognizes the user's emotions while eating or exercising. The emotion engine uses the camera and microphone to analyze changes in the user's facial expressions and voice to identify emotions.

[0711] Server: The emotion data obtained from the emotion engine is sent to the server.

[0712] Specific examples

[0713] Acquiring food information and calculating calories

[0714] User: I eat bread and an omelet for breakfast.

[0715] Terminal (smart glasses): Takes a photo of the meal and sends it to the server.

[0716] Server: Performs image analysis to distinguish between bread and omelet. Calculates the calories as 250 kcal for bread and 150 kcal for omelet, for a total of 400 kcal.

[0717] Device (smartphone): Display the total as 400 kcal to the user and save the data.

[0718] Acquiring exercise information and calculating calories burned

[0719] User: Go for a 30-minute run in the morning.

[0720] Device (smartwatch): Records running exercise data and sends it to the server.

[0721] Server: Calculate the calories burned by running and it is 230 kcal.

[0722] Emotion recognition and advice provision

[0723] User: Feels certain emotions (e.g., satisfaction, fatigue, stress) after eating or during exercise.

[0724] Device (smartphone): Recognizes user emotions using a camera and microphone.

[0725] Server: The recognized emotion data is sent to the server.

[0726] Server: Calculates calorie balance based on calorie intake, calorie expenditure, and emotional data, and generates advice. For example, if a user is feeling stressed, it provides advice on diet and exercise to help them relax.

[0727] Calculates calorie balance and provides advice

[0728] Server: Combines the data of calorie intake (400 kcal) and calorie expenditure (230 kcal) to calculate the daily calorie balance. Here, the calorie balance is calculated as 400 kcal - 230 kcal = 170 kcal.

[0729] Server: Evaluates whether the calorie balance is appropriate for the user's diet goal. If necessary, generates specific advice about the next meal or exercise. For example, it provides advice such as "Keep your calorie intake to 200 kcal or less at your next meal."

[0730] Device (smartphone): Provides advice to the user and instructs them to reflect this in their daily diet and exercise.

[0731] This allows users to accurately manage their calorie intake and expenditure and receive appropriate advice based on their emotional state, making the system a powerful support tool for maintaining a healthy lifestyle.

[0732] The processing flow will be explained below.

[0733] Step 1:

[0734] Device (smart glasses): Before the user starts eating, they take a photo of the meal. When the user presses the start button, the smart glasses automatically take a photo of the meal.

[0735] Step 2:

[0736] Terminal (smart glasses): The captured food images are sent to the server. The image data is often compressed before being transferred.

[0737] Step 3:

[0738] Server: Prepares the received food images for analysis, specifically by preprocessing the images (noise removal, resolution adjustment, etc.) to apply image recognition algorithms.

[0739] Step 4:

[0740] Server: Uses image recognition algorithms to identify the type and quantity of food in a meal image. For example, identify if an image contains bread, an omelet, or a salad.

[0741] Step 5:

[0742] Server: Retrieve calorie information for each identified food type from the database. For example, bread is 250 kcal, omelette is 150 kcal, and salad is 50 kcal.

[0743] Step 6:

[0744] Server: Add up the calories from each food item to calculate your total calorie intake. In this example, we'll use bread, omelet, and salad as a total of 450 kcal.

[0745] Step 7:

[0746] Device (smartphone): The calculated total calories of 450 kcal are notified to the user's smartphone and saved in the meal history.

[0747] Step 8:

[0748] Device (smartwatch): Records exercise data in real time when the user exercises. For example, it records the number of steps, heart rate, and exercise time during a 30-minute run.

[0749] Step 9:

[0750] Device (smartwatch): Periodically sends recorded exercise data to the server.

[0751] Step 10:

[0752] Server: Based on the received exercise data, calculates the calories burned using exercise intensity, time, and the user's physical data (weight, age, etc.). For example, it calculates that 230 kcal is burned by running for 30 minutes.

[0753] Step 11:

[0754] Terminal (smartphone): Notifies the user of the calculated calorie consumption of 230 kcal.

[0755] Step 12:

[0756] Server: Combines the data of calorie intake (450 kcal) and calorie expenditure (230 kcal) to calculate the daily calorie balance. Here, the calorie balance is calculated as 450 kcal - 230 kcal = 220 kcal.

[0757] Step 13:

[0758] Device (smartphone): When the user eats or exercises, the emotion engine uses the camera and microphone to recognize the user's emotions.

[0759] Step 14:

[0760] Server: Emotion data obtained from the emotion engine is sent to the server. For example, it is recognized that the user is feeling stressed.

[0761] Step 15:

[0762] Server: Adapts advice content based on calorie balance and emotional data. For example, it provides advice on relaxation-enhancing diets and exercise to a user who is feeling stressed.

[0763] Step 16:

[0764] Device (smartphone): Notifies the user of the adapted advice. For example, display advice such as "Keep your calorie intake to 200 kcal or less at your next meal and do yoga to relax."

[0765] This series of processes allows users to accurately manage their calorie intake and expenditure, and receive appropriate advice based on their emotional state, making it easier to maintain a healthy lifestyle.

[0766] Example 2

[0767] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0768] Conventional calorie balance management systems collect and analyze users' dietary and exercise data, but no system provides advice that takes into account the user's emotional state. Therefore, our goal is to make it easier for users to maintain a healthier lifestyle by providing more personalized advice based on the user's emotional state in addition to dietary and exercise data.

[0769] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0770] In this invention, the server includes means for analyzing meal images to identify the type and amount of food, means for calculating calories based on the identified food, means for calculating calories burned based on exercise data, means for calculating calorie balance based on calories ingested and calories burned, means for recognizing and analyzing the user's emotions, and means for providing appropriate advice to the user based on the emotion data, thereby making it possible to provide detailed and personalized health advice according to the user's emotional state.

[0771] The term "means for acquiring food images" refers to a device or method for taking an image of the food a user eats and recording that image.

[0772] "Means for analyzing meal images to identify food types and quantities" refers to an algorithm or program for analyzing captured meal images and identifying the type and quantity of each food item in the image.

[0773] "Means for calculating calories" refers to a method or program for calculating total calorie intake by adding up the calories of each food based on the type and amount of food analyzed.

[0774] "Means for acquiring exercise data" refers to a device or method for recording data related to the exercise performed by a user (e.g., number of steps, heart rate, exercise time, etc.) and saving that data.

[0775] "Means for calculating calories burned" refers to a program or algorithm for calculating the amount of energy consumed by a user's exercise based on the acquired exercise data.

[0776] "Means for calculating calorie balance" refers to a method or program for calculating daily calorie balance based on calories consumed and calories burned.

[0777] "Means for providing dietary and exercise advice to a user" refers to a device or method that provides specific instructions or suggestions to a user regarding their next meal or exercise based on the calculated calorie balance.

[0778] "Means for recognizing and analyzing user emotions" refers to programs or algorithms that identify a user's emotional state using a camera or microphone and analyze that data.

[0779] "Means for providing appropriate advice to a user based on emotional data" refers to a device or method for providing more personalized health advice based on calorie intake, calorie expenditure, and calorie balance, taking into account the user's emotional state.

[0780] This invention is a system that manages daily calorie balance and provides appropriate diet advice by combining images of the food a user consumes, exercise data, and the user's emotional data. This system includes means for acquiring food images, analyzing images, calculating calories, acquiring exercise data, calculating calories burned, an emotional engine, calculating calorie balance, and providing advice.

[0781] Specifically, when a user starts eating, they use the device (smart glasses) to take a photo of their meal. When the user presses the start button, the smart glasses take two to three photos in succession and record the images. The captured image data is then temporarily processed using edge computing, compressed, and noise-removed before being sent to the server.

[0782] The server analyzes the received food image data. During this process, an image analysis algorithm (e.g., a deep learning model using TensorFlow) runs to identify the meal contents (bread, omelet, salad, etc.) and the amount of each. The results of the identification are saved along with the identification confidence, and the one with the highest confidence is adopted.

[0783] Next, the server retrieves calorie information corresponding to each identified food category from a database (for example, a dietary database such as the USDA or the Japanese Food Composition Tables). Based on the analyzed foods, the calories of each food are added up to calculate the total calorie intake. The calculation result is notified to the device (smartphone), and a message is displayed to the user saying, "The total calories for breakfast are 450 kcal." This data is also saved on the device as a meal history.

[0784] When a user exercises, the device (smartwatch) automatically records exercise data such as the number of steps, heart rate, and exercise time. The smartwatch's sensors collect data in real time and send it to a server every 10 minutes. The server stores the received exercise data for analysis and classifies it by exercise type. For example, this includes exercise data for running, walking, cycling, etc.

[0785] The server calculates the calories burned based on the received exercise data. It calculates the MET (Metabolic Equivalent of Task) value based on the type, intensity, and duration of exercise, as well as the user's physical data (weight, age). For example, it calculates that 230 kcal will be burned for a 30-minute run using a pre-set MET value (8 METs). The calculation result is sent to the device (smartphone), and a message is displayed saying, "You burned 230 kcal running."

[0786] Furthermore, when the user eats or exercises, the device (smartphone) uses a camera and microphone to recognize the user's emotions. Facial recognition software (for example, Microsoft's Face API or Google's Cloud Vision) and voice analysis software are used to identify emotions. The server receives the recognized emotion data and stores it as time-series data. If a specific emotion (such as satisfaction, fatigue, or stress) is recognized frequently, an alert is generated.

[0787] Finally, the server calculates the daily calorie balance based on the calorie intake, calorie expenditure, and emotional data. If the calorie intake is 450 kcal and the calorie expenditure is 230 kcal, the calorie balance is calculated as (450 kcal - 230 kcal = 220 kcal). Based on this calorie balance, advice is generated according to the user's diet goals and appropriate advice is provided to the user. For example, a specific prompt such as "Aim for 200 kcal or less at your next meal" is generated.

[0788] Examples of prompts for using the system include:

[0789] "Calculate calorie balance and generate appropriate diet advice based on today's calorie intake and expenditure, as well as the user's emotional data."

[0790] This allows users to accurately manage their calorie intake and expenditure and receive personalized advice based on their emotional state, making the system a powerful support tool for maintaining a healthy lifestyle.

[0791] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0792] Step 1:

[0793] Input: A picture of the meal when the user starts eating

[0794] Specific operation: When a user starts eating, they use the device (smart glasses) to take a photo of their meal. When the user presses the start button, the smart glasses take two to three photos in succession. The captured image data is temporarily processed using edge computing, where it is compressed and noise-removed.

[0795] Output: The processed meal image data is sent to the server.

[0796] Step 2:

[0797] Input: Food image data sent to the server

[0798] Specific operation: The server analyzes the received food image data. An image analysis algorithm (e.g., a deep learning model using TensorFlow) runs to identify the meal contents (bread, omelet, salad, etc.) and the amount of each. The identification results are saved along with the identification confidence score, and the one with the highest confidence score is adopted.

[0799] Output: Identified food type and quantity data

[0800] Step 3:

[0801] Input: Identified food type and quantity

[0802] Specific operation: The server retrieves calorie information corresponding to each identified food category from a database (e.g., a dietary database such as the USDA or the Japanese Food Composition Table). Based on the analyzed foods, the server calculates the total calorie intake by adding up the calories of each food.

[0803] Output: Total calorie intake data

[0804] Step 4:

[0805] Input: Total calorie intake data

[0806] Specific operation: The server notifies the device (smartphone) of the calculation results. The device displays a message to the user saying, "The total calories for breakfast is 450 kcal." This data is also saved on the device as a meal history.

[0807] Output: Total calorie intake information notified to the user

[0808] Step 5:

[0809] Input: Exercise data when the user exercises (number of steps, heart rate, exercise time, etc.)

[0810] How it works: When a user exercises, the device (smartwatch) automatically records exercise data. The smartwatch's sensors collect data in real time and send it to the server every 10 minutes.

[0811] Output: Exercise data sent to the server

[0812] Step 6:

[0813] Input: Exercise data sent to the server

[0814] Specific operation: The server calculates the calorie consumption based on the received exercise data. It calculates the MET (Metabolic Equivalent of Task) value based on the type, intensity, and duration of exercise, as well as the user's physical data (weight, age). For example, it calculates that 230 kcal will be consumed for a 30-minute run using a pre-set MET value (8 METs).

[0815] Output: Calculated calorie consumption data

[0816] Step 7:

[0817] Input: Calorie consumption data

[0818] Specific operation: The server notifies the terminal (smartphone) of the calculation result. The terminal displays a message to the user saying, "You burned 230 kcal by running."

[0819] Output: Calorie consumption information notified to the user

[0820] Step 8:

[0821] Input: Data showing the user's emotional state (facial expression images and voice data)

[0822] Specific operation: When the user eats or exercises, the device (smartphone) uses a camera and microphone to recognize the user's emotions, using facial recognition software and voice analysis software to identify emotions (satisfaction, fatigue, stress, etc.).

[0823] Output: Recognized emotion data

[0824] Step 9:

[0825] Input: Recognized emotion data

[0826] Specific operation: The server receives the recognized emotion data and generates appropriate advice based on that data. For example, if the user is feeling stressed, advice such as "do some light exercise to relax" or "eat a low-calorie diet" will be provided.

[0827] Output: Advice generated based on emotional state

[0828] Step 10:

[0829] Input: Calories ingested, calories burned, emotional data

[0830] Specific operation: The server calculates the daily calorie balance based on calorie intake, calorie expenditure, and emotional data. Specifically, if calorie intake is 450 kcal and calorie expenditure is 230 kcal, the server calculates the calorie balance (450 kcal - 230 kcal = 220 kcal). As a result, it generates advice based on the diet goal.

[0831] Output: Calorie balance and diet advice data

[0832] In this way, each processing step of this system performs specific data processing and calculation based on input data, and supports health management by providing the results to the user.

[0833] (Application example 2)

[0834] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0835] Many people today lead busy lifestyles, making it difficult to manage their health and diet. Therefore, it is necessary to provide appropriate advice based on the user's emotional state while managing calorie intake through diet and exercise. However, current systems have the challenge of not only managing dietary and exercise data comprehensively, but also incorporating emotional data to provide optimal advice to users.

[0836] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0837] In this invention, the server includes means for acquiring food images, means for analyzing the food images to identify the types and amounts of food, means for calculating calories based on the identified foods, means for acquiring exercise data, means for calculating calories burned based on the exercise data, means for calculating a calorie balance based on calories ingested and calories burned, means for acquiring user emotion data, means for evaluating the user's condition based on the emotion data, means for providing the user with advice regarding diet and exercise based on the calorie balance and emotion data, and means for notifying the user of the advice. This makes it possible to accurately provide appropriate advice based on the user's emotional state while managing calorie intake and calorie expenditure through exercise.

[0838] The "means for acquiring food images" refers to a device that has the function of taking an image of the food that the user eats and saving it as image data or transmitting it to a server.

[0839] The "means for analyzing the food image to identify the type and amount of food" is software or hardware that includes an algorithm for analyzing the captured food image and identifying the type and amount of food in the image.

[0840] A "calorie calculation means" is a means for calculating the calories of an entire meal based on the types and amounts of food identified and using calorie information stored in a database.

[0841] A "means for acquiring exercise data" is a device that records data about the exercise performed by the user (e.g., number of steps, exercise time, heart rate, etc.) and stores or transmits the data to a server.

[0842] The "means for calculating calories burned" is a means for calculating calories burned through exercise by the user based on the acquired exercise data.

[0843] A "means for calculating calorie balance" is a means for calculating daily calorie balance based on calories ingested and calories burned through exercise.

[0844] The "means for acquiring user emotional data" refers to a means for recognizing the emotional state of the user from their facial expressions and voice using sensors such as a camera and a microphone.

[0845] The "means for assessing the user's state" is software or hardware that includes an algorithm for analyzing the acquired emotional data and assessing the user's emotional state.

[0846] The "means for providing diet and exercise advice" is a means for generating and providing optimal diet and exercise advice to a user based on calorie balance and emotion data.

[0847] The "means for notifying the user of the advice" refers to a device or software having an interface for notifying the user of the generated advice.

[0848] The purpose of this invention is to build a system that not only manages the calorie intake of a user and the calorie consumption through exercise, but also provides more appropriate diet advice by utilizing the user's emotional data. This system is configured using the following hardware and software.

[0849] Hardware Configuration

[0850] Smartphone: Obtains the user's emotional data and provides advice. It also displays the total calories of the meal and the calorie balance.

[0851] Smart glasses: When the user eats a meal, they take a picture of the meal and send the image data to a server.

[0852] Smartwatch: Records exercise data (e.g., number of steps, heart rate, and exercise time) when the user exercises and sends it to a server.

[0853] Server: Analyzes food images, calculates calories, calculates calorie expenditure, analyzes emotional data, and generates advice.

[0854] Software Configuration

[0855] Image analysis algorithm: Runs on the server and analyzes the food images sent from the smart glasses to identify the type and quantity of food.

[0856] Calorie calculation engine: Based on the image analysis results, calculates the calories of each food item and calculates the total calories.

[0857] Exercise data analysis engine: Calculates calories burned based on exercise data sent from the smartwatch.

[0858] Emotion recognition engine: Uses the smartphone's camera and microphone to recognize the user's emotional state.

[0859] Advice generation engine: Integrates total calorie intake, calorie expenditure, and emotional data to generate dietary and exercise advice tailored to the user's condition.

[0860] System Operation

[0861] 1. Acquisition and analysis of meal images

[0862] As a user wearing the smart glasses eats, the glasses take images of the meal and send the image data to a server, which uses image analysis algorithms to identify the meal contents and determine the type and quantity of each food item.

[0863] 2. Calorie Counting

[0864] The server calculates the calorie count of each food item based on the type and amount of food identified, and then calculates the total calorie count using a calorie calculation engine. The calculation results are sent to the smartphone and notified to the user.

[0865] 3. Acquisition of exercise data and calculation of calories burned

[0866] When a user wears a smartwatch and exercises, the exercise data is recorded on the smartwatch and periodically sent to a server. The server uses an exercise data analysis engine to calculate the calories burned through exercise and sends the results to a smartphone.

[0867] 4. Acquisition and analysis of emotion data

[0868] Smartphones use cameras and microphones to recognize the user's emotional state. An emotion recognition engine analyzes this and generates emotional data. The generated emotional data is sent to a server.

[0869] 5. Advice Generation

[0870] The server integrates the total calorie intake, calorie expenditure, and emotion data, and generates appropriate diet and exercise advice for the user using an advice generation engine. The generated advice is sent to the user's smartphone and notified.

[0871] Specific examples

[0872] When a user orders lunch through a food delivery app, the following process occurs:

[0873] Prompt Sentence Examples

[0874] I ordered ramen for lunch today. Please check the calorie information and provide appropriate advice based on my exercise data and calorie intake. My current feeling is recognized as "a little tired."

[0875] Specifically, the process is as follows: When a user orders ramen, the app displays the ramen's calorie information (for example, 700 kcal), and calculates the current calorie balance, taking into account the 200 kcal consumed in the previous exercise data. Emotion recognition determines that the user is "slightly tired," and since the calories consumed are sufficient, advice is provided such as "It would be a good idea to order a low-calorie side dish or a protein drink at the same time."

[0876] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0877] Step 1:

[0878] Acquiring food images

[0879] When a user wearing smart glasses eats a meal, the smart glasses take an image of the meal and send the image data to a server. The input is an image of the meal, and the output is the image data of the meal stored on the server. Specifically, when the user starts eating, the camera in the smart glasses automatically activates and takes an image.

[0880] Step 2:

[0881] Food image analysis

[0882] The server uses an image analysis algorithm to analyze the food image data sent from the smart glasses and identify the type and quantity of food. The input is the food image data, and the output is the identified food type and quantity data. Specifically, the server identifies the foods in the image and labels each food category.

[0883] Step 3:

[0884] Calorie Counting

[0885] The server uses a calorie calculation engine to calculate the calories of each food based on the type and amount of the identified food, and then calculates the total calories. The input is data on the type and amount of food, and the output is the total calorie value. Specifically, the server retrieves the calorie information of each food from the database and calculates the total calories based on that information.

[0886] Step 4:

[0887] Calorie information notification

[0888] The server sends the calculated total calorie information to the smartphone and notifies the user. The input is the total calorie value, and the output is the calorie information displayed on the smartphone. Specifically, the smartphone receives the notification and displays the calorie information on the user interface.

[0889] Step 5:

[0890] Acquisition of exercise data

[0891] When a user wears a smartwatch and exercises, the smartwatch records exercise data (e.g., number of steps, heart rate, and exercise time) and periodically transmits it to a server. The input is raw exercise data, and the output is exercise data stored on the server. Specifically, the smartwatch monitors the user's exercise in real time and records and transmits the data.

[0892] Step 6:

[0893] Calorie Calculation

[0894] The server uses an exercise data analysis engine to calculate calories burned based on the exercise data sent from the smartwatch. The input is exercise data, and the output is the calorie burned value. Specifically, the server calculates calories burned based on the exercise intensity, duration, and the user's physical data.

[0895] Step 7:

[0896] Acquiring emotion data

[0897] Smartphones use cameras and microphones to recognize the user's emotional state. The input is raw emotional data (images and audio), and the output is recognized emotional data. Specifically, the smartphone analyzes facial expressions and tone of voice to identify and record the user's emotional state.

[0898] Step 8:

[0899] Emotional Data Analysis

[0900] The server uses an emotion recognition engine to evaluate the user's emotional state based on the emotion data sent from the smartphone. The input is the emotion data, and the output is the evaluated emotional state. Specifically, the server analyzes the emotion data and labels the user's emotional state.

[0901] Step 9:

[0902] Advice Generation

[0903] The server uses an advice generation engine to generate appropriate dietary and exercise advice for the user based on the total calorie intake, calorie expenditure, and emotional data. The input is the calorie intake, calorie expenditure, and emotional data, and the output is the generated advice. Specifically, the server integrates this data and generates advice that is optimal for the user's situation.

[0904] Step 10:

[0905] Advice Notification

[0906] The server sends the generated advice to the smartphone and notifies the user. The input is the generated advice, and the output is the advice displayed on the smartphone. Specifically, the smartphone receives the notification and displays the advice on its user interface.

[0907] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0908] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0909] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0910] [Third embodiment]

[0911] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0912] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0913] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0914] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0915] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0916] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0917] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0918] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0919] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0920] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0921] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0922] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0923] The present invention relates to a system that combines images of food consumed by a user with exercise data to manage the daily calorie balance and provide appropriate diet advice. This system includes means for acquiring food images, analyzing images, calculating calories, acquiring exercise data, calculating calories burned, calculating calorie balance, and providing advice.

[0924] System configuration

[0925] 1. Acquiring food images

[0926] Device (smart glasses): When the user eats, it takes pictures of the meal. When the user presses the start button, the photo of the meal is automatically taken.

[0927] 2. Analysis of food images

[0928] Server: The image data of the photographed meal is sent to the server, which uses image analysis algorithms to identify the meal contents (e.g., bread, omelet, salad) and the amount of each.

[0929] 3. Calorie Counting

[0930] Server: Retrieves calorie information corresponding to each identified food category from the database and calculates the total calories. For example, if bread is 250 kcal, omelet is 150 kcal, and salad is 50 kcal, the total will be calculated as 450 kcal.

[0931] Terminal (smartphone): Notifies the user of the calculation results and saves them as a meal history.

[0932] 4. Acquisition of exercise data

[0933] Device (smartwatch): When a user exercises, it automatically records exercise data such as the number of steps, heart rate, and exercise time. For example, if a user runs for 30 minutes, the data will be recorded.

[0934] Server: Exercise data is periodically sent to the server and stored for analysis.

[0935] 5. Calorie Calculation

[0936] Server: Calculates the calories burned based on the exercise data sent. For example, it calculates that 230 kcal is burned by running for 30 minutes.

[0937] Terminal (smartphone): Notifies the user of the calculation results.

[0938] Specific examples

[0939] Acquiring food information and calculating calories

[0940] User: I eat bread and an omelet for breakfast.

[0941] Terminal (smart glasses): Takes a photo of the meal and sends it to the server.

[0942] Server: Performs image analysis to distinguish between bread and omelet. Calculates the calories as 250 kcal for bread and 150 kcal for omelet, for a total of 400 kcal.

[0943] Device (smartphone): Display the total as 400 kcal to the user and save the data.

[0944] Acquiring exercise information and calculating calories burned

[0945] User: Go for a 30-minute run in the morning.

[0946] Device (smartwatch): Records running exercise data and sends it to the server.

[0947] Server: Calculate the calories burned by running and it is 230 kcal.

[0948] Calculates calorie balance and provides advice

[0949] Server: Based on the calories ingested (400 kcal from breakfast) and calories expended (230 kcal from running), calculate your calorie balance (400 kcal - 230 kcal = 170 kcal).

[0950] Server: Analyzes whether the calorie balance is appropriate for the user's diet goals. If necessary, provides guidance on reducing calorie intake at the next meal.

[0951] Device (smartphone): Display the advice to the user, "Keep your next meal under 200 kcal."

[0952] In this way, users can accurately manage their daily calorie intake and expenditure and receive appropriate advice on maintaining a healthy weight. This system is expected to dramatically improve the success rate of dieting and health management.

[0953] The processing flow will be explained below.

[0954] Step 1:

[0955] Device (smart glasses): Before the user starts eating, they take a photo of their meal. When the user presses a button, the smart glasses automatically take a photo of the meal.

[0956] Step 2:

[0957] Terminal (smart glasses): The captured food images are sent to the server. The image data is often compressed before being transferred.

[0958] Step 3:

[0959] Server: Prepares the received food images for analysis, specifically by pre-processing the images to apply image recognition algorithms.

[0960] Step 4:

[0961] Server: Uses image recognition algorithms to identify the type and quantity of food in a meal image. For example, identify if an image contains bread, an omelet, or a salad.

[0962] Step 5:

[0963] Server: Retrieve calorie information for each identified food type from the database. For example, bread is 250 kcal, omelette is 150 kcal, and salad is 50 kcal.

[0964] Step 6:

[0965] Server: Add up the calories from each food item to calculate your total calorie intake. In this example, we'll use bread, omelet, and salad as a total of 450 kcal.

[0966] Step 7:

[0967] Device (smartphone): The calculated total calories of 450 kcal are notified to the user's smartphone and saved in the meal history.

[0968] Step 8:

[0969] Device (smartwatch): Records exercise data in real time when the user exercises. For example, it records the number of steps, heart rate, and exercise time during a 30-minute run.

[0970] Step 9:

[0971] Device (smartwatch): Periodically sends recorded exercise data to the server.

[0972] Step 10:

[0973] Server: Based on the received exercise data, calculates the calories burned using exercise intensity, time, and the user's physical data (weight, age, etc.). For example, it calculates that 230 kcal is burned by running for 30 minutes.

[0974] Step 11:

[0975] Terminal (smartphone): Notifies the user of the calculated calorie consumption of 230 kcal.

[0976] Step 12:

[0977] Server: Combines the data of calorie intake (450 kcal) and calorie expenditure (230 kcal) to calculate the daily calorie balance. Here, the calorie balance is calculated as 450 kcal - 230 kcal = 220 kcal.

[0978] Step 13:

[0979] Server: Evaluates whether the calorie balance is appropriate for the user's diet goal. For example, if the user has set a daily calorie goal of 0 kcal, this 220 kcal is outside the goal.

[0980] Step 14:

[0981] Server: Generates specific advice about the next meal and exercise based on the calorie balance assessment. For example, it provides advice such as "keep your calorie intake at your next meal to 200 kcal or less."

[0982] Step 15:

[0983] Device (smartphone): Provides advice to the user and instructs them to reflect this in their daily diet and exercise.

[0984] This allows users to accurately manage their calorie intake and expenditure and maintain a healthy lifestyle.

[0985] Example 1

[0986] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0987] Currently, many people struggle to balance dietary calorie management and exercise as part of their health management. Conventional methods require users to manually input meal contents and calculate calories themselves, which is time-consuming and prone to errors. In addition, the collection and analysis of exercise data is not centrally managed, often preventing users from receiving appropriate diet and health guidance. Therefore, there is a need for a system that allows users to easily manage their daily calorie intake and expenditure and maintain healthy lifestyle habits.

[0988] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0989] In this invention, the server includes means for analyzing food images to identify the type and amount of food, means for calculating calories based on the identified food, means for calculating calories burned based on exercise data, means for calculating a calorie balance based on calories ingested and calories burned, and means for providing the user with advice on diet and exercise based on the calorie balance. This allows the user to automatically and accurately grasp the calorie intake from food and the calories burned from exercise, and by receiving appropriate advice based on this, it becomes possible to efficiently maintain a balance between health management and calorie balance.

[0990] A "meal image" is image data captured to record the contents of a meal that a user eats.

[0991] A "terminal" is an electronic device used by a user, such as smart glasses, a smart watch, or a smartphone.

[0992] The "server" is an information processing device that analyzes image data and exercise data, calculates calories, and provides advice.

[0993] "Type and amount of food" refers to the specific food name and portion size identified by analyzing the meal image.

[0994] A "calorie" is a unit that represents the amount of energy contained in food.

[0995] "Exercise data" refers to data such as the type of exercise performed by the user, the number of steps taken, heart rate, and exercise time.

[0996] "Calories burned" is the amount of energy consumed through exercise, calculated based on exercise data.

[0997] "Calories ingested" refers to the amount of energy a user takes in through food.

[0998] "Calorie balance" is the number obtained by subtracting calories consumed from calories consumed.

[0999] "Providing advice" refers to giving instructions or advice to the user regarding diet and exercise based on calorie balance.

[1000] A "natural language processing model" is one of the artificial intelligence techniques used to analyze food images and exercise data.

[1001] "Biometric data" refers to the user's physical information (e.g., height, weight, age, gender, etc.).

[1002] This invention relates to a system that combines images of the food a user eats with exercise data to manage their daily calorie intake and expenditure and provide appropriate diet advice. This system enables users to efficiently manage their daily calorie intake and expenditure and aim to maintain a healthy weight.

[1003] The main hardware components of the system include a device for acquiring food images (e.g., smart glasses), a device for acquiring exercise data (e.g., a smart watch), a device for notifying the user of the analysis results (e.g., a smartphone), and a server for performing data analysis.

[1004] First, the user takes an image of their meal using the smart glasses. When the user presses the start button on the smart glasses, the camera captures the image and sends the image data wirelessly to the server. Next, the server uses an image analysis algorithm (e.g., TensorFlow) to analyze the captured image of the meal. This analysis identifies the contents of the meal and the type and amount of each food item. The analysis results are stored in a database, and the server calculates the total calories based on the calorie information for each food item. For example, if bread is 250 kcal, an omelet is 150 kcal, and a salad is 50 kcal, the total will be calculated as 450 kcal.

[1005] The calculated calorie information is sent from the server to the smartphone and notified to the user. Using the smartphone application, the user can check the calorie intake information and save it as a meal history.

[1006] Furthermore, when a user exercises using a smartwatch, the smartwatch automatically records exercise data such as the number of steps, heart rate, and exercise time. For example, if a user runs for 30 minutes, the data is recorded. The recorded exercise data is sent to a server at regular intervals. Based on the exercise data, the server calculates the calories burned according to the type and intensity of exercise. For example, it calculates that 230 kcal is burned in a 30-minute run. This calorie burn information is also notified from the server to the smartphone, where it is displayed to the user and saved as exercise history.

[1007] Finally, the server calculates the calorie balance based on the calorie intake and calorie expenditure information. For example, if the calorie intake for breakfast is 400 kcal and the calorie expenditure through exercise is 230 kcal, the calorie balance is 400 kcal - 230 kcal = 170 kcal. The server analyzes whether the calorie balance is appropriate for the user's diet goal and generates advice regarding the next meal or exercise as necessary. For example, advice such as "Keep your next meal to 200 kcal or less" is sent from the server to the user's smartphone and notified.

[1008] This specific system allows users to accurately manage their daily calorie intake and expenditure, and receive appropriate advice based on that, enabling them to efficiently manage their health.

[1009] Prompt Sentence Examples

[1010] "I had bread and an omelet for breakfast, so please count the calories."

[1011] "I ran for 30 minutes this morning. How many calories did I burn?"

[1012] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1013] Step 1:

[1014] Acquiring food images

[1015] Device (smart glasses): When the user starts eating, the device presses the start button to take an image of the meal. The image is captured by the smart glasses' camera and saved in the device's internal memory. The image is then sent to the server via wireless communication.

[1016] Input: The action of the user pressing the start button on the device.

[1017] Output: Food image data sent to the server.

[1018] Step 2:

[1019] Food image analysis

[1020] Server: The server processes the received image data using an analysis algorithm (e.g., TensorFlow) to determine the meal contents and portions. This analysis includes the type of food (e.g., bread, omelet, salad) and the portion size of each.

[1021] Input: Food image data sent from the device.

[1022] Output: Data on the type and quantity of food analyzed.

[1023] Step 3:

[1024] Calorie Counting

[1025] Server: Based on the analysis results, the calorie information for each food category is retrieved from a database (e.g., MySQL) and the total calories are calculated. For example, if bread is 250 kcal, an omelet is 150 kcal, and a salad is 50 kcal, the total is calculated as 450 kcal.

[1026] Input: Data on the type and quantity of food parsed.

[1027] Output: Numeric data showing total calories.

[1028] Step 4:

[1029] Calorie information notification

[1030] Terminal (smartphone): The server sends the calculated total calorie information to the terminal and notifies the user. The smartphone application displays the calorie information and saves it in a database as a meal history.

[1031] Input: Numeric data of total calories sent from the server.

[1032] Output: Calorie notification to user and storage in food history database.

[1033] Step 5:

[1034] Acquisition of exercise data

[1035] Device (smartwatch): When the user starts exercising, the device automatically records exercise data such as the number of steps, heart rate, and exercise time. After the exercise is finished, the data is periodically sent to the server.

[1036] Input: User's athletic activity.

[1037] Output: Exercise data sent to the server.

[1038] Step 6:

[1039] Calorie Calculation

[1040] Server: Based on the received exercise data, calculates the calories burned according to the type and intensity of exercise. For example, it calculates that 230 kcal is burned in 30 minutes of running.

[1041] Input: Submitted exercise data.

[1042] Output: Numerical data showing calories burned.

[1043] Step 7:

[1044] Calorie consumption information notification

[1045] Device (smartphone): The server sends the calculated calorie consumption information to the device and notifies the user. The smartphone application displays the calorie consumption information and saves it in a database as exercise history.

[1046] Input: Numerical data of calories burned sent from the server.

[1047] Output: Notify user of calories burned and save to exercise history database.

[1048] Step 8:

[1049] Calculating calorie balance

[1050] Server: Calculates calorie balance using data on calories in and calories out. For example, if your calorie intake is 400 kcal and your calorie expenditure is 230 kcal, your calorie balance will be 170 kcal.

[1051] Input: Numerical data of calories ingested and calories burned.

[1052] Output: Numerical data showing calorie balance.

[1053] Step 9:

[1054] Providing advice

[1055] Server: Based on the calorie balance, the server generates advice based on the user's diet goals. For example, the server may instruct the user to limit their next meal to 200 kcal or less. The advice is sent from the server to the smartphone.

[1056] Input: Calorie balance data.

[1057] Output: Advice notice to user.

[1058] Step 10:

[1059] User Notification and Data Retention

[1060] Terminal (smartphone): Advice is displayed, and once the user confirms it, the record is saved in the database.

[1061] Input: Advice notification sent by the server.

[1062] Output: Advice notification to the user and record saved in the database.

[1063] (Application example 1)

[1064] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1065] In modern society, individual health management is extremely important, but it is not easy to properly manage daily calorie intake and expenditure and maintain a balanced diet. Furthermore, few systems allow users to easily understand their dietary habits and receive appropriate advice based on that information. Furthermore, there is no integration of food delivery services that allow users to immediately order suggested meals. This leads to people living their daily lives without adequate health management, which increases long-term health risks.

[1066] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1067] In this invention, the server includes a means for acquiring food images, a means for analyzing the food images to identify the type and amount of food, a means for calculating calories based on the identified food, a means for acquiring exercise data, a means for calculating calories burned based on the exercise data, a means for calculating a calorie balance based on calories ingested and calories burned, a means for providing the user with dietary and exercise advice based on the calorie balance, and a means for suggesting and ordering a meal menu suitable for a food delivery service based on the advice. This allows users to manage their own dietary and exercise habits in real time and receive appropriate suggestions for maintaining a healthy lifestyle. Furthermore, the suggested meals can be easily ordered, greatly improving user convenience.

[1068] The "means for acquiring food images" is a device that has the function of taking an image of a meal when the user eats and transmitting the image to the system.

[1069] The "means for analyzing a meal image to identify the type and amount of food" is an algorithm or program for analyzing a captured meal image and identifying the type and amount of food contained in the image.

[1070] The "means for calculating calories based on identified foods" refers to a database or calculation mechanism that has the function of calculating the calories of a food based on the type and amount of the analyzed food.

[1071] "Means for acquiring exercise data" refers to a device that has the function of recording data related to the exercise performed by the user (e.g., number of steps, exercise time, heart rate, etc.) and transmitting that data to the system.

[1072] The "means for calculating calories burned based on exercise data" is an algorithm or program that calculates the calories burned by the user based on the acquired exercise data.

[1073] The "means for calculating the calorie balance based on the calorie intake and the calorie expenditure" is a calculation mechanism having a function of calculating the difference between the calories taken in and the calories consumed, and calculating the calorie balance for that day.

[1074] The "means for providing dietary and exercise advice to a user based on calorie balance" is an algorithm or program for suggesting the next meal or recommending exercise to a user based on calorie balance.

[1075] The "means for proposing a meal menu suitable for a food delivery service and ordering that menu" is a system for proposing a meal menu suitable for a user based on the user's calorie balance and ordering that menu through a food delivery service.

[1076] The present invention relates to a system that combines images of meals consumed by a user with exercise data to manage daily calorie intake and provide appropriate meal menus through a food delivery service.

[1077] System configuration details

[1078] 1. Acquiring food images

[1079] Device (smartphone or smart glasses): Takes pictures of the meal while the user is eating. For example, a user takes a photo of their breakfast with their smartphone and uploads it to the application.

[1080] 2. Analysis of food images

[1081] Server: The image data of the photographed meal is sent to the server, which uses an image analysis algorithm (e.g., a machine learning model such as CNN) to identify the meal contents (e.g., bread, omelet, salad) and the amount of each.

[1082] 3. Calorie Counting

[1083] Server: Retrieves calorie information corresponding to each identified food item from the database and calculates the total calories. For example, if bread is 250 kcal, omelet is 150 kcal, and salad is 50 kcal, the total is calculated as 450 kcal.

[1084] 4. Acquisition of exercise data

[1085] Device (smartwatch): When a user exercises, data such as the number of steps, heart rate, and exercise time is automatically recorded. For example, if a user runs for 30 minutes in the morning, the data is recorded.

[1086] Server: Exercise data is periodically sent to the server and stored.

[1087] 5. Calorie Calculation

[1088] Server: Based on the exercise data sent, the server uses machine learning algorithms to calculate the calories burned. For example, it calculates that 30 minutes of running will burn 230 kcal.

[1089] 6. Calculate calorie balance and provide advice

[1090] Server: Calculates the calorie balance based on the identified calorie intake and expenditure (e.g., intake: 450 kcal - expenditure: 230 kcal = 220 kcal).

[1091] Server: Analyzes whether this calorie balance is appropriate for the user's diet goals and health management standards, and provides advice on reducing or increasing calorie intake at the next meal.

[1092] 7. Food delivery service integration

[1093] Server: Based on the generated advice (e.g., "Keep your next meal under 200 kcal"), it proposes a meal menu suitable for the food delivery service and allows the user to order that menu. In this case, it uses a generative AI model to automatically suggest the optimal menu.

[1094] Specific examples

[1095] A user eats bread and an omelet for breakfast, takes a photo of it with their smartphone, and uploads it to the application. The image is analyzed by the server, and the bread is determined to have 250 kcal and the omelet to have 150 kcal. If the user runs for 30 minutes in the morning, 230 kcal will be burned.

[1096] The server uses this data to calculate calorie balance and suggests ways to reduce calories in your next meal, allowing you to quickly order a salad or snack from a food delivery service.

[1097] Prompt Sentence Examples

[1098] A user has uploaded an image of bread and an omelet that they had for breakfast. Calculate the total calories of this meal and calculate their daily calorie balance using their exercise data as input. Then, suggest their next meal based on their calorie balance and order that menu for delivery.

[1099] This system allows users to manage their calorie balance in real time and receive support to maintain a healthy lifestyle, while also significantly improving user convenience.

[1100] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1101] Step 1:

[1102] Acquiring food images

[1103] Input: An image taken by the user while eating.

[1104] How it works: A user takes a picture of their meal using their smartphone or smart glasses and uploads the image to the application.

[1105] Output: Photographed food image data.

[1106] Step 2:

[1107] Food image analysis

[1108] Input: Food image data sent to the server.

[1109] How it works: The server uses an image analysis algorithm (e.g., a machine learning model such as CNN) to identify the types and amounts of food in the image.

[1110] Output: Data on the type and quantity of food identified.

[1111] Step 3:

[1112] Calorie Counting

[1113] Input: Identified food type and quantity data.

[1114] Operation: Based on this food data, the server retrieves the calorie information for each food from the database and calculates the total calories by adding up the calories for each food.

[1115] Output: Total calorie intake data.

[1116] Step 4:

[1117] Acquisition of exercise data

[1118] Input: User's exercise data (e.g., number of steps, heart rate, exercise duration, etc.).

[1119] How it works: A user uses a device such as a smartwatch to record exercise data and periodically sends that data to a server.

[1120] Output: Recording of exercise data.

[1121] Step 5:

[1122] Calorie Calculation

[1123] Input: Exercise data sent to the server.

[1124] How it works: The server uses machine learning algorithms to calculate calories burned based on your exercise data.

[1125] Output: Calories burned data.

[1126] Step 6:

[1127] Calculates calorie balance and provides advice

[1128] Input: Total calorie intake and calorie expenditure data.

[1129] How it works: The server calculates the calorie balance based on the calories ingested and calories burned, and then uses a generative AI model to create advice about the next meal and exercise based on that balance.

[1130] Output: Calorie balance calculation results and advice data.

[1131] Step 7:

[1132] Proposal and ordering for food delivery services

[1133] Input: Generated advice data.

[1134] Operation: The server suggests an appropriate meal menu based on the user's calorie balance, and provides the user with the ability to order that menu from a food delivery service.

[1135] Output: Suggested meal menu and ordering data.

[1136] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1137] The present invention relates to a system that manages daily calorie intake and expenditure by combining images of the food a user consumes, exercise data, and emotional data of the user, and provides appropriate diet advice. This system includes means for acquiring food images, analyzing images, calculating calories, acquiring exercise data, calculating calorie expenditure, an emotional engine, calculating calorie intake and expenditure, and providing advice.

[1138] System configuration

[1139] 1. Acquiring food images

[1140] Device (smart glasses): When the user eats, it takes pictures of the meal. When the user presses the start button, the photo of the meal is automatically taken.

[1141] 2. Analysis of food images

[1142] Server: The image data of the photographed meal is sent to the server, which uses image analysis algorithms to identify the meal contents (e.g., bread, omelet, salad) and the amount of each.

[1143] 3. Calorie Counting

[1144] Server: Retrieves calorie information corresponding to each identified food category from the database and calculates the total calories. For example, if bread is 250 kcal, omelet is 150 kcal, and salad is 50 kcal, the total will be calculated as 450 kcal.

[1145] Terminal (smartphone): Notifies the user of the calculation results and saves them as a meal history.

[1146] 4. Acquisition of exercise data

[1147] Device (smartwatch): When a user exercises, it automatically records exercise data such as the number of steps, heart rate, and exercise time. For example, if a user runs for 30 minutes, the data will be recorded.

[1148] Server: Exercise data is periodically sent to the server and stored for analysis.

[1149] 5. Calorie Calculation

[1150] Server: Based on the exercise data sent, the server calculates the calories burned using the exercise intensity, duration, and the user's physical data (weight, age, etc.). For example, it calculates that 230 kcal is burned by running for 30 minutes.

[1151] Terminal (smartphone): Notifies the user of the calculation results.

[1152] 6. Introducing the Emotion Engine

[1153] Device (smartphone): Recognizes the user's emotions while eating or exercising. The emotion engine uses the camera and microphone to analyze changes in the user's facial expressions and voice to identify emotions.

[1154] Server: The emotion data obtained from the emotion engine is sent to the server.

[1155] Specific examples

[1156] Acquiring food information and calculating calories

[1157] User: I eat bread and an omelet for breakfast.

[1158] Terminal (smart glasses): Takes a photo of the meal and sends it to the server.

[1159] Server: Performs image analysis to distinguish between bread and omelet. Calculates the calories as 250 kcal for bread and 150 kcal for omelet, for a total of 400 kcal.

[1160] Device (smartphone): Display the total as 400 kcal to the user and save the data.

[1161] Acquiring exercise information and calculating calories burned

[1162] User: Go for a 30-minute run in the morning.

[1163] Device (smartwatch): Records running exercise data and sends it to the server.

[1164] Server: Calculate the calories burned by running and it is 230 kcal.

[1165] Emotion recognition and advice provision

[1166] User: Feels certain emotions (e.g., satisfaction, fatigue, stress) after eating or during exercise.

[1167] Device (smartphone): Recognizes user emotions using a camera and microphone.

[1168] Server: The recognized emotion data is sent to the server.

[1169] Server: Calculates calorie balance based on calorie intake, calorie expenditure, and emotional data, and generates advice. For example, if a user is feeling stressed, it provides advice on diet and exercise to help them relax.

[1170] Calculates calorie balance and provides advice

[1171] Server: Combines the data of calorie intake (400 kcal) and calorie expenditure (230 kcal) to calculate the daily calorie balance. Here, the calorie balance is calculated as 400 kcal - 230 kcal = 170 kcal.

[1172] Server: Evaluates whether the calorie balance is appropriate for the user's diet goal. If necessary, generates specific advice about the next meal or exercise. For example, it provides advice such as "Keep your calorie intake to 200 kcal or less at your next meal."

[1173] Device (smartphone): Provides advice to the user and instructs them to reflect this in their daily diet and exercise.

[1174] This allows users to accurately manage their calorie intake and expenditure and receive appropriate advice based on their emotional state, making the system a powerful support tool for maintaining a healthy lifestyle.

[1175] The processing flow will be explained below.

[1176] Step 1:

[1177] Device (smart glasses): Before the user starts eating, they take a photo of the meal. When the user presses the start button, the smart glasses automatically take a photo of the meal.

[1178] Step 2:

[1179] Terminal (smart glasses): The captured food images are sent to the server. The image data is often compressed before being transferred.

[1180] Step 3:

[1181] Server: Prepares the received food images for analysis, specifically by preprocessing the images (noise removal, resolution adjustment, etc.) to apply image recognition algorithms.

[1182] Step 4:

[1183] Server: Uses image recognition algorithms to identify the type and quantity of food in a meal image. For example, identify if an image contains bread, an omelet, or a salad.

[1184] Step 5:

[1185] Server: Retrieve calorie information for each identified food type from the database. For example, bread is 250 kcal, omelette is 150 kcal, and salad is 50 kcal.

[1186] Step 6:

[1187] Server: Add up the calories from each food item to calculate your total calorie intake. In this example, we'll use bread, omelet, and salad as a total of 450 kcal.

[1188] Step 7:

[1189] Device (smartphone): The calculated total calories of 450 kcal are notified to the user's smartphone and saved in the meal history.

[1190] Step 8:

[1191] Device (smartwatch): Records exercise data in real time when the user exercises. For example, it records the number of steps, heart rate, and exercise time during a 30-minute run.

[1192] Step 9:

[1193] Device (smartwatch): Periodically sends recorded exercise data to the server.

[1194] Step 10:

[1195] Server: Based on the received exercise data, calculates the calories burned using exercise intensity, time, and the user's physical data (weight, age, etc.). For example, it calculates that 230 kcal is burned by running for 30 minutes.

[1196] Step 11:

[1197] Terminal (smartphone): Notifies the user of the calculated calorie consumption of 230 kcal.

[1198] Step 12:

[1199] Server: Combines the data of calorie intake (450 kcal) and calorie expenditure (230 kcal) to calculate the daily calorie balance. Here, the calorie balance is calculated as 450 kcal - 230 kcal = 220 kcal.

[1200] Step 13:

[1201] Device (smartphone): When the user eats or exercises, the emotion engine uses the camera and microphone to recognize the user's emotions.

[1202] Step 14:

[1203] Server: Emotion data obtained from the emotion engine is sent to the server. For example, it is recognized that the user is feeling stressed.

[1204] Step 15:

[1205] Server: Adapts advice content based on calorie balance and emotional data. For example, it provides advice on relaxation-enhancing diets and exercise to a user who is feeling stressed.

[1206] Step 16:

[1207] Device (smartphone): Notifies the user of the adapted advice. For example, display advice such as "Keep your calorie intake to 200 kcal or less at your next meal and do yoga to relax."

[1208] This series of processes allows users to accurately manage their calorie intake and expenditure, and receive appropriate advice based on their emotional state, making it easier to maintain a healthy lifestyle.

[1209] Example 2

[1210] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1211] Conventional calorie balance management systems collect and analyze users' dietary and exercise data, but no system provides advice that takes into account the user's emotional state. Therefore, our goal is to make it easier for users to maintain a healthier lifestyle by providing more personalized advice based on the user's emotional state in addition to dietary and exercise data.

[1212] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1213] In this invention, the server includes means for analyzing meal images to identify the type and amount of food, means for calculating calories based on the identified food, means for calculating calories burned based on exercise data, means for calculating calorie balance based on calories ingested and calories burned, means for recognizing and analyzing the user's emotions, and means for providing appropriate advice to the user based on the emotion data, thereby making it possible to provide detailed and personalized health advice according to the user's emotional state.

[1214] The term "means for acquiring food images" refers to a device or method for taking an image of the food a user eats and recording that image.

[1215] "Means for analyzing meal images to identify food types and quantities" refers to an algorithm or program for analyzing captured meal images and identifying the type and quantity of each food item in the image.

[1216] "Means for calculating calories" refers to a method or program for calculating total calorie intake by adding up the calories of each food based on the type and amount of food analyzed.

[1217] "Means for acquiring exercise data" refers to a device or method for recording data related to the exercise performed by a user (e.g., number of steps, heart rate, exercise time, etc.) and saving that data.

[1218] "Means for calculating calories burned" refers to a program or algorithm for calculating the amount of energy consumed by a user's exercise based on the acquired exercise data.

[1219] "Means for calculating calorie balance" refers to a method or program for calculating daily calorie balance based on calories consumed and calories burned.

[1220] "Means for providing dietary and exercise advice to a user" refers to a device or method that provides specific instructions or suggestions to a user regarding their next meal or exercise based on the calculated calorie balance.

[1221] "Means for recognizing and analyzing user emotions" refers to programs or algorithms that identify a user's emotional state using a camera or microphone and analyze that data.

[1222] "Means for providing appropriate advice to a user based on emotional data" refers to a device or method for providing more personalized health advice based on calorie intake, calorie expenditure, and calorie balance, taking into account the user's emotional state.

[1223] This invention is a system that manages daily calorie balance and provides appropriate diet advice by combining images of the food a user consumes, exercise data, and the user's emotional data. This system includes means for acquiring food images, analyzing images, calculating calories, acquiring exercise data, calculating calories burned, an emotional engine, calculating calorie balance, and providing advice.

[1224] Specifically, when a user starts eating, they use the device (smart glasses) to take a photo of their meal. When the user presses the start button, the smart glasses take two to three photos in succession and record the images. The captured image data is then temporarily processed using edge computing, compressed, and noise-removed before being sent to the server.

[1225] The server analyzes the received food image data. During this process, an image analysis algorithm (e.g., a deep learning model using TensorFlow) runs to identify the meal contents (bread, omelet, salad, etc.) and the amount of each. The results of the identification are saved along with the identification confidence, and the one with the highest confidence is adopted.

[1226] Next, the server retrieves calorie information corresponding to each identified food category from a database (for example, a dietary database such as the USDA or the Japanese Food Composition Tables). Based on the analyzed foods, the calories of each food are added up to calculate the total calorie intake. The calculation result is notified to the device (smartphone), and a message is displayed to the user saying, "The total calories for breakfast are 450 kcal." This data is also saved on the device as a meal history.

[1227] When a user exercises, the device (smartwatch) automatically records exercise data such as the number of steps, heart rate, and exercise time. The smartwatch's sensors collect data in real time and send it to a server every 10 minutes. The server stores the received exercise data for analysis and classifies it by exercise type. For example, this includes exercise data for running, walking, cycling, etc.

[1228] The server calculates the calories burned based on the received exercise data. It calculates the MET (Metabolic Equivalent of Task) value based on the type, intensity, and duration of exercise, as well as the user's physical data (weight, age). For example, it calculates that 230 kcal will be burned for a 30-minute run using a pre-set MET value (8 METs). The calculation result is sent to the device (smartphone), and a message is displayed saying, "You burned 230 kcal running."

[1229] Furthermore, when the user eats or exercises, the device (smartphone) uses a camera and microphone to recognize the user's emotions. Facial recognition software (for example, Microsoft's Face API or Google's Cloud Vision) and voice analysis software are used to identify emotions. The server receives the recognized emotion data and stores it as time-series data. If a specific emotion (such as satisfaction, fatigue, or stress) is recognized frequently, an alert is generated.

[1230] Finally, the server calculates the daily calorie balance based on the calorie intake, calorie expenditure, and emotional data. If the calorie intake is 450 kcal and the calorie expenditure is 230 kcal, the calorie balance is calculated as (450 kcal - 230 kcal = 220 kcal). Based on this calorie balance, advice is generated according to the user's diet goals and appropriate advice is provided to the user. For example, a specific prompt such as "Aim for 200 kcal or less at your next meal" is generated.

[1231] Examples of prompts for using the system include:

[1232] "Calculate calorie balance and generate appropriate diet advice based on today's calorie intake and expenditure, as well as the user's emotional data."

[1233] This allows users to accurately manage their calorie intake and expenditure and receive personalized advice based on their emotional state, making the system a powerful support tool for maintaining a healthy lifestyle.

[1234] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1235] Step 1:

[1236] Input: A picture of the meal when the user starts eating

[1237] Specific operation: When a user starts eating, they use the device (smart glasses) to take a photo of their meal. When the user presses the start button, the smart glasses take two to three photos in succession. The captured image data is temporarily processed using edge computing, where it is compressed and noise-removed.

[1238] Output: The processed meal image data is sent to the server.

[1239] Step 2:

[1240] Input: Food image data sent to the server

[1241] Specific operation: The server analyzes the received food image data. An image analysis algorithm (e.g., a deep learning model using TensorFlow) runs to identify the meal contents (bread, omelet, salad, etc.) and the amount of each. The identification results are saved along with the identification confidence score, and the one with the highest confidence score is adopted.

[1242] Output: Identified food type and quantity data

[1243] Step 3:

[1244] Input: Identified food type and quantity

[1245] Specific operation: The server retrieves calorie information corresponding to each identified food category from a database (e.g., a dietary database such as the USDA or the Japanese Food Composition Table). Based on the analyzed foods, the server calculates the total calorie intake by adding up the calories of each food.

[1246] Output: Total calorie intake data

[1247] Step 4:

[1248] Input: Total calorie intake data

[1249] Specific operation: The server notifies the device (smartphone) of the calculation results. The device displays a message to the user saying, "The total calories for breakfast is 450 kcal." This data is also saved on the device as a meal history.

[1250] Output: Total calorie intake information notified to the user

[1251] Step 5:

[1252] Input: Exercise data when the user exercises (number of steps, heart rate, exercise time, etc.)

[1253] How it works: When a user exercises, the device (smartwatch) automatically records exercise data. The smartwatch's sensors collect data in real time and send it to the server every 10 minutes.

[1254] Output: Exercise data sent to the server

[1255] Step 6:

[1256] Input: Exercise data sent to the server

[1257] Specific operation: The server calculates the calorie consumption based on the received exercise data. It calculates the MET (Metabolic Equivalent of Task) value based on the type, intensity, and duration of exercise, as well as the user's physical data (weight, age). For example, it calculates that 230 kcal will be consumed for a 30-minute run using a pre-set MET value (8 METs).

[1258] Output: Calculated calorie consumption data

[1259] Step 7:

[1260] Input: Calorie consumption data

[1261] Specific operation: The server notifies the terminal (smartphone) of the calculation result. The terminal displays a message to the user saying, "You burned 230 kcal by running."

[1262] Output: Calorie consumption information notified to the user

[1263] Step 8:

[1264] Input: Data showing the user's emotional state (facial expression images and voice data)

[1265] Specific operation: When the user eats or exercises, the device (smartphone) uses a camera and microphone to recognize the user's emotions, using facial recognition software and voice analysis software to identify emotions (satisfaction, fatigue, stress, etc.).

[1266] Output: Recognized emotion data

[1267] Step 9:

[1268] Input: Recognized emotion data

[1269] Specific operation: The server receives the recognized emotion data and generates appropriate advice based on that data. For example, if the user is feeling stressed, advice such as "do some light exercise to relax" or "eat a low-calorie diet" will be provided.

[1270] Output: Advice generated based on emotional state

[1271] Step 10:

[1272] Input: Calories ingested, calories burned, emotional data

[1273] Specific operation: The server calculates the daily calorie balance based on calorie intake, calorie expenditure, and emotional data. Specifically, if calorie intake is 450 kcal and calorie expenditure is 230 kcal, the server calculates the calorie balance (450 kcal - 230 kcal = 220 kcal). As a result, it generates advice based on the diet goal.

[1274] Output: Calorie balance and diet advice data

[1275] In this way, each processing step of this system performs specific data processing and calculation based on input data, and supports health management by providing the results to the user.

[1276] (Application example 2)

[1277] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1278] Many people today lead busy lifestyles, making it difficult to manage their health and diet. Therefore, it is necessary to provide appropriate advice based on the user's emotional state while managing calorie intake through diet and exercise. However, current systems have the challenge of not only managing dietary and exercise data comprehensively, but also incorporating emotional data to provide optimal advice to users.

[1279] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1280] In this invention, the server includes means for acquiring food images, means for analyzing the food images to identify the types and amounts of food, means for calculating calories based on the identified foods, means for acquiring exercise data, means for calculating calories burned based on the exercise data, means for calculating a calorie balance based on calories ingested and calories burned, means for acquiring user emotion data, means for evaluating the user's condition based on the emotion data, means for providing the user with advice regarding diet and exercise based on the calorie balance and emotion data, and means for notifying the user of the advice. This makes it possible to accurately provide appropriate advice based on the user's emotional state while managing calorie intake and calorie expenditure through exercise.

[1281] The "means for acquiring food images" refers to a device that has the function of taking an image of the food that the user eats and saving it as image data or transmitting it to a server.

[1282] The "means for analyzing the food image to identify the type and amount of food" is software or hardware that includes an algorithm for analyzing the captured food image and identifying the type and amount of food in the image.

[1283] A "calorie calculation means" is a means for calculating the calories of an entire meal based on the types and amounts of food identified and using calorie information stored in a database.

[1284] A "means for acquiring exercise data" is a device that records data about the exercise performed by the user (e.g., number of steps, exercise time, heart rate, etc.) and stores or transmits the data to a server.

[1285] The "means for calculating calories burned" is a means for calculating calories burned through exercise by the user based on the acquired exercise data.

[1286] A "means for calculating calorie balance" is a means for calculating daily calorie balance based on calories ingested and calories burned through exercise.

[1287] The "means for acquiring user emotional data" refers to a means for recognizing the emotional state of the user from their facial expressions and voice using sensors such as a camera and a microphone.

[1288] The "means for assessing the user's state" is software or hardware that includes an algorithm for analyzing the acquired emotional data and assessing the user's emotional state.

[1289] The "means for providing diet and exercise advice" is a means for generating and providing optimal diet and exercise advice to a user based on calorie balance and emotion data.

[1290] The "means for notifying the user of the advice" refers to a device or software having an interface for notifying the user of the generated advice.

[1291] The purpose of this invention is to build a system that not only manages the calorie intake of a user and the calorie consumption through exercise, but also provides more appropriate diet advice by utilizing the user's emotional data. This system is configured using the following hardware and software.

[1292] Hardware Configuration

[1293] Smartphone: Obtains the user's emotional data and provides advice. It also displays the total calories of the meal and the calorie balance.

[1294] Smart glasses: When the user eats a meal, they take a picture of the meal and send the image data to a server.

[1295] Smartwatch: Records exercise data (e.g., number of steps, heart rate, and exercise time) when the user exercises and sends it to a server.

[1296] Server: Analyzes food images, calculates calories, calculates calorie expenditure, analyzes emotional data, and generates advice.

[1297] Software Configuration

[1298] Image analysis algorithm: Runs on the server and analyzes the food images sent from the smart glasses to identify the type and quantity of food.

[1299] Calorie calculation engine: Based on the image analysis results, calculates the calories of each food item and calculates the total calories.

[1300] Exercise data analysis engine: Calculates calories burned based on exercise data sent from the smartwatch.

[1301] Emotion recognition engine: Uses the smartphone's camera and microphone to recognize the user's emotional state.

[1302] Advice generation engine: Integrates total calorie intake, calorie expenditure, and emotional data to generate dietary and exercise advice tailored to the user's condition.

[1303] System Operation

[1304] 1. Acquisition and analysis of meal images

[1305] As a user wearing the smart glasses eats, the glasses take images of the meal and send the image data to a server, which uses image analysis algorithms to identify the meal contents and determine the type and quantity of each food item.

[1306] 2. Calorie Counting

[1307] The server calculates the calorie count of each food item based on the type and amount of food identified, and then calculates the total calorie count using a calorie calculation engine. The calculation results are sent to the smartphone and notified to the user.

[1308] 3. Acquisition of exercise data and calculation of calories burned

[1309] When a user wears a smartwatch and exercises, the exercise data is recorded on the smartwatch and periodically sent to a server. The server uses an exercise data analysis engine to calculate the calories burned through exercise and sends the results to a smartphone.

[1310] 4. Acquisition and analysis of emotion data

[1311] Smartphones use cameras and microphones to recognize the user's emotional state. An emotion recognition engine analyzes this and generates emotional data. The generated emotional data is sent to a server.

[1312] 5. Advice Generation

[1313] The server integrates the total calorie intake, calorie expenditure, and emotion data, and generates appropriate diet and exercise advice for the user using an advice generation engine. The generated advice is sent to the user's smartphone and notified.

[1314] Specific examples

[1315] When a user orders lunch through a food delivery app, the following process occurs:

[1316] Prompt Sentence Examples

[1317] I ordered ramen for lunch today. Please check the calorie information and provide appropriate advice based on my exercise data and calorie intake. My current feeling is recognized as "a little tired."

[1318] Specifically, the process is as follows: When a user orders ramen, the app displays the ramen's calorie information (for example, 700 kcal), and calculates the current calorie balance, taking into account the 200 kcal consumed in the previous exercise data. Emotion recognition determines that the user is "slightly tired," and since the calories consumed are sufficient, advice is provided such as "It would be a good idea to order a low-calorie side dish or a protein drink at the same time."

[1319] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1320] Step 1:

[1321] Acquiring food images

[1322] When a user wearing smart glasses eats a meal, the smart glasses take an image of the meal and send the image data to a server. The input is an image of the meal, and the output is the image data of the meal stored on the server. Specifically, when the user starts eating, the camera in the smart glasses automatically activates and takes an image.

[1323] Step 2:

[1324] Food image analysis

[1325] The server uses an image analysis algorithm to analyze the food image data sent from the smart glasses and identify the type and quantity of food. The input is the food image data, and the output is the identified food type and quantity data. Specifically, the server identifies the foods in the image and labels each food category.

[1326] Step 3:

[1327] Calorie Counting

[1328] The server uses a calorie calculation engine to calculate the calories of each food based on the type and amount of the identified food, and then calculates the total calories. The input is data on the type and amount of food, and the output is the total calorie value. Specifically, the server retrieves the calorie information of each food from the database and calculates the total calories based on that information.

[1329] Step 4:

[1330] Calorie information notification

[1331] The server sends the calculated total calorie information to the smartphone and notifies the user. The input is the total calorie value, and the output is the calorie information displayed on the smartphone. Specifically, the smartphone receives the notification and displays the calorie information on the user interface.

[1332] Step 5:

[1333] Acquisition of exercise data

[1334] When a user wears a smartwatch and exercises, the smartwatch records exercise data (e.g., number of steps, heart rate, and exercise time) and periodically transmits it to a server. The input is raw exercise data, and the output is exercise data stored on the server. Specifically, the smartwatch monitors the user's exercise in real time and records and transmits the data.

[1335] Step 6:

[1336] Calorie Calculation

[1337] The server uses an exercise data analysis engine to calculate calories burned based on the exercise data sent from the smartwatch. The input is exercise data, and the output is the calorie burned value. Specifically, the server calculates calories burned based on the exercise intensity, duration, and the user's physical data.

[1338] Step 7:

[1339] Acquiring emotion data

[1340] Smartphones use cameras and microphones to recognize the user's emotional state. The input is raw emotional data (images and audio), and the output is recognized emotional data. Specifically, the smartphone analyzes facial expressions and tone of voice to identify and record the user's emotional state.

[1341] Step 8:

[1342] Emotional Data Analysis

[1343] The server uses an emotion recognition engine to evaluate the user's emotional state based on the emotion data sent from the smartphone. The input is the emotion data, and the output is the evaluated emotional state. Specifically, the server analyzes the emotion data and labels the user's emotional state.

[1344] Step 9:

[1345] Advice Generation

[1346] The server uses an advice generation engine to generate appropriate dietary and exercise advice for the user based on the total calorie intake, calorie expenditure, and emotional data. The input is the calorie intake, calorie expenditure, and emotional data, and the output is the generated advice. Specifically, the server integrates this data and generates advice that is optimal for the user's situation.

[1347] Step 10:

[1348] Advice Notification

[1349] The server sends the generated advice to the smartphone and notifies the user. The input is the generated advice, and the output is the advice displayed on the smartphone. Specifically, the smartphone receives the notification and displays the advice on its user interface.

[1350] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1351] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1352] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1353] [Fourth embodiment]

[1354] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1355] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1356] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1357] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1358] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1359] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1360] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1361] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1362] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1363] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1364] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1365] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1366] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1367] The present invention relates to a system that combines images of food consumed by a user with exercise data to manage the daily calorie balance and provide appropriate diet advice. This system includes means for acquiring food images, analyzing images, calculating calories, acquiring exercise data, calculating calories burned, calculating calorie balance, and providing advice.

[1368] System configuration

[1369] 1. Acquiring food images

[1370] Device (smart glasses): When the user eats, it takes pictures of the meal. When the user presses the start button, the photo of the meal is automatically taken.

[1371] 2. Analysis of food images

[1372] Server: The image data of the photographed meal is sent to the server, which uses image analysis algorithms to identify the meal contents (e.g., bread, omelet, salad) and the amount of each.

[1373] 3. Calorie Counting

[1374] Server: Retrieves calorie information corresponding to each identified food category from the database and calculates the total calories. For example, if bread is 250 kcal, omelet is 150 kcal, and salad is 50 kcal, the total will be calculated as 450 kcal.

[1375] Terminal (smartphone): Notifies the user of the calculation results and saves them as a meal history.

[1376] 4. Acquisition of exercise data

[1377] Device (smartwatch): When a user exercises, it automatically records exercise data such as the number of steps, heart rate, and exercise time. For example, if a user runs for 30 minutes, the data will be recorded.

[1378] Server: Exercise data is periodically sent to the server and stored for analysis.

[1379] 5. Calorie Calculation

[1380] Server: Calculates the calories burned based on the exercise data sent. For example, it calculates that 230 kcal is burned by running for 30 minutes.

[1381] Terminal (smartphone): Notifies the user of the calculation results.

[1382] Specific examples

[1383] Acquiring food information and calculating calories

[1384] User: I eat bread and an omelet for breakfast.

[1385] Terminal (smart glasses): Takes a photo of the meal and sends it to the server.

[1386] Server: Performs image analysis to distinguish between bread and omelet. Calculates the calories as 250 kcal for bread and 150 kcal for omelet, for a total of 400 kcal.

[1387] Device (smartphone): Display the total as 400 kcal to the user and save the data.

[1388] Acquiring exercise information and calculating calories burned

[1389] User: Go for a 30-minute run in the morning.

[1390] Device (smartwatch): Records running exercise data and sends it to the server.

[1391] Server: Calculate the calories burned by running and it is 230 kcal.

[1392] Calculates calorie balance and provides advice

[1393] Server: Based on the calories ingested (400 kcal from breakfast) and calories expended (230 kcal from running), calculate your calorie balance (400 kcal - 230 kcal = 170 kcal).

[1394] Server: Analyzes whether the calorie balance is appropriate for the user's diet goals. If necessary, provides guidance on reducing calorie intake at the next meal.

[1395] Device (smartphone): Display the advice to the user, "Keep your next meal under 200 kcal."

[1396] In this way, users can accurately manage their daily calorie intake and expenditure and receive appropriate advice on maintaining a healthy weight. This system is expected to dramatically improve the success rate of dieting and health management.

[1397] The processing flow will be explained below.

[1398] Step 1:

[1399] Device (smart glasses): Before the user starts eating, they take a photo of their meal. When the user presses a button, the smart glasses automatically take a photo of the meal.

[1400] Step 2:

[1401] Terminal (smart glasses): The captured food images are sent to the server. The image data is often compressed before being transferred.

[1402] Step 3:

[1403] Server: Prepares the received food images for analysis, specifically by pre-processing the images to apply image recognition algorithms.

[1404] Step 4:

[1405] Server: Uses image recognition algorithms to identify the type and quantity of food in a meal image. For example, identify if an image contains bread, an omelet, or a salad.

[1406] Step 5:

[1407] Server: Retrieve calorie information for each identified food type from the database. For example, bread is 250 kcal, omelette is 150 kcal, and salad is 50 kcal.

[1408] Step 6:

[1409] Server: Add up the calories from each food item to calculate your total calorie intake. In this example, we'll use bread, omelet, and salad as a total of 450 kcal.

[1410] Step 7:

[1411] Device (smartphone): The calculated total calories of 450 kcal are notified to the user's smartphone and saved in the meal history.

[1412] Step 8:

[1413] Device (smartwatch): Records exercise data in real time when the user exercises. For example, it records the number of steps, heart rate, and exercise time during a 30-minute run.

[1414] Step 9:

[1415] Device (smartwatch): Periodically sends recorded exercise data to the server.

[1416] Step 10:

[1417] Server: Based on the received exercise data, calculates the calories burned using exercise intensity, time, and the user's physical data (weight, age, etc.). For example, it calculates that 230 kcal is burned by running for 30 minutes.

[1418] Step 11:

[1419] Terminal (smartphone): Notifies the user of the calculated calorie consumption of 230 kcal.

[1420] Step 12:

[1421] Server: Combines the data of calorie intake (450 kcal) and calorie expenditure (230 kcal) to calculate the daily calorie balance. Here, the calorie balance is calculated as 450 kcal - 230 kcal = 220 kcal.

[1422] Step 13:

[1423] Server: Evaluates whether the calorie balance is appropriate for the user's diet goal. For example, if the user has set a daily calorie goal of 0 kcal, this 220 kcal is outside the goal.

[1424] Step 14:

[1425] Server: Generates specific advice about the next meal and exercise based on the calorie balance assessment. For example, it provides advice such as "keep your calorie intake at your next meal to 200 kcal or less."

[1426] Step 15:

[1427] Device (smartphone): Provides advice to the user and instructs them to reflect this in their daily diet and exercise.

[1428] This allows users to accurately manage their calorie intake and expenditure and maintain a healthy lifestyle.

[1429] Example 1

[1430] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1431] Currently, many people struggle to balance dietary calorie management and exercise as part of their health management. Conventional methods require users to manually input meal contents and calculate calories themselves, which is time-consuming and prone to errors. In addition, the collection and analysis of exercise data is not centrally managed, often preventing users from receiving appropriate diet and health guidance. Therefore, there is a need for a system that allows users to easily manage their daily calorie intake and expenditure and maintain healthy lifestyle habits.

[1432] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1433] In this invention, the server includes means for analyzing food images to identify the type and amount of food, means for calculating calories based on the identified food, means for calculating calories burned based on exercise data, means for calculating a calorie balance based on calories ingested and calories burned, and means for providing the user with advice on diet and exercise based on the calorie balance. This allows the user to automatically and accurately grasp the calorie intake from food and the calories burned from exercise, and by receiving appropriate advice based on this, it becomes possible to efficiently maintain a balance between health management and calorie balance.

[1434] A "meal image" is image data captured to record the contents of a meal that a user eats.

[1435] A "terminal" is an electronic device used by a user, such as smart glasses, a smart watch, or a smartphone.

[1436] The "server" is an information processing device that analyzes image data and exercise data, calculates calories, and provides advice.

[1437] "Type and amount of food" refers to the specific food name and portion size identified by analyzing the meal image.

[1438] A "calorie" is a unit that represents the amount of energy contained in food.

[1439] "Exercise data" refers to data such as the type of exercise performed by the user, the number of steps taken, heart rate, and exercise time.

[1440] "Calories burned" is the amount of energy consumed through exercise, calculated based on exercise data.

[1441] "Calories ingested" refers to the amount of energy a user takes in through food.

[1442] "Calorie balance" is the number obtained by subtracting calories consumed from calories consumed.

[1443] "Providing advice" refers to giving instructions or advice to the user regarding diet and exercise based on calorie balance.

[1444] A "natural language processing model" is one of the artificial intelligence techniques used to analyze food images and exercise data.

[1445] "Biometric data" refers to the user's physical information (e.g., height, weight, age, gender, etc.).

[1446] This invention relates to a system that combines images of the food a user eats with exercise data to manage their daily calorie intake and expenditure and provide appropriate diet advice. This system enables users to efficiently manage their daily calorie intake and expenditure and aim to maintain a healthy weight.

[1447] The main hardware components of the system include a device for acquiring food images (e.g., smart glasses), a device for acquiring exercise data (e.g., a smart watch), a device for notifying the user of the analysis results (e.g., a smartphone), and a server for performing data analysis.

[1448] First, the user takes an image of their meal using the smart glasses. When the user presses the start button on the smart glasses, the camera captures the image and sends the image data wirelessly to the server. Next, the server uses an image analysis algorithm (e.g., TensorFlow) to analyze the captured image of the meal. This analysis identifies the contents of the meal and the type and amount of each food item. The analysis results are stored in a database, and the server calculates the total calories based on the calorie information for each food item. For example, if bread is 250 kcal, an omelet is 150 kcal, and a salad is 50 kcal, the total will be calculated as 450 kcal.

[1449] The calculated calorie information is sent from the server to the smartphone and notified to the user. Using the smartphone application, the user can check the calorie intake information and save it as a meal history.

[1450] Furthermore, when a user exercises using a smartwatch, the smartwatch automatically records exercise data such as the number of steps, heart rate, and exercise time. For example, if a user runs for 30 minutes, the data is recorded. The recorded exercise data is sent to a server at regular intervals. Based on the exercise data, the server calculates the calories burned according to the type and intensity of exercise. For example, it calculates that 230 kcal is burned in a 30-minute run. This calorie burn information is also notified from the server to the smartphone, where it is displayed to the user and saved as exercise history.

[1451] Finally, the server calculates the calorie balance based on the calorie intake and calorie expenditure information. For example, if the calorie intake for breakfast is 400 kcal and the calorie expenditure through exercise is 230 kcal, the calorie balance is 400 kcal - 230 kcal = 170 kcal. The server analyzes whether the calorie balance is appropriate for the user's diet goal and generates advice regarding the next meal or exercise as necessary. For example, advice such as "Keep your next meal to 200 kcal or less" is sent from the server to the user's smartphone and notified.

[1452] This specific system allows users to accurately manage their daily calorie intake and expenditure, and receive appropriate advice based on that, enabling them to efficiently manage their health.

[1453] Prompt Sentence Examples

[1454] "I had bread and an omelet for breakfast, so please count the calories."

[1455] "I ran for 30 minutes this morning. How many calories did I burn?"

[1456] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1457] Step 1:

[1458] Acquiring food images

[1459] Device (smart glasses): When the user starts eating, the device presses the start button to take an image of the meal. The image is captured by the smart glasses' camera and saved in the device's internal memory. The image is then sent to the server via wireless communication.

[1460] Input: The action of the user pressing the start button on the device.

[1461] Output: Food image data sent to the server.

[1462] Step 2:

[1463] Food image analysis

[1464] Server: The server processes the received image data using an analysis algorithm (e.g., TensorFlow) to determine the meal contents and portions. This analysis includes the type of food (e.g., bread, omelet, salad) and the portion size of each.

[1465] Input: Food image data sent from the device.

[1466] Output: Data on the type and quantity of food analyzed.

[1467] Step 3:

[1468] Calorie Counting

[1469] Server: Based on the analysis results, the calorie information for each food category is retrieved from a database (e.g., MySQL) and the total calories are calculated. For example, if bread is 250 kcal, an omelet is 150 kcal, and a salad is 50 kcal, the total is calculated as 450 kcal.

[1470] Input: Data on the type and quantity of food parsed.

[1471] Output: Numeric data showing total calories.

[1472] Step 4:

[1473] Calorie information notification

[1474] Terminal (smartphone): The server sends the calculated total calorie information to the terminal and notifies the user. The smartphone application displays the calorie information and saves it in a database as a meal history.

[1475] Input: Numeric data of total calories sent from the server.

[1476] Output: Calorie notification to user and storage in food history database.

[1477] Step 5:

[1478] Acquisition of exercise data

[1479] Device (smartwatch): When the user starts exercising, the device automatically records exercise data such as the number of steps, heart rate, and exercise time. After the exercise is finished, the data is periodically sent to the server.

[1480] Input: User's athletic activity.

[1481] Output: Exercise data sent to the server.

[1482] Step 6:

[1483] Calorie Calculation

[1484] Server: Based on the received exercise data, calculates the calories burned according to the type and intensity of exercise. For example, it calculates that 230 kcal is burned in 30 minutes of running.

[1485] Input: Submitted exercise data.

[1486] Output: Numerical data showing calories burned.

[1487] Step 7:

[1488] Calorie consumption information notification

[1489] Device (smartphone): The server sends the calculated calorie consumption information to the device and notifies the user. The smartphone application displays the calorie consumption information and saves it in a database as exercise history.

[1490] Input: Numerical data of calories burned sent from the server.

[1491] Output: Notify user of calories burned and save to exercise history database.

[1492] Step 8:

[1493] Calculating calorie balance

[1494] Server: Calculates calorie balance using data on calories in and calories out. For example, if your calorie intake is 400 kcal and your calorie expenditure is 230 kcal, your calorie balance will be 170 kcal.

[1495] Input: Numerical data of calories ingested and calories burned.

[1496] Output: Numerical data showing calorie balance.

[1497] Step 9:

[1498] Providing advice

[1499] Server: Based on the calorie balance, the server generates advice based on the user's diet goals. For example, the server may instruct the user to limit their next meal to 200 kcal or less. The advice is sent from the server to the smartphone.

[1500] Input: Calorie balance data.

[1501] Output: Advice notice to user.

[1502] Step 10:

[1503] User Notification and Data Retention

[1504] Terminal (smartphone): Advice is displayed, and once the user confirms it, the record is saved in the database.

[1505] Input: Advice notification sent by the server.

[1506] Output: Advice notification to the user and record saved in the database.

[1507] (Application example 1)

[1508] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1509] In modern society, individual health management is extremely important, but it is not easy to properly manage daily calorie intake and expenditure and maintain a balanced diet. Furthermore, few systems allow users to easily understand their dietary habits and receive appropriate advice based on that information. Furthermore, there is no integration of food delivery services that allow users to immediately order suggested meals. This leads to people living their daily lives without adequate health management, which increases long-term health risks.

[1510] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1511] In this invention, the server includes a means for acquiring food images, a means for analyzing the food images to identify the type and amount of food, a means for calculating calories based on the identified food, a means for acquiring exercise data, a means for calculating calories burned based on the exercise data, a means for calculating a calorie balance based on calories ingested and calories burned, a means for providing the user with dietary and exercise advice based on the calorie balance, and a means for suggesting and ordering a meal menu suitable for a food delivery service based on the advice. This allows users to manage their own dietary and exercise habits in real time and receive appropriate suggestions for maintaining a healthy lifestyle. Furthermore, the suggested meals can be easily ordered, greatly improving user convenience.

[1512] The "means for acquiring food images" is a device that has the function of taking an image of a meal when the user eats and transmitting the image to the system.

[1513] The "means for analyzing a meal image to identify the type and amount of food" is an algorithm or program for analyzing a captured meal image and identifying the type and amount of food contained in the image.

[1514] The "means for calculating calories based on identified foods" refers to a database or calculation mechanism that has the function of calculating the calories of a food based on the type and amount of the analyzed food.

[1515] "Means for acquiring exercise data" refers to a device that has the function of recording data related to the exercise performed by the user (e.g., number of steps, exercise time, heart rate, etc.) and transmitting that data to the system.

[1516] The "means for calculating calories burned based on exercise data" is an algorithm or program that calculates the calories burned by the user based on the acquired exercise data.

[1517] The "means for calculating the calorie balance based on the calorie intake and the calorie expenditure" is a calculation mechanism having a function of calculating the difference between the calories taken in and the calories consumed, and calculating the calorie balance for that day.

[1518] The "means for providing dietary and exercise advice to a user based on calorie balance" is an algorithm or program for suggesting the next meal or recommending exercise to a user based on calorie balance.

[1519] The "means for proposing a meal menu suitable for a food delivery service and ordering that menu" is a system for proposing a meal menu suitable for a user based on the user's calorie balance and ordering that menu through a food delivery service.

[1520] The present invention relates to a system that combines images of meals consumed by a user with exercise data to manage daily calorie intake and provide appropriate meal menus through a food delivery service.

[1521] System configuration details

[1522] 1. Acquiring food images

[1523] Device (smartphone or smart glasses): Takes pictures of the meal while the user is eating. For example, a user takes a photo of their breakfast with their smartphone and uploads it to the application.

[1524] 2. Analysis of food images

[1525] Server: The image data of the photographed meal is sent to the server, which uses an image analysis algorithm (e.g., a machine learning model such as CNN) to identify the meal contents (e.g., bread, omelet, salad) and the amount of each.

[1526] 3. Calorie Counting

[1527] Server: Retrieves calorie information corresponding to each identified food item from the database and calculates the total calories. For example, if bread is 250 kcal, omelet is 150 kcal, and salad is 50 kcal, the total is calculated as 450 kcal.

[1528] 4. Acquisition of exercise data

[1529] Device (smartwatch): When a user exercises, data such as the number of steps, heart rate, and exercise time is automatically recorded. For example, if a user runs for 30 minutes in the morning, the data is recorded.

[1530] Server: Exercise data is periodically sent to the server and stored.

[1531] 5. Calorie Calculation

[1532] Server: Based on the exercise data sent, the server uses machine learning algorithms to calculate the calories burned. For example, it calculates that 30 minutes of running will burn 230 kcal.

[1533] 6. Calculate calorie balance and provide advice

[1534] Server: Calculates the calorie balance based on the identified calorie intake and expenditure (e.g., intake: 450 kcal - expenditure: 230 kcal = 220 kcal).

[1535] Server: Analyzes whether this calorie balance is appropriate for the user's diet goals and health management standards, and provides advice on reducing or increasing calorie intake at the next meal.

[1536] 7. Food delivery service integration

[1537] Server: Based on the generated advice (e.g., "Keep your next meal under 200 kcal"), it proposes a meal menu suitable for the food delivery service and allows the user to order that menu. In this case, it uses a generative AI model to automatically suggest the optimal menu.

[1538] Specific examples

[1539] A user eats bread and an omelet for breakfast, takes a photo of it with their smartphone, and uploads it to the application. The image is analyzed by the server, and the bread is determined to have 250 kcal and the omelet to have 150 kcal. If the user runs for 30 minutes in the morning, 230 kcal will be burned.

[1540] The server uses this data to calculate calorie balance and suggests ways to reduce calories in your next meal, allowing you to quickly order a salad or snack from a food delivery service.

[1541] Prompt Sentence Examples

[1542] A user has uploaded an image of bread and an omelet that they had for breakfast. Calculate the total calories of this meal and calculate their daily calorie balance using their exercise data as input. Then, suggest their next meal based on their calorie balance and order that menu for delivery.

[1543] This system allows users to manage their calorie balance in real time and receive support to maintain a healthy lifestyle, while also significantly improving user convenience.

[1544] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1545] Step 1:

[1546] Acquiring food images

[1547] Input: An image taken by the user while eating.

[1548] How it works: A user takes a picture of their meal using their smartphone or smart glasses and uploads the image to the application.

[1549] Output: Photographed food image data.

[1550] Step 2:

[1551] Food image analysis

[1552] Input: Food image data sent to the server.

[1553] How it works: The server uses an image analysis algorithm (e.g., a machine learning model such as CNN) to identify the types and amounts of food in the image.

[1554] Output: Data on the type and quantity of food identified.

[1555] Step 3:

[1556] Calorie Counting

[1557] Input: Identified food type and quantity data.

[1558] Operation: Based on this food data, the server retrieves the calorie information for each food from the database and calculates the total calories by adding up the calories for each food.

[1559] Output: Total calorie intake data.

[1560] Step 4:

[1561] Acquisition of exercise data

[1562] Input: User's exercise data (e.g., number of steps, heart rate, exercise duration, etc.).

[1563] How it works: A user uses a device such as a smartwatch to record exercise data and periodically sends that data to a server.

[1564] Output: Recording of exercise data.

[1565] Step 5:

[1566] Calorie Calculation

[1567] Input: Exercise data sent to the server.

[1568] How it works: The server uses machine learning algorithms to calculate calories burned based on your exercise data.

[1569] Output: Calories burned data.

[1570] Step 6:

[1571] Calculates calorie balance and provides advice

[1572] Input: Total calorie intake and calorie expenditure data.

[1573] How it works: The server calculates the calorie balance based on the calories ingested and calories burned, and then uses a generative AI model to create advice about the next meal and exercise based on that balance.

[1574] Output: Calorie balance calculation results and advice data.

[1575] Step 7:

[1576] Proposal and ordering for food delivery services

[1577] Input: Generated advice data.

[1578] Operation: The server suggests an appropriate meal menu based on the user's calorie balance, and provides the user with the ability to order that menu from a food delivery service.

[1579] Output: Suggested meal menu and ordering data.

[1580] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1581] The present invention relates to a system that manages daily calorie intake and expenditure by combining images of the food a user consumes, exercise data, and emotional data of the user, and provides appropriate diet advice. This system includes means for acquiring food images, analyzing images, calculating calories, acquiring exercise data, calculating calorie expenditure, an emotional engine, calculating calorie intake and expenditure, and providing advice.

[1582] System configuration

[1583] 1. Acquiring food images

[1584] Device (smart glasses): When the user eats, it takes pictures of the meal. When the user presses the start button, the photo of the meal is automatically taken.

[1585] 2. Analysis of food images

[1586] Server: The image data of the photographed meal is sent to the server, which uses image analysis algorithms to identify the meal contents (e.g., bread, omelet, salad) and the amount of each.

[1587] 3. Calorie Counting

[1588] Server: Retrieves calorie information corresponding to each identified food category from the database and calculates the total calories. For example, if bread is 250 kcal, omelet is 150 kcal, and salad is 50 kcal, the total will be calculated as 450 kcal.

[1589] Terminal (smartphone): Notifies the user of the calculation results and saves them as a meal history.

[1590] 4. Acquisition of exercise data

[1591] Device (smartwatch): When a user exercises, it automatically records exercise data such as the number of steps, heart rate, and exercise time. For example, if a user runs for 30 minutes, the data will be recorded.

[1592] Server: Exercise data is periodically sent to the server and stored for analysis.

[1593] 5. Calorie Calculation

[1594] Server: Based on the exercise data sent, the server calculates the calories burned using the exercise intensity, duration, and the user's physical data (weight, age, etc.). For example, it calculates that 230 kcal is burned by running for 30 minutes.

[1595] Terminal (smartphone): Notifies the user of the calculation results.

[1596] 6. Introducing the Emotion Engine

[1597] Device (smartphone): Recognizes the user's emotions while eating or exercising. The emotion engine uses the camera and microphone to analyze changes in the user's facial expressions and voice to identify emotions.

[1598] Server: The emotion data obtained from the emotion engine is sent to the server.

[1599] Specific examples

[1600] Acquiring food information and calculating calories

[1601] User: I eat bread and an omelet for breakfast.

[1602] Terminal (smart glasses): Takes a photo of the meal and sends it to the server.

[1603] Server: Performs image analysis to distinguish between bread and omelet. Calculates the calories as 250 kcal for bread and 150 kcal for omelet, for a total of 400 kcal.

[1604] Device (smartphone): Display the total as 400 kcal to the user and save the data.

[1605] Acquiring exercise information and calculating calories burned

[1606] User: Go for a 30-minute run in the morning.

[1607] Device (smartwatch): Records running exercise data and sends it to the server.

[1608] Server: Calculate the calories burned by running and it is 230 kcal.

[1609] Emotion recognition and advice provision

[1610] User: Feels certain emotions (e.g., satisfaction, fatigue, stress) after eating or during exercise.

[1611] Device (smartphone): Recognizes user emotions using a camera and microphone.

[1612] Server: The recognized emotion data is sent to the server.

[1613] Server: Calculates calorie balance based on calorie intake, calorie expenditure, and emotional data, and generates advice. For example, if a user is feeling stressed, it provides advice on diet and exercise to help them relax.

[1614] Calculates calorie balance and provides advice

[1615] Server: Combines the data of calorie intake (400 kcal) and calorie expenditure (230 kcal) to calculate the daily calorie balance. Here, the calorie balance is calculated as 400 kcal - 230 kcal = 170 kcal.

[1616] Server: Evaluates whether the calorie balance is appropriate for the user's diet goal. If necessary, generates specific advice about the next meal or exercise. For example, it provides advice such as "Keep your calorie intake to 200 kcal or less at your next meal."

[1617] Device (smartphone): Provides advice to the user and instructs them to reflect this in their daily diet and exercise.

[1618] This allows users to accurately manage their calorie intake and expenditure and receive appropriate advice based on their emotional state, making the system a powerful support tool for maintaining a healthy lifestyle.

[1619] The processing flow will be explained below.

[1620] Step 1:

[1621] Device (smart glasses): Before the user starts eating, they take a photo of the meal. When the user presses the start button, the smart glasses automatically take a photo of the meal.

[1622] Step 2:

[1623] Terminal (smart glasses): The captured food images are sent to the server. The image data is often compressed before being transferred.

[1624] Step 3:

[1625] Server: Prepares the received food images for analysis, specifically by preprocessing the images (noise removal, resolution adjustment, etc.) to apply image recognition algorithms.

[1626] Step 4:

[1627] Server: Uses image recognition algorithms to identify the type and quantity of food in a meal image. For example, identify if an image contains bread, an omelet, or a salad.

[1628] Step 5:

[1629] Server: Retrieve calorie information for each identified food type from the database. For example, bread is 250 kcal, omelette is 150 kcal, and salad is 50 kcal.

[1630] Step 6:

[1631] Server: Add up the calories from each food item to calculate your total calorie intake. In this example, we'll use bread, omelet, and salad as a total of 450 kcal.

[1632] Step 7:

[1633] Device (smartphone): The calculated total calories of 450 kcal are notified to the user's smartphone and saved in the meal history.

[1634] Step 8:

[1635] Device (smartwatch): Records exercise data in real time when the user exercises. For example, it records the number of steps, heart rate, and exercise time during a 30-minute run.

[1636] Step 9:

[1637] Device (smartwatch): Periodically sends recorded exercise data to the server.

[1638] Step 10:

[1639] Server: Based on the received exercise data, calculates the calories burned using exercise intensity, time, and the user's physical data (weight, age, etc.). For example, it calculates that 230 kcal is burned by running for 30 minutes.

[1640] Step 11:

[1641] Terminal (smartphone): Notifies the user of the calculated calorie consumption of 230 kcal.

[1642] Step 12:

[1643] Server: Combines the data of calorie intake (450 kcal) and calorie expenditure (230 kcal) to calculate the daily calorie balance. Here, the calorie balance is calculated as 450 kcal - 230 kcal = 220 kcal.

[1644] Step 13:

[1645] Device (smartphone): When the user eats or exercises, the emotion engine uses the camera and microphone to recognize the user's emotions.

[1646] Step 14:

[1647] Server: Emotion data obtained from the emotion engine is sent to the server. For example, it is recognized that the user is feeling stressed.

[1648] Step 15:

[1649] Server: Adapts advice content based on calorie balance and emotional data. For example, it provides advice on relaxation-enhancing diets and exercise to a user who is feeling stressed.

[1650] Step 16:

[1651] Device (smartphone): Notifies the user of the adapted advice. For example, display advice such as "Keep your calorie intake to 200 kcal or less at your next meal and do yoga to relax."

[1652] This series of processes allows users to accurately manage their calorie intake and expenditure, and receive appropriate advice based on their emotional state, making it easier to maintain a healthy lifestyle.

[1653] Example 2

[1654] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1655] Conventional calorie balance management systems collect and analyze users' dietary and exercise data, but no system provides advice that takes into account the user's emotional state. Therefore, our goal is to make it easier for users to maintain a healthier lifestyle by providing more personalized advice based on the user's emotional state in addition to dietary and exercise data.

[1656] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1657] In this invention, the server includes means for analyzing meal images to identify the type and amount of food, means for calculating calories based on the identified food, means for calculating calories burned based on exercise data, means for calculating calorie balance based on calories ingested and calories burned, means for recognizing and analyzing the user's emotions, and means for providing appropriate advice to the user based on the emotion data, thereby making it possible to provide detailed and personalized health advice according to the user's emotional state.

[1658] The term "means for acquiring food images" refers to a device or method for taking an image of the food a user eats and recording that image.

[1659] "Means for analyzing meal images to identify food types and quantities" refers to an algorithm or program for analyzing captured meal images and identifying the type and quantity of each food item in the image.

[1660] "Means for calculating calories" refers to a method or program for calculating total calorie intake by adding up the calories of each food based on the type and amount of food analyzed.

[1661] "Means for acquiring exercise data" refers to a device or method for recording data related to the exercise performed by a user (e.g., number of steps, heart rate, exercise time, etc.) and saving that data.

[1662] "Means for calculating calories burned" refers to a program or algorithm for calculating the amount of energy consumed by a user's exercise based on the acquired exercise data.

[1663] "Means for calculating calorie balance" refers to a method or program for calculating daily calorie balance based on calories consumed and calories burned.

[1664] "Means for providing dietary and exercise advice to a user" refers to a device or method that provides specific instructions or suggestions to a user regarding their next meal or exercise based on the calculated calorie balance.

[1665] "Means for recognizing and analyzing user emotions" refers to programs or algorithms that identify a user's emotional state using a camera or microphone and analyze that data.

[1666] "Means for providing appropriate advice to a user based on emotional data" refers to a device or method for providing more personalized health advice based on calorie intake, calorie expenditure, and calorie balance, taking into account the user's emotional state.

[1667] This invention is a system that manages daily calorie balance and provides appropriate diet advice by combining images of the food a user consumes, exercise data, and the user's emotional data. This system includes means for acquiring food images, analyzing images, calculating calories, acquiring exercise data, calculating calories burned, an emotional engine, calculating calorie balance, and providing advice.

[1668] Specifically, when a user starts eating, they use the device (smart glasses) to take a photo of their meal. When the user presses the start button, the smart glasses take two to three photos in succession and record the images. The captured image data is then temporarily processed using edge computing, compressed, and noise-removed before being sent to the server.

[1669] The server analyzes the received food image data. During this process, an image analysis algorithm (e.g., a deep learning model using TensorFlow) runs to identify the meal contents (bread, omelet, salad, etc.) and the amount of each. The results of the identification are saved along with the identification confidence, and the one with the highest confidence is adopted.

[1670] Next, the server retrieves calorie information corresponding to each identified food category from a database (for example, a dietary database such as the USDA or the Japanese Food Composition Tables). Based on the analyzed foods, the calories of each food are added up to calculate the total calorie intake. The calculation result is notified to the device (smartphone), and a message is displayed to the user saying, "The total calories for breakfast are 450 kcal." This data is also saved on the device as a meal history.

[1671] When a user exercises, the device (smartwatch) automatically records exercise data such as the number of steps, heart rate, and exercise time. The smartwatch's sensors collect data in real time and send it to a server every 10 minutes. The server stores the received exercise data for analysis and classifies it by exercise type. For example, this includes exercise data for running, walking, cycling, etc.

[1672] The server calculates the calories burned based on the received exercise data. It calculates the MET (Metabolic Equivalent of Task) value based on the type, intensity, and duration of exercise, as well as the user's physical data (weight, age). For example, it calculates that 230 kcal will be burned for a 30-minute run using a pre-set MET value (8 METs). The calculation result is sent to the device (smartphone), and a message is displayed saying, "You burned 230 kcal running."

[1673] Furthermore, when the user eats or exercises, the device (smartphone) uses a camera and microphone to recognize the user's emotions. Facial recognition software (for example, Microsoft's Face API or Google's Cloud Vision) and voice analysis software are used to identify emotions. The server receives the recognized emotion data and stores it as time-series data. If a specific emotion (such as satisfaction, fatigue, or stress) is recognized frequently, an alert is generated.

[1674] Finally, the server calculates the daily calorie balance based on the calorie intake, calorie expenditure, and emotional data. If the calorie intake is 450 kcal and the calorie expenditure is 230 kcal, the calorie balance is calculated as (450 kcal - 230 kcal = 220 kcal). Based on this calorie balance, advice is generated according to the user's diet goals and appropriate advice is provided to the user. For example, a specific prompt such as "Aim for 200 kcal or less at your next meal" is generated.

[1675] Examples of prompts for using the system include:

[1676] "Calculate calorie balance and generate appropriate diet advice based on today's calorie intake and expenditure, as well as the user's emotional data."

[1677] This allows users to accurately manage their calorie intake and expenditure and receive personalized advice based on their emotional state, making the system a powerful support tool for maintaining a healthy lifestyle.

[1678] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1679] Step 1:

[1680] Input: A picture of the meal when the user starts eating

[1681] Specific operation: When a user starts eating, they use the device (smart glasses) to take a photo of their meal. When the user presses the start button, the smart glasses take two to three photos in succession. The captured image data is temporarily processed using edge computing, where it is compressed and noise-removed.

[1682] Output: The processed meal image data is sent to the server.

[1683] Step 2:

[1684] Input: Food image data sent to the server

[1685] Specific operation: The server analyzes the received food image data. An image analysis algorithm (e.g., a deep learning model using TensorFlow) runs to identify the meal contents (bread, omelet, salad, etc.) and the amount of each. The identification results are saved along with the identification confidence score, and the one with the highest confidence score is adopted.

[1686] Output: Identified food type and quantity data

[1687] Step 3:

[1688] Input: Identified food type and quantity

[1689] Specific operation: The server retrieves calorie information corresponding to each identified food category from a database (e.g., a dietary database such as the USDA or the Japanese Food Composition Table). Based on the analyzed foods, the server calculates the total calorie intake by adding up the calories of each food.

[1690] Output: Total calorie intake data

[1691] Step 4:

[1692] Input: Total calorie intake data

[1693] Specific operation: The server notifies the device (smartphone) of the calculation results. The device displays a message to the user saying, "The total calories for breakfast is 450 kcal." This data is also saved on the device as a meal history.

[1694] Output: Total calorie intake information notified to the user

[1695] Step 5:

[1696] Input: Exercise data when the user exercises (number of steps, heart rate, exercise time, etc.)

[1697] How it works: When a user exercises, the device (smartwatch) automatically records exercise data. The smartwatch's sensors collect data in real time and send it to the server every 10 minutes.

[1698] Output: Exercise data sent to the server

[1699] Step 6:

[1700] Input: Exercise data sent to the server

[1701] Specific operation: The server calculates the calorie consumption based on the received exercise data. It calculates the MET (Metabolic Equivalent of Task) value based on the type, intensity, and duration of exercise, as well as the user's physical data (weight, age). For example, it calculates that 230 kcal will be consumed for a 30-minute run using a pre-set MET value (8 METs).

[1702] Output: Calculated calorie consumption data

[1703] Step 7:

[1704] Input: Calorie consumption data

[1705] Specific operation: The server notifies the terminal (smartphone) of the calculation result. The terminal displays a message to the user saying, "You burned 230 kcal by running."

[1706] Output: Calorie consumption information notified to the user

[1707] Step 8:

[1708] Input: Data showing the user's emotional state (facial expression images and voice data)

[1709] Specific operation: When the user eats or exercises, the device (smartphone) uses a camera and microphone to recognize the user's emotions, using facial recognition software and voice analysis software to identify emotions (satisfaction, fatigue, stress, etc.).

[1710] Output: Recognized emotion data

[1711] Step 9:

[1712] Input: Recognized emotion data

[1713] Specific operation: The server receives the recognized emotion data and generates appropriate advice based on that data. For example, if the user is feeling stressed, advice such as "do some light exercise to relax" or "eat a low-calorie diet" will be provided.

[1714] Output: Advice generated based on emotional state

[1715] Step 10:

[1716] Input: Calories ingested, calories burned, emotional data

[1717] Specific operation: The server calculates the daily calorie balance based on calorie intake, calorie expenditure, and emotional data. Specifically, if calorie intake is 450 kcal and calorie expenditure is 230 kcal, the server calculates the calorie balance (450 kcal - 230 kcal = 220 kcal). As a result, it generates advice based on the diet goal.

[1718] Output: Calorie balance and diet advice data

[1719] In this way, each processing step of this system performs specific data processing and calculation based on input data, and supports health management by providing the results to the user.

[1720] (Application example 2)

[1721] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1722] Many people today lead busy lifestyles, making it difficult to manage their health and diet. Therefore, it is necessary to provide appropriate advice based on the user's emotional state while managing calorie intake through diet and exercise. However, current systems have the challenge of not only managing dietary and exercise data comprehensively, but also incorporating emotional data to provide optimal advice to users.

[1723] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1724] In this invention, the server includes means for acquiring food images, means for analyzing the food images to identify the types and amounts of food, means for calculating calories based on the identified foods, means for acquiring exercise data, means for calculating calories burned based on the exercise data, means for calculating a calorie balance based on calories ingested and calories burned, means for acquiring user emotion data, means for evaluating the user's condition based on the emotion data, means for providing the user with advice regarding diet and exercise based on the calorie balance and emotion data, and means for notifying the user of the advice. This makes it possible to accurately provide appropriate advice based on the user's emotional state while managing calorie intake and calorie expenditure through exercise.

[1725] The "means for acquiring food images" refers to a device that has the function of taking an image of the food that the user eats and saving it as image data or transmitting it to a server.

[1726] The "means for analyzing the food image to identify the type and amount of food" is software or hardware that includes an algorithm for analyzing the captured food image and identifying the type and amount of food in the image.

[1727] A "calorie calculation means" is a means for calculating the calories of an entire meal based on the types and amounts of food identified and using calorie information stored in a database.

[1728] A "means for acquiring exercise data" is a device that records data about the exercise performed by the user (e.g., number of steps, exercise time, heart rate, etc.) and stores or transmits the data to a server.

[1729] The "means for calculating calories burned" is a means for calculating calories burned through exercise by the user based on the acquired exercise data.

[1730] A "means for calculating calorie balance" is a means for calculating daily calorie balance based on calories ingested and calories burned through exercise.

[1731] The "means for acquiring user emotional data" refers to a means for recognizing the emotional state of the user from their facial expressions and voice using sensors such as a camera and a microphone.

[1732] The "means for assessing the user's state" is software or hardware that includes an algorithm for analyzing the acquired emotional data and assessing the user's emotional state.

[1733] The "means for providing diet and exercise advice" is a means for generating and providing optimal diet and exercise advice to a user based on calorie balance and emotion data.

[1734] The "means for notifying the user of the advice" refers to a device or software having an interface for notifying the user of the generated advice.

[1735] The purpose of this invention is to build a system that not only manages the calorie intake of a user and the calorie consumption through exercise, but also provides more appropriate diet advice by utilizing the user's emotional data. This system is configured using the following hardware and software.

[1736] Hardware Configuration

[1737] Smartphone: Obtains the user's emotional data and provides advice. It also displays the total calories of the meal and the calorie balance.

[1738] Smart glasses: When the user eats a meal, they take a picture of the meal and send the image data to a server.

[1739] Smartwatch: Records exercise data (e.g., number of steps, heart rate, and exercise time) when the user exercises and sends it to a server.

[1740] Server: Analyzes food images, calculates calories, calculates calorie expenditure, analyzes emotional data, and generates advice.

[1741] Software Configuration

[1742] Image analysis algorithm: Runs on the server and analyzes the food images sent from the smart glasses to identify the type and quantity of food.

[1743] Calorie calculation engine: Based on the image analysis results, calculates the calories of each food item and calculates the total calories.

[1744] Exercise data analysis engine: Calculates calories burned based on exercise data sent from the smartwatch.

[1745] Emotion recognition engine: Uses the smartphone's camera and microphone to recognize the user's emotional state.

[1746] Advice generation engine: Integrates total calorie intake, calorie expenditure, and emotional data to generate dietary and exercise advice tailored to the user's condition.

[1747] System Operation

[1748] 1. Acquisition and analysis of meal images

[1749] As a user wearing the smart glasses eats, the glasses take images of the meal and send the image data to a server, which uses image analysis algorithms to identify the meal contents and determine the type and quantity of each food item.

[1750] 2. Calorie Counting

[1751] The server calculates the calorie count of each food item based on the type and amount of food identified, and then calculates the total calorie count using a calorie calculation engine. The calculation results are sent to the smartphone and notified to the user.

[1752] 3. Acquisition of exercise data and calculation of calories burned

[1753] When a user wears a smartwatch and exercises, the exercise data is recorded on the smartwatch and periodically sent to a server. The server uses an exercise data analysis engine to calculate the calories burned through exercise and sends the results to a smartphone.

[1754] 4. Acquisition and analysis of emotion data

[1755] Smartphones use cameras and microphones to recognize the user's emotional state. An emotion recognition engine analyzes this and generates emotional data. The generated emotional data is sent to a server.

[1756] 5. Advice Generation

[1757] The server integrates the total calorie intake, calorie expenditure, and emotion data, and generates appropriate diet and exercise advice for the user using an advice generation engine. The generated advice is sent to the user's smartphone and notified.

[1758] Specific examples

[1759] When a user orders lunch through a food delivery app, the following process occurs:

[1760] Prompt Sentence Examples

[1761] I ordered ramen for lunch today. Please check the calorie information and provide appropriate advice based on my exercise data and calorie intake. My current feeling is recognized as "a little tired."

[1762] Specifically, the process is as follows: When a user orders ramen, the app displays the ramen's calorie information (for example, 700 kcal), and calculates the current calorie balance, taking into account the 200 kcal consumed in the previous exercise data. Emotion recognition determines that the user is "slightly tired," and since the calories consumed are sufficient, advice is provided such as "It would be a good idea to order a low-calorie side dish or a protein drink at the same time."

[1763] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1764] Step 1:

[1765] Acquiring food images

[1766] When a user wearing smart glasses eats a meal, the smart glasses take an image of the meal and send the image data to a server. The input is an image of the meal, and the output is the image data of the meal stored on the server. Specifically, when the user starts eating, the camera in the smart glasses automatically activates and takes an image.

[1767] Step 2:

[1768] Food image analysis

[1769] The server uses an image analysis algorithm to analyze the food image data sent from the smart glasses and identify the type and quantity of food. The input is the food image data, and the output is the identified food type and quantity data. Specifically, the server identifies the foods in the image and labels each food category.

[1770] Step 3:

[1771] Calorie Counting

[1772] The server uses a calorie calculation engine to calculate the calories of each food based on the type and amount of the identified food, and then calculates the total calories. The input is data on the type and amount of food, and the output is the total calorie value. Specifically, the server retrieves the calorie information of each food from the database and calculates the total calories based on that information.

[1773] Step 4:

[1774] Calorie information notification

[1775] The server sends the calculated total calorie information to the smartphone and notifies the user. The input is the total calorie value, and the output is the calorie information displayed on the smartphone. Specifically, the smartphone receives the notification and displays the calorie information on the user interface.

[1776] Step 5:

[1777] Acquisition of exercise data

[1778] When a user wears a smartwatch and exercises, the smartwatch records exercise data (e.g., number of steps, heart rate, and exercise time) and periodically transmits it to a server. The input is raw exercise data, and the output is exercise data stored on the server. Specifically, the smartwatch monitors the user's exercise in real time and records and transmits the data.

[1779] Step 6:

[1780] Calorie Calculation

[1781] The server uses an exercise data analysis engine to calculate calories burned based on the exercise data sent from the smartwatch. The input is exercise data, and the output is the calorie burned value. Specifically, the server calculates calories burned based on the exercise intensity, duration, and the user's physical data.

[1782] Step 7:

[1783] Acquiring emotion data

[1784] Smartphones use cameras and microphones to recognize the user's emotional state. The input is raw emotional data (images and audio), and the output is recognized emotional data. Specifically, the smartphone analyzes facial expressions and tone of voice to identify and record the user's emotional state.

[1785] Step 8:

[1786] Emotional Data Analysis

[1787] The server uses an emotion recognition engine to evaluate the user's emotional state based on the emotion data sent from the smartphone. The input is the emotion data, and the output is the evaluated emotional state. Specifically, the server analyzes the emotion data and labels the user's emotional state.

[1788] Step 9:

[1789] Advice Generation

[1790] The server uses an advice generation engine to generate appropriate dietary and exercise advice for the user based on the total calorie intake, calorie expenditure, and emotional data. The input is the calorie intake, calorie expenditure, and emotional data, and the output is the generated advice. Specifically, the server integrates this data and generates advice that is optimal for the user's situation.

[1791] Step 10:

[1792] Advice Notification

[1793] The server sends the generated advice to the smartphone and notifies the user. The input is the generated advice, and the output is the advice displayed on the smartphone. Specifically, the smartphone receives the notification and displays the advice on its user interface.

[1794] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1795] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1796] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1797] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1798] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1799] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1800] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1801] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1802] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1803] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1804] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1805] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1806] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1807] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1808] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1809] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1810] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1811] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1812] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1813] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1814] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1815] The following is further disclosed regarding the above embodiment.

[1816] (Claim 1)

[1817] A means for acquiring a meal image;

[1818] means for analyzing the meal image to identify the type and amount of food;

[1819] means for calculating calories based on the identified foods;

[1820] a means for acquiring exercise data;

[1821] means for calculating calories burned based on the exercise data;

[1822] A means for calculating a calorie balance based on calorie intake and calorie expenditure;

[1823] means for providing dietary and exercise advice to a user based on said calorie balance.

[1824] (Claim 2)

[1825] 10. The system of claim 1, further comprising means for periodically transmitting the athletic data.

[1826] (Claim 3)

[1827] 10. The system of claim 1, further comprising means for calculating calories burned based on the user's physical data.

[1828] "Example 1"

[1829] (Claim 1)

[1830] A terminal means for acquiring food images;

[1831] a server means for analyzing the food image and identifying the type and amount of food;

[1832] a server means for calculating calories based on the identified food;

[1833] a terminal means for acquiring exercise data;

[1834] a server means for calculating calories burned based on the exercise data;

[1835] A server means for calculating a calorie balance based on calorie intake and calorie expenditure;

[1836] a terminal means for providing advice on diet and exercise to a user based on the calorie balance;

[1837] means for analyzing the food image using a natural language processing model;

[1838] The system includes means for periodically transmitting the exercise data to a server.

[1839] (Claim 2)

[1840] 2. The system according to claim 1, further comprising a terminal means for periodically transmitting the exercise data.

[1841] (Claim 3)

[1842] 10. The system according to claim 1, further comprising means for calculating calories burned based on biometric data of the user.

[1843] "Application Example 1"

[1844] (Claim 1)

[1845] A means for acquiring a meal image;

[1846] means for analyzing the meal image to identify the type and amount of food;

[1847] means for calculating calories based on the identified foods;

[1848] a means for acquiring exercise data;

[1849] means for calculating calories burned based on the exercise data;

[1850] A means for calculating a calorie balance based on calorie intake and calorie expenditure;

[1851] means for providing dietary and exercise advice to a user based on said calorie balance;

[1852] Based on the advice, a meal menu suitable for the food delivery service is proposed, and a method for ordering the menu is provided.

[1853] Including system.

[1854] (Claim 2)

[1855] 10. The system of claim 1, further comprising means for periodically transmitting the athletic data.

[1856] (Claim 3)

[1857] 10. The system of claim 1, further comprising means for calculating calories burned based on the user's physical data.

[1858] "Example 2: Combining Emotion Engines"

[1859] (Claim 1)

[1860] A means for acquiring a meal image;

[1861] means for analyzing the meal image to identify the type and amount of food;

[1862] means for calculating calories based on the identified foods;

[1863] a means for acquiring exercise data;

[1864] means for calculating calories burned based on the exercise data;

[1865] A means for calculating a calorie balance based on calorie intake and calorie expenditure;

[1866] means for providing dietary and exercise advice to a user based on said calorie balance;

[1867] means for recognizing and analyzing user emotions;

[1868] The system includes a means for providing appropriate advice to the user based on the emotion data.

[1869] (Claim 2)

[1870] 10. The system of claim 1, further comprising means for periodically transmitting the athletic data.

[1871] (Claim 3)

[1872] 10. The system of claim 1, further comprising means for calculating calories burned based on the user's physical data.

[1873] "Application example 2 when combining emotion engines"

[1874] (Claim 1)

[1875] A means for acquiring a meal image;

[1876] means for analyzing the meal image to identify the type and amount of food;

[1877] means for calculating calories based on the identified foods;

[1878] a means for acquiring exercise data;

[1879] means for calculating calories burned based on the exercise data;

[1880] A means for calculating a calorie balance based on calorie intake and calorie expenditure;

[1881] A means for acquiring user emotion data;

[1882] means for evaluating a state of a user based on the emotion data;

[1883] means for providing dietary and exercise advice to a user based on said calorie balance and emotional data;

[1884] The system includes means for notifying a user of the advice.

[1885] (Claim 2)

[1886] 10. The system of claim 1, further comprising means for periodically transmitting the athletic data.

[1887] (Claim 3)

[1888] 10. The system of claim 1, further comprising means for calculating calories burned based on the user's physical data. [Explanation of symbols]

[1889] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for acquiring a meal image; means for analyzing the meal image to identify the type and amount of food; means for calculating calories based on the identified foods; a means for acquiring exercise data; means for calculating calories burned based on the exercise data; A means for calculating a calorie balance based on calorie intake and calorie expenditure; means for providing dietary and exercise advice to a user based on said calorie balance.

2. 10. The system of claim 1, further comprising means for periodically transmitting the athletic data.

3. The system according to claim 1 , further comprising means for calculating calories burned based on the user's physical data.

Images