system

Abstract

Provide a system. 【Solution means】Means for the user to take a picture of a meal, Means for uploading the image to the server, Means for analyzing the image in the server to obtain food ingredient information and calorie information, Means for sending the calorie information to the user's terminal, Means for displaying the calorie information on the terminal, Means for the server to propose a menu for the next day based on the user's meal history and health goals, Means for sending the menu information for the next day to the user's terminal, [[ID=十六]]Means for displaying the menu information on the terminal, Means for calculating the required amount of exercise in the server and sending an exercise plan to the user's terminal, Means for displaying the exercise plan on the terminal A system including.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, the method including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In modern society, due to busy lifestyles and irregular eating patterns, it is difficult to maintain a healthy diet. Also, many people find it difficult to understand and manage the calories and nutritional balance of their diet. Furthermore, it is not easy to know the appropriate amount of exercise, making health management even more complicated. There is a need for a system that can solve such problems and enable users to easily practice a healthy diet.

Means for Solving the Problems

[0005] This invention relates to a system that automatically calculates calories based on a user's photograph of a meal. The system includes the following means: First, it provides means for the user to take a photograph of a meal and upload that photograph to a server. Next, the server analyzes the photograph using an image analysis API to obtain ingredient information and calorie information. The obtained calorie information is sent to the user's terminal, and the terminal is equipped with means to display this information. Furthermore, it includes means to suggest a menu for the next day based on the user's meal history and health goals, and to send and display the suggestion to the user's terminal. Finally, the server calculates the required amount of exercise and provides means to send and display an exercise plan to the user's terminal, thereby comprehensively supporting the user's health management.

[0006] "User" refers to an individual or group that uses the system.

[0007] "Means of taking an image" refers to the act or device of taking a photograph of a meal using a device with a camera function.

[0008] "Means of uploading images to a server" refers to the act or function of sending image data from a device to a server via the internet.

[0009] A "server" refers to a computer system that provides data processing and storage functions over the internet.

[0010] "Means of analyzing images" refers to software or algorithms used to analyze images of food and extract information about the ingredients.

[0011] "Ingredient information" refers to the types of ingredients included in the image and their detailed information.

[0012] "Calorie information" refers to the energy value calculated based on the type and quantity of food ingredients.

[0013] "User's device" refers to information processing devices used by the user, such as smartphones, tablets, and personal computers.

[0014] "Metadata" refers to additional information related to image data (e.g., user ID, timestamp, etc.).

[0015] "Health goals" refer to health-related objectives set by the user (e.g., weight loss, muscle gain, etc.).

[0016] "A means of suggesting the next day's menu" refers to a function that calculates and suggests recommended meal menus for the following day and beyond, based on the user's past eating history and health goals.

[0017] "Means for calculating exercise volume" refers to a function that calculates the necessary calorie expenditure based on the user's diet and goals, and then calculates an appropriate exercise plan.

[0018] An "exercise plan" refers to the specific types and duration of exercises designed to achieve calorie expenditure. [Brief explanation of the drawing]

[0019] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7]It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which a plurality of emotions are mapped. [Figure 10] It shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when an emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when an emotion engine is combined.

Mode for Carrying Out the Invention

[0020] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0021] First, the terms used in the following description will be explained.

[0022] In the following embodiments, the signed processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Furthermore, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include CPU (Central Processing Unit), GPU (Graphics Processing Unit), GPGPU (General-Purpose computing on Graphics Processing Units), and APU (Accelerated Processing Unit).

[0023] In the following embodiments, signed RAM (Random Access Memory) is a memory that temporarily stores information and is used as work memory by the processor.

[0024] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0025] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0026] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0027] [First Embodiment]

[0028] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0029] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0030] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0031] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0032] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0033] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0034] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0035] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0036] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0037] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0038] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0039] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0040] This invention relates to a system in which a user takes a picture of their meal, which is then automatically calculated based on the image, and which also provides menu suggestions for the following days and guidance on the amount of exercise needed. This system supports the user's healthy eating habits through communication between a server and the user's terminal.

[0041] overview

[0042] Users take photos of their meals using devices such as smartphones or tablets. These photos are uploaded to a server. The server uses image analysis technology to recognize the ingredients in the images and obtain their calorie information. Furthermore, the server suggests appropriate menus and exercise plans for the following day based on the user's health goals.

[0043] Program Description

[0044] 1. Take a photo of your meal and upload the image.

[0045] The user takes a photo of their meal with their device. For example, they might take a photo of their breakfast toast, eggs, and vegetable salad.

[0046] The device reads the captured image from local storage and uploads it to the server along with metadata such as the user's ID and timestamp.

[0047] 2. Image analysis and calorie calculation

[0048] The server receives the uploaded image and calls an image analysis API (e.g., Google® Vision API) to obtain information about the ingredients.

[0049] Based on the list of ingredients returned from the API (e.g., "toast", "eggs", "vegetable salad"), the calorie information for each ingredient is retrieved from the database.

[0050] Add up the calories of each ingredient to calculate the total calories. For example, if you have toast (150kcal), an egg (70kcal), and a vegetable salad (30kcal), the total calories would be 250kcal.

[0051] 3. Display of calorie information

[0052] The server sends the calculated calorie information to the terminal in JSON format.

[0053] The device parses the received JSON data and displays calorie information to the user. Example: "This morning's breakfast was 250 kcal."

[0054] 4. Menu suggestions for the following day

[0055] The server retrieves the user's meal history from the database based on their ID. Furthermore, it takes into account the user's health goals (e.g., dieting).

[0056] Using a machine learning model, the system generates a menu for the next day based on the user's past eating history and health goals. For example, it might suggest a low-calorie, high-protein meal.

[0057] The generated menu information for the following day is sent to the terminal in JSON format.

[0058] The terminal parses the received menu information and displays it to the user. Example: "For tomorrow's menu, we recommend yogurt and fruit for breakfast, chicken salad for lunch, and vegetable soup for dinner."

[0059] 5. Guidance on recommended exercise levels

[0060] The server retrieves the total calorie intake for the day from the database. For example, if the intake is 1800kcal, it calculates the difference of 300kcal between that and the target intake of 1500kcal.

[0061] Based on the difference, calculate an exercise plan to burn 300kcal. Example: 30 minutes of jogging, 60 minutes of walking.

[0062] A recommended exercise plan will be sent to your device in JSON format.

[0063] The device parses the received exercise plan and displays it to the user. Example: "To burn off today's calorie intake, we recommend 30 minutes of jogging or 60 minutes of walking."

[0064] This system allows users to obtain appropriate calorie information from their meals and, furthermore, suggests menus and exercise plans for the next day based on their health goals, enabling them to live a healthy lifestyle.

[0065] The following describes the processing flow.

[0066] Step 1:

[0067] The user takes a photo of the meal with their device. The user takes a photo of the meal using the device's camera function, launches the application, and selects the photo.

[0068] Step 2:

[0069] The device reads photos taken from local storage and generates metadata such as the user's ID and timestamp. This integrates the photos with related information.

[0070] Step 3:

[0071] The device sends the photo and metadata to the server as an HTTP POST request. The data is securely transferred to the server using Internet protocols.

[0072] Step 4:

[0073] The server receives the uploaded photos and metadata. The received data is prepared for the image analysis process.

[0074] Step 5:

[0075] The server calls an image analysis API to analyze the content of the photograph. The analysis API uses general-purpose image analysis services such as Google Vision API or Amazon Rekognition.

[0076] Step 6:

[0077] The server parses the list of ingredients received as an API response. For example, let's assume the response contains ingredient names such as "toast", "eggs", and "vegetable salad".

[0078] Step 7:

[0079] The server queries the database to retrieve the calorie value of each food item. The database stores information on various food items and their calorie content.

[0080] Step 8:

[0081] The server adds up the calories of each ingredient to calculate the total calories. For example, if toast is 150kcal, eggs are 70kcal, and a vegetable salad is 30kcal, the total calories would be 250kcal.

[0082] Step 9:

[0083] The server generates the calculation results (list of ingredients and calorie information) in JSON format and sends them back to the terminal. The JSON data includes the analysis results and detailed calorie information.

[0084] Step 10:

[0085] The terminal parses the JSON data received from the server. The parsed data is then converted into a format that can be displayed to the user within the application.

[0086] Step 11:

[0087] The device displays the calorie content of each food item and the total calorie content to the user. For example, it might display in the format: "Toast: 150kcal, Egg: 70kcal, Vegetable salad: 30kcal. Total calories: 250kcal."

[0088] Step 12:

[0089] The server retrieves the user's meal history and health goals from the database. Health goals include things like weight loss and muscle gain.

[0090] Step 13:

[0091] The server uses a machine learning model to generate a menu for the next day based on the user's past eating history and health goals. The model calculates the optimal meal for the user's health condition.

[0092] Step 14:

[0093] The server generates the next day's menu information in JSON format and sends it to the terminal. The generated menu includes recommended foods and their quantities.

[0094] Step 15:

[0095] The terminal parses the menu information it receives. The parsed menu information is then displayed to the user in a visually easy-to-understand format.

[0096] Step 16:

[0097] The device displays the user's menu for the next day. For example, it might display something like, "We recommend yogurt and fruit for breakfast, chicken salad for lunch, and vegetable soup for dinner."

[0098] Step 17:

[0099] The server retrieves the total calorie intake for the day from the database. For example, let's say the total calories from all meals for the day are 1800 kcal.

[0100] Step 18:

[0101] The server calculates the difference between the user's target calorie intake and their current intake. If the target calorie intake is 1500kcal, the difference will be 300kcal.

[0102] Step 19:

[0103] The server calculates an exercise plan to burn 300kcal. For example, 30 minutes of jogging or 60 minutes of walking might be considered the amount of exercise that burns 300kcal.

[0104] Step 20:

[0105] The server generates a recommended exercise plan in JSON format and sends it to the device. The generated exercise plan includes specific exercise types and durations.

[0106] Step 21:

[0107] The device parses the received exercise plan information. The parsed information is converted into a format that can be displayed to the user.

[0108] Step 22:

[0109] The device displays an exercise plan recommended to the user. For example, it might display something like, "To burn off today's calorie intake, we recommend a 30-minute jog or a 60-minute walk."

[0110] (Example 1)

[0111] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0112] In today's lifestyle, many people struggle to maintain a healthy diet. In particular, excessive calorie intake and insufficient exercise lead to obesity and other health problems. Traditional health management systems often require users to manually input calorie calculations and create exercise plans. Against this backdrop, there is a need for a system that allows users to easily record their meals and centrally manage calorie information, optimal menu suggestions, and exercise plans.

[0113] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0114] In this invention, the server includes means for the user to take an image of a meal, means for uploading the image to the server, means for the server to analyze the image and obtain ingredient information and calorie information, means for transmitting the calorie information to the user's terminal, means for displaying the calorie information on the terminal, means for the server to suggest the next day's menu based on the user's meal history and health goals, means for transmitting the next day's menu information to the user's terminal, means for displaying the menu information on the terminal, means for the server to calculate the required amount of exercise and transmit an exercise plan to the user's terminal, and means for displaying the exercise plan on the terminal. As a result, the user can easily upload an image to automatically obtain ingredient information and calorie information and receive menu suggestions and exercise plans tailored to their health goals.

[0115] 1. A "user" refers to an individual who uses the system to take pictures of their meals and receives calorie information, menu suggestions, and exercise plans.

[0116] 2. "Terminal" refers to a device such as a smartphone or tablet used by the user, which is used to take pictures of food or to receive and display information from a server.

[0117] 3. A "server" refers to a computer system that analyzes images sent by users, obtains information on ingredients and calories, and generates menu suggestions and exercise plans to send to the user's terminal.

[0118] 4. "Image analysis" is the process that a server performs to obtain information about ingredients from images of food, and is generally carried out using an image analysis API.

[0119] 5. "Calorie information" refers to the amount of energy calculated by the server based on ingredient information obtained from the image of the meal.

[0120] 6. "Menu information" refers to the meal plan for the following day that the server generates based on the user's meal history and health goals.

[0121] 7. "Exercise plan" refers to the type and amount of exercise suggested by the server based on the user's calorie intake.

[0122] 8. "Image analysis API" refers to an application programming interface used by a server to analyze images of food and obtain information about the ingredients.

[0123] 9. "JSON format" refers to the JavaScript® Object Notation format used for lightweight and easy data exchange.

[0124] 10. A "machine learning model" refers to the algorithms and analytical models that the server uses to generate the next day's menu based on the user's past eating history and health goals.

[0125] 11. "Meal history" refers to data that records information about meals a user has eaten in the past.

[0126] 12. "Health goals" refer to health-related goals that users wish to achieve, such as weight loss or muscle building.

[0127] This invention relates to a system in which a user takes a picture of their meal, which is then automatically calculated based on the image, and which also provides menu suggestions for the following days and guidance on the amount of exercise needed. This system supports the user's healthy eating habits through communication between a server and the user's terminal.

[0128] overview

[0129] Taking photos of food and uploading the images

[0130] The user takes a photo of their meal using a device such as a smartphone or tablet. For example, they might take a photo of their breakfast toast, eggs, and vegetable salad.

[0131] The device reads this photo from local storage and uploads it to the server along with metadata such as the user's ID and timestamp.

[0132] Image analysis and calorie calculation

[0133] The server receives the uploaded image and calls an image analysis API (e.g., Google Vision API) to obtain information about the ingredients.

[0134] Based on the retrieved list of ingredients (e.g., "toast", "eggs", "vegetable salad"), the server retrieves calorie information for each ingredient from the database.

[0135] The server adds up the calories of each ingredient to calculate the total calorie count. For example, if you have toast (150kcal), an egg (70kcal), and a vegetable salad (30kcal), the total calorie count will be 250kcal.

[0136] Display of calorie information

[0137] The server sends the calculated calorie information to the terminal in JSON format.

[0138] The device parses the received JSON data and displays calorie information to the user. For example, it might display, "This morning's breakfast was 250 kcal."

[0139] Menu suggestions for the next day

[0140] The server retrieves the user's meal history and health goals from the database based on their ID. Furthermore, it takes into account the user's health goals (e.g., diet).

[0141] The server uses a machine learning model to generate a menu for the next day based on the user's past eating history and health goals. For example, it might suggest a low-calorie, high-protein meal.

[0142] The generated menu information for the following day is sent to the terminal in JSON format.

[0143] The terminal parses the received menu information and displays it to the user. For example, it might display, "For tomorrow's menu, we recommend yogurt and fruit for breakfast, chicken salad for lunch, and vegetable soup for dinner."

[0144] Guide to recommended exercise levels

[0145] The server retrieves the total calorie intake for the day from the database. For example, if the intake is 1800kcal, it calculates the difference of 300kcal between that and the target intake of 1500kcal.

[0146] The server calculates an exercise plan to burn 300kcal based on the difference. An example might include 30 minutes of jogging and 60 minutes of walking.

[0147] A recommended exercise plan will be sent to your device in JSON format.

[0148] The device parses the received exercise plan and displays it to the user. For example, it might display, "To burn off today's calorie intake, we recommend 30 minutes of jogging or 60 minutes of walking."

[0149] Specific example

[0150] For example, if a user takes a photo of "pancakes, bacon, and fruit" for breakfast, the process would proceed as follows:

[0151] The device uploads this photo to the server.

[0152] The server uses the Google Vision API to recognize these ingredients and retrieves their respective calorie counts from a database.

[0153] The total calories from the pancakes (200kcal), bacon (100kcal), and fruit (80kcal) amount to 380kcal. This is sent to the terminal in JSON format, and the user is shown "This morning's breakfast is 380kcal."

[0154] Furthermore, the server takes the user's past data and health goals into consideration and suggests "oatmeal, yogurt, and fruit" as breakfast for the next day. This is also sent to the terminal in JSON format and displays the message, "We recommend oatmeal, yogurt, and fruit for breakfast tomorrow."

[0155] Finally, the server calculates the calories to be burned and suggests "20 minutes of jogging or 50 minutes of walking," sending this information to the terminal. The terminal then displays a message to the user saying, "To burn off today's calorie intake, we recommend 20 minutes of jogging or 50 minutes of walking."

[0156] Example of a prompt

[0157] "A user has uploaded a photo of their breakfast (pancakes, bacon, fruit). Calculate the calories in these ingredients and display the total calories to the user. Also, suggest a healthy breakfast menu for the next day and display an optimal exercise plan to burn off the user's calories."

[0158] This system allows users to obtain appropriate calorie information from their meals and, furthermore, suggests menus and exercise plans for the next day based on their health goals, enabling them to live a healthy lifestyle.

[0159] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0160] Step 1:

[0161] Taking photos of food and uploading the images

[0162] The user takes a picture of their meal using a device such as a smartphone or tablet. For example, they might take a picture of toast, eggs, and a vegetable salad for breakfast. The captured image is saved to the device's local storage. The device then uploads this photo to the server along with metadata such as the user's ID and a timestamp. The input includes the image of the meal, the user's ID, and the timestamp, while the output is the image data sent to the server.

[0163] Step 2:

[0164] Image analysis and acquisition of food ingredient information

[0165] The server receives the uploaded image and saves it to the specified directory. Next, the server calls an image analysis API (e.g., an image analysis API) to obtain information about the ingredients in the image. Specifically, it sends the image data as input to the image analysis API and receives a list of identified ingredients (e.g., "toast", "egg", "vegetable salad") as output. In this step, the server passes the image data to the API and analyzes the data returned from the API.

[0166] Step 3:

[0167] Calorie calculation

[0168] The server retrieves calorie information for each ingredient from the database based on the retrieved ingredient list. Taking the ingredient list (e.g., "toast", "egg", "vegetable salad") as input, it retrieves the corresponding calorie values ​​from the database (e.g., toast is 150kcal, egg is 70kcal, vegetable salad is 30kcal) as output. The server then sums these calories to calculate the total calorie count (e.g., 250kcal). In this case, the input is the ingredient list, the calorie information is from the database, and the output is the total calorie count.

[0169] Step 4:

[0170] Display of calorie information

[0171] The server sends the calculated calorie information to the terminal in JSON format. It takes calorie information (e.g., {"Breakfast Calories": 250}) as input, converts it to JSON, and sends it to the terminal. The terminal parses the received JSON data and displays "This morning's breakfast was 250kcal." to the user. Here, the input is calorie information in JSON format, and the output is the display of the parsed calorie information.

[0172] Step 5:

[0173] Menu suggestions for the next day

[0174] The server retrieves the user's past meal history and health goals from a database based on their ID. It then inputs the meal history and health goals from the database and uses the output data to generate a menu for the next day based on a machine learning model. For example, it might suggest yogurt and fruit for breakfast, chicken salad for lunch, and vegetable soup for dinner. The generated menu information is then converted to JSON format and sent to the terminal. In this case, the input is the meal history and health goals, and the output is the generated menu information in JSON format.

[0175] Step 6:

[0176] Displaying menu information for the following day

[0177] The terminal parses the received menu information and displays to the user, "For tomorrow's menu, we recommend yogurt and fruit for breakfast, chicken salad for lunch, and vegetable soup for dinner." Here, the input is menu information in JSON format, and the output is a display of the parsed menu information.

[0178] Step 7:

[0179] Calculation of Recommended Exercise Amount

[0180] The server retrieves the total calorie intake for the day from the database. For example, if a user has consumed 1800 kcal and their target intake is 1500 kcal, the server calculates the difference of 300 kcal. Based on this difference, it generates an exercise plan to burn 300 kcal (e.g., 30 minutes of jogging, 60 minutes of walking). In this step, the input is the calorie intake information retrieved from the database, and the output is the generated exercise plan.

[0181] Step 8:

[0182] Displaying exercise plan

[0183] The server sends the exercise plan to the terminal in JSON format. The server takes the exercise plan (e.g., {"exercise plan": "30 minutes of jogging or 60 minutes of walking"}) as input, converts it to JSON, and sends it to the terminal. The terminal parses the received exercise plan and displays to the user, "To burn off today's calorie intake, we recommend 30 minutes of jogging or 60 minutes of walking." Here again, the input is the exercise plan in JSON format, and the output is a display of the parsed exercise plan.

[0184] (Application Example 1)

[0185] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0186] In modern society, there is a growing concern for health, and it is essential to understand the calorie content of meals and maintain a healthy diet. However, calculating calories and creating appropriate meal and exercise plans in daily life is time-consuming and difficult to maintain. In particular, users of food delivery services have a need to easily find out the calorie information of the meals they order, but current technology lacks a system that can provide this information automatically and accurately.

[0187] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0188] In this invention, the server includes means for the user to take an image of a meal, means for uploading the image to the server, means for the server to analyze the image and obtain ingredient information and calorie information, means for transmitting the calorie information to the user's terminal, means for displaying the calorie information on the terminal, means for the server to suggest the next day's menu based on the user's meal history and health goals, means for transmitting the next day's menu information to the user's terminal, means for displaying the menu information on the terminal, means for the server to calculate the required amount of exercise and transmit an exercise plan to the user's terminal, means for providing calorie information of the order and guiding the user to suggest the next meal and exercise plan when using the food delivery service, and means for displaying the exercise plan on the terminal. As a result, when a user uses the food delivery service, they can easily obtain calorie information of their order and check healthy meal suggestions and necessary exercise plans for the next meal.

[0189] "User" refers to any person who uses the system of the present invention.

[0190] "Food images" refers to photographic data of meals taken by users.

[0191] A "server" refers to a central computer system that performs image analysis, data processing, and the transmission and reception of various types of information.

[0192] "Device" refers to a device used by a user (e.g., smartphone, tablet).

[0193] "Method for uploading images" refers to the function that sends image data of a meal from the device to the server.

[0194] "Means for analyzing images" refers to a function that processes and extracts food ingredient information from uploaded images.

[0195] "Ingredient information" refers to the specific types and contents of ingredients obtained from the image analysis results.

[0196] "Calorie information" refers to data on the amount of energy calculated based on information about the ingredients.

[0197] "Meal history" refers to a record of meals a user has eaten in the past.

[0198] "Health goals" refer to objectives or targets set by the user for maintaining or improving their health (e.g., weight loss, muscle building).

[0199] "Next day's menu information" refers to the meal plan for the following day, suggested based on the user's health goals.

[0200] "Required exercise" refers to the amount of energy that should be expended, calculated based on the daily calorie intake.

[0201] An "exercise plan" refers to the specific exercises recommended to burn the necessary amount of energy.

[0202] "Food delivery service" refers to any service where users order food online and have it delivered.

[0203] "Order details" refers to the specific details of the meals and food items selected and confirmed by the user through the food delivery service.

[0204] "Healthy next meal suggestion" refers to the content of the next meal recommended based on the user's health goals and eating history.

[0205] This invention relates to a system that allows users to take pictures of their meals, calculate calories based on those images, and then provide suggestions for their next meal and necessary exercise plans. Users take pictures of their meals using a device such as a smartphone or tablet and upload those images to a server.

[0206] The server analyzes the received image and retrieves information about the ingredients within it. This analysis uses an image analysis API (such as the Google Vision API). After obtaining a list of ingredients from the analysis results, the server searches the database for the calorie information of each ingredient and calculates the total calories. The server sends this calculation result to the terminal in JSON format, and the terminal displays the calorie information to the user.

[0207] Furthermore, the server suggests the next day's menu based on the user's meal history and health goals. A machine learning model is used for this menu suggestion. The menu information is sent in JSON format, and the device displays the next meal suggestion to the user. The server also calculates the amount of exercise needed based on the total calorie intake for the day and sends a specific exercise plan to the device. The device also displays the recommended exercise plan to the user.

[0208] A key feature of this system is its ability to provide calorie information for food delivery orders and offer suggestions for future meals and exercise plans. When a user orders food delivery, the order details are sent to the server upon confirmation. The server retrieves calorie information based on the order and calculates and presents suggestions for healthy meals and exercise plans for the next order.

[0209] As a concrete example, suppose a user eats toast, eggs, and a vegetable salad for breakfast. The user takes pictures of these meals with their device and uploads them to the server. The server uses an image analysis API to obtain information about the ingredients, adds up the calories of each, and sends the total to the user's device. For example, if the toast is 150kcal, the eggs are 70kcal, and the vegetable salad is 30kcal, the total is 250kcal. This information is then displayed on the device.

[0210] Next, when a user orders lunch using a food delivery service, the details (e.g., chicken salad) are sent to the server. The server analyzes the details, retrieves calorie information, displays it to the user, and also provides suggestions for the next meal (e.g., yogurt and fruit) and a necessary exercise plan (e.g., a 30-minute jog).

[0211] Example of a prompt:

[0212] User ID: 12345

[0213] Image of the meal: path_to_image.jpg

[0214] Please retrieve calorie information and suggest the next meal and exercise plan.

[0215] This system makes it easy for users to easily understand calorie information based on their meals and manage their health appropriately. Furthermore, even for meals ordered through food delivery services, users can check calorie information, healthy meal suggestions for the next meal, and exercise plans, providing support for maintaining a healthy lifestyle.

[0216] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0217] Step 1:

[0218] The user takes a picture of their meal. They use a smartphone or tablet to take a photo of the meal and save this image data to the device. The input is the image of the meal, and the output is the image data saved on the device.

[0219] Step 2:

[0220] Images stored on the device are uploaded to the server. The user operates the application to select an image and sends it to the server along with metadata such as the user ID and timestamp. The input is the image data and metadata stored on the device, and the output is the image data sent to the server.

[0221] Step 3:

[0222] The server analyzes the uploaded image. It then calls an image analysis API (e.g., Google Vision API) to extract information about the ingredients. The input is the image data sent to the server, and the output is the list of ingredients returned by the image analysis API.

[0223] Step 4:

[0224] The server retrieves calorie information based on the food ingredients. It searches the database for the calorie information of each ingredient and calculates the total calorie count. The input is a list of ingredients obtained from an image analysis API, and the output is the calorie information of each ingredient and the total calorie count.

[0225] Step 5:

[0226] The server sends the calculated calorie information to the terminal. It generates calorie information in JSON format and sends it to the user's terminal. The input is the calorie information calculated on the server side, and the output is the JSON data sent to the terminal.

[0227] Step 6:

[0228] This application displays calorie information received by the device. It parses the received JSON data and displays the calorie information in an easy-to-understand format for the user. The input is the received JSON data, and the output is the calorie information displayed on the device screen.

[0229] Step 7:

[0230] The server suggests the next day's menu based on the user's meal history and health goals. It retrieves the user's meal history and health goal data from a database and uses a machine learning model to generate the next meal suggestion. The input is the user's meal history and health goals, and the output is the suggested menu information for the following day.

[0231] Step 8:

[0232] The server sends the next day's menu information to the user's device. It generates the menu information in JSON format and sends it to the user's device. The input is the next day's menu information generated by a machine learning model, and the output is the JSON data sent to the device.

[0233] Step 9:

[0234] This application displays menu information received by the device. It parses the received JSON data and displays menu suggestions for the following day to the user. The input is the received JSON data, and the output is the menu information displayed on the device screen.

[0235] Step 10:

[0236] The server calculates the required amount of exercise and sends an exercise plan to the user's device. It calculates the required amount of exercise based on the total calorie intake for the day and generates a specific exercise plan. The calculated exercise plan is sent to the user's device in JSON format. The input is the total calorie intake for the day, and the output is the exercise plan information sent to the device.

[0237] Step 11:

[0238] This program displays the exercise plan received by the device. It parses the received JSON data and displays the recommended exercise plan for the user. The input is the received JSON data, and the output is the exercise plan information displayed on the device screen.

[0239] Step 12:

[0240] This system sends food delivery service order details to a server. When a user places an order using a food delivery service, the order details are sent to the server. The input is the food delivery service order details, and the output is the order data sent to the server.

[0241] Step 13:

[0242] The server retrieves calorie information based on the order and provides suggestions for the next meal and exercise plan. It analyzes the order to obtain calorie information and generates suggestions for the next meal and exercise plan. The input is the submitted order, and the output is calorie information, suggestions for the next meal, and an exercise plan.

[0243] Step 14:

[0244] Calorie information, meal suggestions for the next meal, and exercise plans are sent to the user's device, which then displays them. The input is the calorie information, meal suggestions for the next meal, and exercise plan sent from the server, and the output is this information displayed on the device's screen.

[0245] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0246] This invention relates to a system in which a user takes a picture of their meal, which is then automatically calculated based on the image, and which also provides menu suggestions for the following days and guidance on the amount of exercise needed. This system supports the user's healthy eating habits through communication between the server and the user's terminal. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it can provide more personalized support.

[0247] overview

[0248] The user takes a photo of their meal using a device such as a smartphone or tablet. The device uploads this photo to a server. The server uses image analysis technology to recognize the ingredients in the image and obtain their respective calorie information. Furthermore, the server suggests an appropriate menu and exercise plan for the next day based on the user's health goals. In this invention, an emotion engine is used to recognize the user's emotions and adjust the meal and exercise plan based on the user's emotions and psychological state.

[0249] Program Description

[0250] 1. Take a photo of your meal and upload the image.

[0251] The user takes a photo of their meal with their device. For example, they might take a photo of their breakfast toast, eggs, and vegetable salad.

[0252] The device reads the captured image from local storage and uploads it to the server along with metadata such as the user's ID and timestamp.

[0253] 2. Image analysis and calorie calculation

[0254] The server receives the uploaded image and calls an image analysis API (e.g., Google Vision API) to obtain information about the ingredients.

[0255] Based on the list of ingredients returned from the API (e.g., "toast", "eggs", "vegetable salad"), the calorie information for each ingredient is retrieved from the database.

[0256] Add up the calories of each ingredient to calculate the total calories. For example, if you have toast (150kcal), an egg (70kcal), and a vegetable salad (30kcal), the total calories would be 250kcal.

[0257] 3. Display of calorie information and user sentiment.

[0258] Along with the calculated calorie information, the server uses an emotion engine to recognize the user's emotions from their facial expressions, voice, and entered text.

[0259] The results will be sent to the terminal in JSON format.

[0260] The device parses the received JSON data and displays calorie information and perceived emotion information to the user. Example: "Your breakfast this morning was 250kcal. You seem to be feeling a little tired right now."

[0261] 4. Menu suggestions for the following day

[0262] The server retrieves the user's meal history from the database based on their ID. Furthermore, it takes into account the user's health goals (e.g., dieting).

[0263] Using a machine learning model, the system generates a menu for the next day based on the user's past eating history and health goals. The model also reflects the user's emotional state.

[0264] The generated menu information for the following day is sent to the terminal in JSON format.

[0265] The terminal parses the received menu information and displays it to the user. Example: "For tomorrow's menu, we recommend yogurt and fruit for breakfast, chicken salad for lunch, and vegetable soup for dinner."

[0266] 5. Guidance on recommended exercise levels

[0267] The server retrieves the total calorie intake for the day from the database. For example, let's say the total calories from all meals for the day are 1800 kcal.

[0268] The server calculates the difference between the user's target calorie intake and their current intake. If the target calorie intake is 1500kcal, the difference will be 300kcal.

[0269] Based on the difference, calculate an exercise plan to burn 300kcal. For example, assume that 30 minutes of jogging and 60 minutes of walking will burn 300kcal.

[0270] The exercise plan is adjusted based on the user's emotional state, as determined by the emotion engine. For example, if the user is "tired," it will suggest light, relaxing exercises.

[0271] A recommended exercise plan will be sent to your device in JSON format.

[0272] The device parses the received exercise plan and displays it to the user. Example: "To burn off today's calorie intake, we recommend a 30-minute jog or a 60-minute relaxing walk."

[0273] This system allows users to obtain appropriate calorie information from their diet and also suggests menus and exercise plans for the next day based on their health goals and emotional state, enabling them to live a healthier and less mentally stressful life.

[0274] The following describes the processing flow.

[0275] Step 1:

[0276] The user takes a picture of the meal with their device. The user takes a picture of the meal using the device's camera function, launches the application, and selects the picture.

[0277] Step 2:

[0278] The device reads photos taken from local storage and generates metadata such as the user's ID and timestamp. This integrates the photos with related information.

[0279] Step 3:

[0280] The terminal sends the photo and metadata to the server as an HTTP POST request. The data sent is securely transferred to the server using the Internet protocol.

[0281] Step 4:

[0282] The server receives the uploaded photo and metadata. The received data is prepared for the image analysis process.

[0283] Step 5:

[0284] The server calls an image analysis API to analyze the content of the photo. General-purpose image analysis services such as Google Vision API or Amazon Rekognition are used for the analysis API.

[0285] Step 6:

[0286] The server parses the list of ingredients obtained as the response of the API. For example, assume that the response contains ingredient names such as "toast", "egg", and "vegetable salad".

[0287] Step 7:

[0288] The server queries the database to obtain the calorie value of each ingredient. Various ingredients and their calorie information are stored in the database.

[0289] Step 8:

[0290] The server sums up the calories of each ingredient and calculates the total calories. For example, if toast is 150 kcal, egg is 70 kcal, and vegetable salad is 30 kcal, the total calories will be 250 kcal.

[0291] Step 9:

[0292] To recognize the user's emotions, the device uses its camera and microphone to capture the user's facial expressions and voice. Alternatively, the user fills out a questionnaire about their emotions.

[0293] Step 10:

[0294] The device uses an emotion engine to analyze the user's emotions from the acquired data. For example, it can determine if the user is feeling tired or stressed.

[0295] Step 11:

[0296] The server sends calorie information and recognized emotion information to the terminal in JSON format. The JSON data includes the analysis results, detailed calorie information, and recognized emotion.

[0297] Step 12:

[0298] The terminal parses the JSON data received from the server. The parsed data is then converted into a format that can be displayed to the user within the application.

[0299] Step 13:

[0300] The device displays the calories for each food item, the total calories, and the user's mood information. For example, it might display in the format: "Toast: 150kcal, Egg: 70kcal, Vegetable salad: 30kcal. Total calories: 250kcal. You seem to be feeling a little tired right now."

[0301] Step 14:

[0302] The server retrieves the user's meal history from the database based on their ID. Furthermore, it takes into account the user's health goals (e.g., dieting).

[0303] Step 15:

[0304] The server uses a machine learning model to generate a menu for the next day based on the user's past meal history and health goals. The model also reflects the user's emotional state.

[0305] Step 16:

[0306] The server generates menu information for the next day in JSON format and sends it to the terminal. The generated menu includes recommended foods and their quantities.

[0307] Step 17:

[0308] The terminal parses the received menu information. The parsed menu information is visually displayed to the user in an easy-to-understand manner. <000,0973> Step 18:

[0310] The terminal displays the menu for the next day to the user. For example, it is displayed in the form of "As tomorrow's menu, we recommend yogurt and fruit for breakfast, salad chicken for lunch, and vegetable soup for dinner."

[0311] Step 19:

[0312] The server retrieves the total calorie intake for the day from the database. For example, assume the total calories of all meals for the day is 1800 kcal.

[0313] Step 20:

[0314] The server calculates the difference from the user's target calorie intake. If the target calorie is 1500 kcal, the difference is 300 kcal.

[0315] Step 21:

[0316] The server calculates an exercise plan to burn 300 kcal. For example, assume jogging for 30 minutes and walking for 60 minutes is the amount of exercise to burn 300 kcal.

[0317] Step 22:

[0318] The server adjusts the exercise plan based on the user's emotional state, as determined by the emotion engine. For example, if the user is "tired," it suggests light, relaxing exercise.

[0319] Step 23:

[0320] The server generates a recommended exercise plan in JSON format and sends it to the device. The generated exercise plan includes specific exercise types and durations.

[0321] Step 24:

[0322] The device parses the received exercise plan information. The parsed information is converted into a format that can be displayed to the user.

[0323] Step 25:

[0324] The device displays an exercise plan recommended to the user. For example, it might say, "To burn off today's calorie intake, we recommend a 30-minute jog or a 60-minute relaxing walk."

[0325] (Example 2)

[0326] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0327] In recent years, the importance of users accurately understanding their diet and calorie expenditure, and implementing appropriate meal and exercise plans, has increased in order to maintain and improve a healthy lifestyle. However, conventional methods require manual recording and calculation, which is time-consuming and laborious, and they have the problem of not being able to take into account the user's emotions and psychological state. Furthermore, calorie calculation and exercise plan suggestions are unified, and do not provide optimal support for the user. This invention aims to solve these problems and provide a health management system that is more convenient and effective for users.

[0328] The identification processing by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for the user to take an image of a meal, means for uploading the image to the server, means for the server to analyze the image and obtain ingredient information and calorie information, means for transmitting the calorie information to the user's terminal, means for displaying the calorie information on the terminal, means for the server to suggest the next day's menu based on the user's meal history and health goals, means for recognizing the user's emotional state and adjusting the meal and exercise plan, means for transmitting the next day's menu information to the user's terminal, means for displaying the menu information on the terminal, means for the server to calculate the required amount of exercise based on the day's calorie intake and health goals and transmit the exercise plan to the user's terminal, and means for displaying the exercise plan on the terminal. As a result, the user can not only receive calorie calculations for the contents of their meals and menu suggestions for the next day, but also receive suggestions for exercise plans that take their emotional state into consideration.

[0329] A "user" is an individual who uses this system to manage their own health.

[0330] A "device" refers to an electronic device operated by a user, such as a smartphone or tablet.

[0331] A "server" is a computer that analyzes data sent from a terminal and provides the necessary information.

[0332] An "image analysis API" is an application program interface for recognizing food information within an image.

[0333] "Calorie information" refers to data that shows the energy intake of each food item.

[0334] "User's meal history" refers to a record of the meals a user has eaten in the past.

[0335] "Health goals" refer to health-related objectives set by the user, such as weight loss or muscle building.

[0336] "Tomorrow's menu" refers to the meal plan for the following day that is suggested to the user.

[0337] "Emotional state" refers to information that indicates the user's psychological or emotional condition.

[0338] An "exercise plan" is a plan of exercises that a user uses to burn off the calories they have consumed.

[0339] "Metadata" refers to supplementary information added to data, such as user IDs and timestamps.

[0340] A "machine learning model" is an algorithm that analyzes user data and performs predictions and classifications.

[0341] "JSON format" refers to a lightweight text format used for exchanging data.

[0342] This invention relates to a system that allows users to take pictures of their meals, automatically calculates calories based on those images, and then suggests menus for the following days and provides guidance on the amount of exercise needed. Specific proper nouns, names of people, and company names are converted to common names, and nouns are replaced with broader terms for explanation.

[0343] First, the user takes a picture of their meal using a device such as a smartphone or tablet. For example, consider a case where the user takes a picture of toast, eggs, and a vegetable salad for breakfast. The device saves this photo to local storage, adds metadata such as the user's ID and timestamp, and uploads it to the server.

[0344] The server receives the uploaded image and uses an image analysis API such as the Google Vision API to obtain information about the ingredients in the image. Based on the list of ingredients returned from the API (e.g., "toast", "egg", "vegetable salad"), it retrieves the calorie information for each ingredient from its internal database and sums them up. For example, if toast is 150kcal, egg is 70kcal, and vegetable salad is 30kcal, the total calorie count would be 250kcal.

[0345] Next, the server uses an emotion engine to recognize the user's emotions from their facial expressions, voice, and entered text. This calorie and emotion information is sent to the device in JSON format, and the device displays it. For example, a message such as "Your breakfast this morning was 250kcal. You seem to be feeling a little tired right now." might be displayed.

[0346] Furthermore, the server retrieves the user's meal history from a database based on their ID and suggests the next day's menu, taking into account the user's health goals and emotional state. The menu information, generated using a machine learning model, is sent to the device in JSON format, which the device then displays to the user. For example, a suggestion might be, "For tomorrow's menu, we recommend yogurt and fruit for breakfast, chicken salad for lunch, and vegetable soup for dinner."

[0347] The server also calculates the difference between the user's total calorie intake for the day and their target calorie intake, and then calculates a recommended amount of exercise based on this difference. The emotion engine checks the user's emotional state and adjusts the exercise plan accordingly. This is sent to the device in JSON format, and the device displays a message such as, "To burn off today's calorie intake, we recommend a 30-minute jog or a 60-minute relaxing walk."

[0348] This system can automatically obtain appropriate calorie information from the user's diet and further supports a healthier and less mentally stressful lifestyle by suggesting menus and exercise plans for the next day based on health goals and emotional state.

[0349] Example: Example of a prompt message to input to a generative AI model

[0350] Example 1: Taking photos of food and uploading the images

[0351] Prompt: "Please upload a photo of toast, eggs, and a vegetable salad taken with your smartphone to the server."

[0352] Example 2: Image analysis and calorie calculation

[0353] Prompt: "Retrieve ingredient information from the uploaded breakfast image and calculate the calories of each ingredient."

[0354] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0355] Step 1:

[0356] The user takes a photo of their meal with their device. For example, the process of taking a photo of toast, eggs, and a vegetable salad for breakfast is described.

[0357] Input: The user launches the camera app and takes a picture of the food.

[0358] Output: Image files of the meal saved on the device.

[0359] Specific action: A photo of breakfast is taken using the smartphone's camera app, and that image is saved to the device's local storage.

[0360] Step 2:

[0361] The device reads the captured image from local storage and uploads it to the server along with metadata such as the user's ID and timestamp.

[0362] Input: Image file of the meal saved on the device, user ID, timestamp

[0363] Output: Image data and metadata transferred to the server

[0364] Specific action: The device sends the image and metadata to the server as an HTTP POST request.

[0365] Step 3:

[0366] The server receives the uploaded image and calls an image analysis API to retrieve information about the ingredients in the image.

[0367] Input: Image data sent from the device

[0368] Output: List of ingredients returned from the image analysis API

[0369] Specific operation: The server sends images to an image analysis API such as the Google Vision API and receives the analysis results.

[0370] Step 4:

[0371] Based on the acquired food information, the server retrieves calorie information for each food item from the database.

[0372] Input: List of ingredients obtained from an image analysis API

[0373] Output: Calorie information for each food item retrieved from the database.

[0374] Specific operation: The server sends a query to an internal database to retrieve calorie information, for example, toast (150kcal), eggs (70kcal), and a vegetable salad (30kcal).

[0375] Step 5:

[0376] The server adds up the calories of each ingredient and calculates the total calories.

[0377] Input: Calorie information for each ingredient

[0378] Output: Total calorie value

[0379] Specific operation: The server adds up the calories of each ingredient and calculates a total calorie count of 250kcal as an example.

[0380] Step 6:

[0381] The server uses an emotion engine to recognize emotions from the user's facial expressions, voice, and input text.

[0382] Input: User facial image, audio data, input text

[0383] Output: Emotional state data returned from the emotion engine.

[0384] Specific operation: The server sends user data to the emotion engine and receives the emotion analysis results.

[0385] Step 7:

[0386] The server sends calorie information and emotional information to the terminal in JSON format.

[0387] Input: Total calorie value, emotional state data

[0388] Output: Data converted to JSON format

[0389] Specific operation: The server converts calorie information and emotion information into JSON and sends it to the terminal as an HTTP response.

[0390] Step 8:

[0391] The device parses the received JSON data and displays calorie information and emotional information to the user.

[0392] Input: Calorie information and emotion information in JSON format

[0393] Output: Calorie information and emotional information displayed on the screen

[0394] Specific action: The terminal parses the JSON data and displays the message, "Your breakfast this morning was 250kcal. You seem to be feeling a little tired right now."

[0395] Step 9:

[0396] The server retrieves past meal history from a database based on the user's ID and suggests the next day's menu, taking into account the user's health goals and emotional state.

[0397] Input: User ID, health goal information, emotional state

[0398] Output: Suggested menu for the next day

[0399] Specific operation: The server issues SQL queries to retrieve meal history and health goals, and uses a machine learning model to generate the next day's menu.

[0400] Step 10:

[0401] The server sends the suggested menu information for the following day to the terminal in JSON format.

[0402] Input: Suggestion menu data

[0403] Output: Suggested menu converted to JSON format

[0404] Specific operation: The server converts the suggested menu into JSON format and sends it to the terminal as an HTTP response.

[0405] Step 11:

[0406] The terminal analyzes the received menu information and displays it to the user.

[0407] Input: Suggestion menu in JSON format

[0408] Output: Suggestion menu displayed on the screen

[0409] Specific action: The terminal parses the JSON data and displays the message, "For tomorrow's menu, we recommend yogurt and fruit for breakfast, chicken salad for lunch, and vegetable soup for dinner."

[0410] Step 12:

[0411] The server calculates the difference between the user's total calorie intake for the day and their target calorie intake, and then calculates the amount of exercise required based on that difference.

[0412] Input: Total calorie intake for the day, target calorie intake

[0413] Output: Recommended exercise plan

[0414] Specific operation: The server calculates an exercise plan that burns, for example, 300 kcal, based on the difference.

[0415] Step 13:

[0416] The server checks the user's emotional state based on the emotion engine and adjusts the exercise plan accordingly.

[0417] Input: Recommended exercise plan, emotional state

[0418] Output: Adjusted exercise plan

[0419] Specific action: The server suggests a relaxing exercise plan (e.g., a 60-minute walk) when the user is tired.

[0420] Step 14:

[0421] The server sends a recommended exercise plan to the device in JSON format.

[0422] Input: Data from the adjusted exercise plan

[0423] Output: Exercise plan converted to JSON format

[0424] Specific operation: The server converts the motion plan into JSON format and sends it to the terminal as an HTTP response.

[0425] Step 15:

[0426] The device analyzes the received exercise plan and displays it to the user.

[0427] Input: Exercise plan in JSON format

[0428] Output: Exercise plan displayed on the screen

[0429] Specific action: The device parses the JSON data and displays the message, "To burn off today's calorie intake, we recommend a 30-minute jog or a 60-minute relaxing walk."

[0430] (Application Example 2)

[0431] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0432] In modern society, it is crucial for users to manage their own health and implement appropriate dietary and exercise plans. However, there are limited ways for users to access real-time calorie information and health recommendations when choosing food. Furthermore, it is difficult to provide health recommendations that take into account the user's emotional state. Therefore, there is a need for a system that enables users to make appropriate dietary choices and implement exercise plans based on their emotional state and health goals.

[0433] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0434] In this invention, the server includes means for the user to take an image of a meal, means for uploading the image to the server, means for the server to analyze the image and obtain ingredient information and calorie information, means for transmitting the calorie information to the user's terminal, means for displaying the calorie information on the terminal, means for the server to suggest the next day's menu based on the user's meal history and health goals, means for transmitting the next day's menu information to the user's terminal, means for displaying the menu information on the terminal, means for the server to calculate the required amount of exercise and transmit an exercise plan to the user's terminal, means for displaying the exercise plan on the terminal, means for a head-mounted display worn by the user to recognize ingredients and food products and display calorie information in real time, and means for the head-mounted display to recognize the user's emotional state and adjust health suggestions accordingly. As a result, when the user selects food in a store, they can check calorie information in real time and receive health suggestions tailored to their emotional state.

[0435] A "user" refers to an individual who uses the system to take photos of their meals or manage their health.

[0436] "Food images" refer to images taken by users to record the contents of their meals.

[0437] A "server" refers to a device or computer system that processes data sent by a user, calculates ingredient information and calorie information, and transmits it to the user's terminal.

[0438] "Uploading" refers to the operation or process of sending data from a user's device to a server.

[0439] "Image analysis" refers to the process of extracting information about ingredients and calories from images of food that have been photographed.

[0440] "Ingredient information" refers to data about the types and characteristics of each ingredient included in a meal.

[0441] "Calorie information" refers to data that shows the energy content of food and ingredients.

[0442] "Device" refers to a smartphone, tablet, or other electronic device owned by the user.

[0443] "Meal history" refers to data that records the types of meals a user has eaten in the past.

[0444] "Health goals" refer to health-related achievement goals set by the user (e.g., weight loss, muscle gain, etc.).

[0445] "Menu suggestions" refer to meal menus that are suggested based on the user's health goals and eating history.

[0446] An "exercise plan" refers to an exercise plan proposed to help the user achieve their desired level of physical activity.

[0447] A "head-mounted display" refers to a display device that a user wears on their head (e.g., a head-mounted display).

[0448] "Emotional state" refers to the user's psychological or emotional state.

[0449] "Health suggestions" refer to suggestions regarding diet and exercise that are tailored based on the user's emotional state and health goals.

[0450] "Real-time" refers to operations and data processing being performed instantly.

[0451] "Recognition" refers to the process by which a system extracts and understands specific information from input data (e.g., images or audio).

[0452] This invention is a system that supports users' health management by utilizing a head-mounted display (HMD) that users use on a daily basis. This system consistently performs tasks such as capturing images of meals, calculating calories, recognizing emotional states, and providing health recommendations. The specific implementation of the system is described below.

[0453] Hardware and software to be used

[0454] 1. Head-mounted display (HMD):

[0455] It is used by users when selecting food items in a store. The HMD has a built-in camera and display, and displays calorie information in real time when the user takes a picture of the food.

[0456] A typical HMD device is used as a concrete example.

[0457] 2. Server:

[0458] The system performs image analysis and calorie calculation, manages a database of meal history, generates health recommendations, and recognizes the user's emotional state. The server utilizes image analysis APIs (e.g., general image analysis services) and the EmotionRecognition library.

[0459] 3. User terminal:

[0460] The system displays calorie information and health suggestions sent from the server on the user's smartphone, tablet, or other device.

[0461] System processing and functions

[0462] 1. Take and upload photos of your meal:

[0463] The user takes pictures of food using the HMD's camera. The captured images are temporarily stored in the HMD's local storage and then uploaded to the server.

[0464] 2. Image analysis and acquisition of calorie information:

[0465] The server sends the received image to an image analysis API to obtain ingredient and calorie information. This allows detailed nutritional data of the food photographed by the user to be stored on the server.

[0466] 3. Display of calorie information:

[0467] The acquired calorie information is displayed in real time on the user's HMD (Head-Mounted Display). This allows the user to check the calorie content of food on the spot.

[0468] 4. Recognition of emotional states:

[0469] The HMD's camera and microphone are used to capture the user's facial expressions and voice, and an emotion recognition library is used to analyze the user's emotional state. The results are also sent to the server.

[0470] 5. Generating and displaying health suggestions:

[0471] The server generates health suggestions, including menu suggestions for the next day and recommended exercise levels, based on the user's eating history, health goals, and current emotional state. The generated suggestions are sent to and displayed on the user's HMD and device.

[0472] Specific example

[0473] For example, when a user chooses a sandwich at a cafe, the HMD (Head-Mounted Display) takes a picture of the sandwich and instantly displays its calorie count. Furthermore, if the server determines that the user has been lacking exercise recently and is feeling a bit tired, it will suggest light walking or relaxing exercises.

[0474] Example of a prompt

[0475] "Create a model for an application that calculates calories based on food images taken by the user using an HMD (Head-Mounted Display) and displays health suggestions tailored to the user's emotional state. The model will use a common image analysis service to obtain calorie information from food ingredients and the EmotionRecognition library to recognize the user's emotions. Implement the generated program in Python and explain the overall flow."

[0476] This system allows users to check calorie information on food in real time and select appropriate meals and exercise based on their health and emotional state.

[0477] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0478] Step 1:

[0479] The user takes a picture of their meal using the camera on a head-mounted display (HMD). Specifically, the user points the HMD's camera at the meal and the image is saved to local storage. The input is the image of the meal, and the output is the image file saved to local storage.

[0480] Step 2:

[0481] The device uploads the saved image to the server. This operation uses a communication module to send the image file over the internet. The input is the image file in local storage, and the output is the image data sent to the server.

[0482] Step 3:

[0483] The server analyzes the received image to obtain information about the ingredients and calories. Specifically, the server calls an image analysis API to recognize the ingredients in the image and obtain the component data for each ingredient. The input is the uploaded image of the meal, and the output is the ingredient information and its corresponding calorie information.

[0484] Step 4:

[0485] The server sends the acquired calorie information to the user's terminal. The server packages the calorie information in JSON format and sends it to the terminal. The input is the calorie information and the user ID, and the output is the JSON data sent to the terminal.

[0486] Step 5:

[0487] The terminal analyzes and displays the received calorie information. The terminal parses the JSON data and displays the calorie information clearly on the screen. The input is the JSON data received from the server, and the output is the calorie information displayed on the terminal's screen.

[0488] Step 6:

[0489] The server analyzes the user's facial expressions and voice, captured by the HMD's camera and microphone, to recognize the user's emotional state. It uses an emotion recognition library to analyze the emotional state and obtain the results. The input is the user's facial expression and voice data sent from the HMD, and the output is emotional state data.

[0490] Step 7:

[0491] The server generates menu suggestions and exercise plans for the next day based on the user's meal history, health goals, and emotional state. It uses a machine learning model to create optimal suggestions considering each data point. Inputs are the user's meal history, health goals, and emotional state, while outputs are menu suggestions and exercise plans for the next day.

[0492] Step 8:

[0493] The server sends the generated menu suggestions and exercise plan to the user's device and HMD. The packaged data is sent in JSON format, and receipt is confirmed. The input is the menu suggestion and exercise plan data, and the output is the suggestion data sent to the device and HMD.

[0494] Step 9:

[0495] The terminal and HMD display the received menu suggestions and exercise plans. The user reviews the suggestions displayed on the screen and plans their next course of action. This allows the user to manage their health in real time. The input is the suggestion data sent to the terminal and HMD, and the output is the displayed menu suggestions and exercise plans.

[0496] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0497] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0498] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0499] [Second Embodiment]

[0500] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0501] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0502] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0503] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0504] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0505] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0506] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0507] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0508] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0509] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0510] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0511] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0512] This invention relates to a system in which a user takes a picture of their meal, which is then automatically calculated based on the image, and which also provides menu suggestions for the following days and guidance on the amount of exercise needed. This system supports the user's healthy eating habits through communication between a server and the user's terminal.

[0513] overview

[0514] Users take photos of their meals using devices such as smartphones or tablets. These photos are uploaded to a server. The server uses image analysis technology to recognize the ingredients in the images and obtain their calorie information. Furthermore, the server suggests appropriate menus and exercise plans for the following day based on the user's health goals.

[0515] Program Description

[0516] 1. Take a photo of your meal and upload the image.

[0517] The user takes a photo of their meal with their device. For example, they might take a photo of their breakfast toast, eggs, and vegetable salad.

[0518] The device reads the captured image from local storage and uploads it to the server along with metadata such as the user's ID and timestamp.

[0519] 2. Image analysis and calorie calculation

[0520] The server receives the uploaded image and calls an image analysis API (e.g., Google Vision API) to obtain information about the ingredients.

[0521] Based on the list of ingredients returned from the API (e.g., "toast", "eggs", "vegetable salad"), the calorie information for each ingredient is retrieved from the database.

[0522] Add up the calories of each ingredient to calculate the total calories. For example, if you have toast (150kcal), an egg (70kcal), and a vegetable salad (30kcal), the total calories would be 250kcal.

[0523] 3. Display of calorie information

[0524] The server sends the calculated calorie information to the terminal in JSON format.

[0525] The device parses the received JSON data and displays calorie information to the user. Example: "This morning's breakfast was 250 kcal."

[0526] 4. Menu suggestions for the following day

[0527] The server retrieves the user's meal history from the database based on their ID. Furthermore, it takes into account the user's health goals (e.g., dieting).

[0528] Using a machine learning model, the system generates a menu for the next day based on the user's past eating history and health goals. For example, it might suggest a low-calorie, high-protein meal.

[0529] The generated menu information for the following day is sent to the terminal in JSON format.

[0530] The terminal parses the received menu information and displays it to the user. Example: "For tomorrow's menu, we recommend yogurt and fruit for breakfast, chicken salad for lunch, and vegetable soup for dinner."

[0531] 5. Guidance on recommended exercise levels

[0532] The server retrieves the total calorie intake for the day from the database. For example, if the intake is 1800kcal, it calculates the difference of 300kcal between that and the target intake of 1500kcal.

[0533] Based on the difference, calculate an exercise plan to burn 300kcal. Example: 30 minutes of jogging, 60 minutes of walking.

[0534] A recommended exercise plan will be sent to your device in JSON format.

[0535] The device parses the received exercise plan and displays it to the user. Example: "To burn off today's calorie intake, we recommend 30 minutes of jogging or 60 minutes of walking."

[0536] This system allows users to obtain appropriate calorie information from their meals and, furthermore, suggests menus and exercise plans for the next day based on their health goals, enabling them to live a healthy lifestyle.

[0537] The following describes the processing flow.

[0538] Step 1:

[0539] The user takes a photo of the meal with their device. The user takes a photo of the meal using the device's camera function, launches the application, and selects the photo.

[0540] Step 2:

[0541] The device reads photos taken from local storage and generates metadata such as the user's ID and timestamp. This integrates the photos with related information.

[0542] Step 3:

[0543] The device sends the photo and metadata to the server as an HTTP POST request. The data is securely transferred to the server using Internet protocols.

[0544] Step 4:

[0545] The server receives the uploaded photos and metadata. The received data is prepared for the image analysis process.

[0546] Step 5:

[0547] The server calls an image analysis API to analyze the content of the photograph. The analysis API uses general-purpose image analysis services such as Google Vision API or Amazon Rekognition.

[0548] Step 6:

[0549] The server parses the list of ingredients received as an API response. For example, let's assume the response contains ingredient names such as "toast", "eggs", and "vegetable salad".

[0550] Step 7:

[0551] The server queries the database to retrieve the calorie value of each food item. The database stores information on various food items and their calorie content.

[0552] Step 8:

[0553] The server adds up the calories of each ingredient to calculate the total calories. For example, if toast is 150kcal, eggs are 70kcal, and a vegetable salad is 30kcal, the total calories would be 250kcal.

[0554] Step 9:

[0555] The server generates the calculation results (list of ingredients and calorie information) in JSON format and sends them back to the terminal. The JSON data includes the analysis results and detailed calorie information.

[0556] Step 10:

[0557] The terminal parses the JSON data received from the server. The parsed data is then converted into a format that can be displayed to the user within the application.

[0558] Step 11:

[0559] The device displays the calorie content of each food item and the total calorie content to the user. For example, it might display in the format: "Toast: 150kcal, Egg: 70kcal, Vegetable salad: 30kcal. Total calories: 250kcal."

[0560] Step 12:

[0561] The server retrieves the user's meal history and health goals from the database. Health goals include things like weight loss and muscle gain.

[0562] Step 13:

[0563] The server uses a machine learning model to generate a menu for the next day based on the user's past eating history and health goals. The model calculates the optimal meal for the user's health condition.

[0564] Step 14:

[0565] The server generates the next day's menu information in JSON format and sends it to the terminal. The generated menu includes recommended foods and their quantities.

[0566] Step 15:

[0567] The terminal parses the menu information it receives. The parsed menu information is then displayed to the user in a visually easy-to-understand format.

[0568] Step 16:

[0569] The device displays the user's menu for the next day. For example, it might display something like, "We recommend yogurt and fruit for breakfast, chicken salad for lunch, and vegetable soup for dinner."

[0570] Step 17:

[0571] The server retrieves the total calorie intake for the day from the database. For example, let's say the total calories from all meals for the day are 1800 kcal.

[0572] Step 18:

[0573] The server calculates the difference between the user's target calorie intake and their current intake. If the target calorie intake is 1500kcal, the difference will be 300kcal.

[0574] Step 19:

[0575] The server calculates an exercise plan to burn 300kcal. For example, 30 minutes of jogging or 60 minutes of walking might be considered the amount of exercise that burns 300kcal.

[0576] Step 20:

[0577] The server generates a recommended exercise plan in JSON format and sends it to the device. The generated exercise plan includes specific exercise types and durations.

[0578] Step 21:

[0579] The device parses the received exercise plan information. The parsed information is converted into a format that can be displayed to the user.

[0580] Step 22:

[0581] The device displays an exercise plan recommended to the user. For example, it might display something like, "To burn off today's calorie intake, we recommend a 30-minute jog or a 60-minute walk."

[0582] (Example 1)

[0583] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0584] In today's lifestyle, many people struggle to maintain a healthy diet. In particular, excessive calorie intake and insufficient exercise lead to obesity and other health problems. Traditional health management systems often require users to manually input calorie calculations and create exercise plans. Against this backdrop, there is a need for a system that allows users to easily record their meals and centrally manage calorie information, optimal menu suggestions, and exercise plans.

[0585] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0586] In this invention, the server includes means for the user to take an image of a meal, means for uploading the image to the server, means for the server to analyze the image and obtain ingredient information and calorie information, means for transmitting the calorie information to the user's terminal, means for displaying the calorie information on the terminal, means for the server to suggest the next day's menu based on the user's meal history and health goals, means for transmitting the next day's menu information to the user's terminal, means for displaying the menu information on the terminal, means for the server to calculate the required amount of exercise and transmit an exercise plan to the user's terminal, and means for displaying the exercise plan on the terminal. As a result, the user can easily upload an image to automatically obtain ingredient information and calorie information and receive menu suggestions and exercise plans tailored to their health goals.

[0587] 1. A "user" refers to an individual who uses the system to take pictures of their meals and receives calorie information, menu suggestions, and exercise plans.

[0588] 2. "Terminal" refers to a device such as a smartphone or tablet used by the user, which is used to take pictures of food or to receive and display information from a server.

[0589] 3. A "server" refers to a computer system that analyzes images sent by users, obtains information on ingredients and calories, and generates menu suggestions and exercise plans to send to the user's terminal.

[0590] 4. "Image analysis" is the process that a server performs to obtain information about ingredients from images of food, and is generally carried out using an image analysis API.

[0591] 5. "Calorie information" refers to the amount of energy calculated by the server based on ingredient information obtained from the image of the meal.

[0592] 6. "Menu information" refers to the meal plan for the following day that the server generates based on the user's meal history and health goals.

[0593] 7. "Exercise plan" refers to the type and amount of exercise suggested by the server based on the user's calorie intake.

[0594] 8. "Image analysis API" refers to an application programming interface used by a server to analyze images of food and obtain information about the ingredients.

[0595] 9. "JSON format" refers to the JavaScript Object Notation format used for lightweight and easy data exchange.

[0596] 10. A "machine learning model" refers to the algorithms and analytical models that the server uses to generate the next day's menu based on the user's past eating history and health goals.

[0597] 11. "Meal history" refers to data that records information about meals a user has eaten in the past.

[0598] 12. "Health goals" refer to health-related goals that users wish to achieve, such as weight loss or muscle building.

[0599] This invention relates to a system in which a user takes a picture of their meal, which is then automatically calculated based on the image, and which also provides menu suggestions for the following days and guidance on the amount of exercise needed. This system supports the user's healthy eating habits through communication between a server and the user's terminal.

[0600] overview

[0601] Taking photos of food and uploading the images

[0602] The user takes a photo of their meal using a device such as a smartphone or tablet. For example, they might take a photo of their breakfast toast, eggs, and vegetable salad.

[0603] The device reads this photo from local storage and uploads it to the server along with metadata such as the user's ID and timestamp.

[0604] Image analysis and calorie calculation

[0605] The server receives the uploaded image and calls an image analysis API (e.g., Google Vision API) to obtain information about the ingredients.

[0606] Based on the retrieved list of ingredients (e.g., "toast", "eggs", "vegetable salad"), the server retrieves calorie information for each ingredient from the database.

[0607] The server adds up the calories of each ingredient to calculate the total calorie count. For example, if you have toast (150kcal), an egg (70kcal), and a vegetable salad (30kcal), the total calorie count will be 250kcal.

[0608] Display of calorie information

[0609] The server sends the calculated calorie information to the terminal in JSON format.

[0610] The device parses the received JSON data and displays calorie information to the user. For example, it might display, "This morning's breakfast was 250 kcal."

[0611] Menu suggestions for the next day

[0612] The server retrieves the user's meal history and health goals from the database based on their ID. Furthermore, it takes into account the user's health goals (e.g., diet).

[0613] The server uses a machine learning model to generate a menu for the next day based on the user's past eating history and health goals. For example, it might suggest a low-calorie, high-protein meal.

[0614] The generated menu information for the following day is sent to the terminal in JSON format.

[0615] The terminal parses the received menu information and displays it to the user. For example, it might display, "For tomorrow's menu, we recommend yogurt and fruit for breakfast, chicken salad for lunch, and vegetable soup for dinner."

[0616] Guide to recommended exercise levels

[0617] The server retrieves the total calorie intake for the day from the database. For example, if the intake is 1800kcal, it calculates the difference of 300kcal between that and the target intake of 1500kcal.

[0618] The server calculates an exercise plan to burn 300kcal based on the difference. An example might include 30 minutes of jogging and 60 minutes of walking.

[0619] A recommended exercise plan will be sent to your device in JSON format.

[0620] The device parses the received exercise plan and displays it to the user. For example, it might display, "To burn off today's calorie intake, we recommend 30 minutes of jogging or 60 minutes of walking."

[0621] Specific example

[0622] For example, if a user takes a photo of "pancakes, bacon, and fruit" for breakfast, the process would proceed as follows:

[0623] The device uploads this photo to the server.

[0624] The server uses the Google Vision API to recognize these ingredients and retrieves their respective calorie counts from a database.

[0625] The total calories from the pancakes (200kcal), bacon (100kcal), and fruit (80kcal) amount to 380kcal. This is sent to the terminal in JSON format, and the user is shown "This morning's breakfast is 380kcal."

[0626] Furthermore, the server takes the user's past data and health goals into consideration and suggests "oatmeal, yogurt, and fruit" as breakfast for the next day. This is also sent to the terminal in JSON format and displays the message, "We recommend oatmeal, yogurt, and fruit for breakfast tomorrow."

[0627] Finally, the server calculates the calories to be burned and suggests "20 minutes of jogging or 50 minutes of walking," sending this information to the terminal. The terminal then displays a message to the user saying, "To burn off today's calorie intake, we recommend 20 minutes of jogging or 50 minutes of walking."

[0628] Example of a prompt

[0629] "A user has uploaded a photo of their breakfast (pancakes, bacon, fruit). Calculate the calories in these ingredients and display the total calories to the user. Also, suggest a healthy breakfast menu for the next day and display an optimal exercise plan to burn off the user's calories."

[0630] This system allows users to obtain appropriate calorie information from their meals and, furthermore, suggests menus and exercise plans for the next day based on their health goals, enabling them to live a healthy lifestyle.

[0631] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0632] Step 1:

[0633] Taking photos of food and uploading the images

[0634] The user takes a picture of their meal using a device such as a smartphone or tablet. For example, they might take a picture of toast, eggs, and a vegetable salad for breakfast. The captured image is saved to the device's local storage. The device then uploads this photo to the server along with metadata such as the user's ID and a timestamp. The input includes the image of the meal, the user's ID, and the timestamp, while the output is the image data sent to the server.

[0635] Step 2:

[0636] Image analysis and acquisition of food ingredient information

[0637] The server receives the uploaded image and saves it to the specified directory. Next, the server calls an image analysis API (e.g., an image analysis API) to obtain information about the ingredients in the image. Specifically, it sends the image data as input to the image analysis API and receives a list of identified ingredients (e.g., "toast", "egg", "vegetable salad") as output. In this step, the server passes the image data to the API and analyzes the data returned from the API.

[0638] Step 3:

[0639] Calorie calculation

[0640] The server retrieves calorie information for each ingredient from the database based on the retrieved ingredient list. Taking the ingredient list (e.g., "toast", "egg", "vegetable salad") as input, it retrieves the corresponding calorie values ​​from the database (e.g., toast is 150kcal, egg is 70kcal, vegetable salad is 30kcal) as output. The server then sums these calories to calculate the total calorie count (e.g., 250kcal). In this case, the input is the ingredient list, the calorie information is from the database, and the output is the total calorie count.

[0641] Step 4:

[0642] Display of calorie information

[0643] The server sends the calculated calorie information to the terminal in JSON format. It takes calorie information (e.g., {"Breakfast Calories": 250}) as input, converts it to JSON, and sends it to the terminal. The terminal parses the received JSON data and displays "This morning's breakfast was 250kcal." to the user. Here, the input is calorie information in JSON format, and the output is the display of the parsed calorie information.

[0644] Step 5:

[0645] Menu suggestions for the next day

[0646] The server retrieves the user's past meal history and health goals from a database based on their ID. It then inputs the meal history and health goals from the database and uses the output data to generate a menu for the next day based on a machine learning model. For example, it might suggest yogurt and fruit for breakfast, chicken salad for lunch, and vegetable soup for dinner. The generated menu information is then converted to JSON format and sent to the terminal. In this case, the input is the meal history and health goals, and the output is the generated menu information in JSON format.

[0647] Step 6:

[0648] Displaying menu information for the following day

[0649] The terminal parses the received menu information and displays to the user, "For tomorrow's menu, we recommend yogurt and fruit for breakfast, chicken salad for lunch, and vegetable soup for dinner." Here, the input is menu information in JSON format, and the output is a display of the parsed menu information.

[0650] Step 7:

[0651] Calculation of Recommended Exercise Amount

[0652] The server retrieves the total calorie intake for the day from the database. For example, if a user has consumed 1800 kcal and their target intake is 1500 kcal, the server calculates the difference of 300 kcal. Based on this difference, it generates an exercise plan to burn 300 kcal (e.g., 30 minutes of jogging, 60 minutes of walking). In this step, the input is the calorie intake information retrieved from the database, and the output is the generated exercise plan.

[0653] Step 8:

[0654] Displaying exercise plan

[0655] The server sends the exercise plan to the terminal in JSON format. The server takes the exercise plan (e.g., {"exercise plan": "30 minutes of jogging or 60 minutes of walking"}) as input, converts it to JSON, and sends it to the terminal. The terminal parses the received exercise plan and displays to the user, "To burn off today's calorie intake, we recommend 30 minutes of jogging or 60 minutes of walking." Here again, the input is the exercise plan in JSON format, and the output is a display of the parsed exercise plan.

[0656] (Application Example 1)

[0657] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0658] In modern society, there is a growing concern for health, and it is essential to understand the calorie content of meals and maintain a healthy diet. However, calculating calories and creating appropriate meal and exercise plans in daily life is time-consuming and difficult to maintain. In particular, users of food delivery services have a need to easily find out the calorie information of the meals they order, but current technology lacks a system that can provide this information automatically and accurately.

[0659] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0660] In this invention, the server includes means for the user to take an image of a meal, means for uploading the image to the server, means for the server to analyze the image and obtain ingredient information and calorie information, means for transmitting the calorie information to the user's terminal, means for displaying the calorie information on the terminal, means for the server to suggest the next day's menu based on the user's meal history and health goals, means for transmitting the next day's menu information to the user's terminal, means for displaying the menu information on the terminal, means for the server to calculate the required amount of exercise and transmit an exercise plan to the user's terminal, means for providing calorie information of the order and guiding the user to suggest the next meal and exercise plan when using the food delivery service, and means for displaying the exercise plan on the terminal. As a result, when a user uses the food delivery service, they can easily obtain calorie information of their order and check healthy meal suggestions and necessary exercise plans for the next meal.

[0661] "User" refers to any person who uses the system of the present invention.

[0662] "Food images" refers to photographic data of meals taken by users.

[0663] A "server" refers to a central computer system that performs image analysis, data processing, and the transmission and reception of various types of information.

[0664] "Device" refers to a device used by a user (e.g., smartphone, tablet).

[0665] "Method for uploading images" refers to the function that sends image data of a meal from the device to the server.

[0666] "Means for analyzing images" refers to a function that processes and extracts food ingredient information from uploaded images.

[0667] "Ingredient information" refers to the specific types and contents of ingredients obtained from the image analysis results.

[0668] "Calorie information" refers to data on the amount of energy calculated based on information about the ingredients.

[0669] "Meal history" refers to a record of meals a user has eaten in the past.

[0670] "Health goals" refer to objectives or targets set by the user for maintaining or improving their health (e.g., weight loss, muscle building).

[0671] "Next day's menu information" refers to the meal plan for the following day, suggested based on the user's health goals.

[0672] "Required exercise" refers to the amount of energy that should be expended, calculated based on the daily calorie intake.

[0673] An "exercise plan" refers to the specific exercises recommended to burn the necessary amount of energy.

[0674] "Food delivery service" refers to any service where users order food online and have it delivered.

[0675] "Order details" refers to the specific details of the meals and food items selected and confirmed by the user through the food delivery service.

[0676] "Healthy next meal suggestion" refers to the content of the next meal recommended based on the user's health goals and eating history.

[0677] This invention relates to a system that allows users to take pictures of their meals, calculate calories based on those images, and then provide suggestions for their next meal and necessary exercise plans. Users take pictures of their meals using a device such as a smartphone or tablet and upload those images to a server.

[0678] The server analyzes the received image and retrieves information about the ingredients within it. This analysis uses an image analysis API (such as the Google Vision API). After obtaining a list of ingredients from the analysis results, the server searches the database for the calorie information of each ingredient and calculates the total calories. The server sends this calculation result to the terminal in JSON format, and the terminal displays the calorie information to the user.

[0679] Furthermore, the server suggests the next day's menu based on the user's meal history and health goals. A machine learning model is used for this menu suggestion. The menu information is sent in JSON format, and the device displays the next meal suggestion to the user. The server also calculates the amount of exercise needed based on the total calorie intake for the day and sends a specific exercise plan to the device. The device also displays the recommended exercise plan to the user.

[0680] A key feature of this system is its ability to provide calorie information for food delivery orders and offer suggestions for future meals and exercise plans. When a user orders food delivery, the order details are sent to the server upon confirmation. The server retrieves calorie information based on the order and calculates and presents suggestions for healthy meals and exercise plans for the next order.

[0681] As a concrete example, suppose a user eats toast, eggs, and a vegetable salad for breakfast. The user takes pictures of these meals with their device and uploads them to the server. The server uses an image analysis API to obtain information about the ingredients, adds up the calories of each, and sends the total to the user's device. For example, if the toast is 150kcal, the eggs are 70kcal, and the vegetable salad is 30kcal, the total is 250kcal. This information is then displayed on the device.

[0682] Next, when a user orders lunch using a food delivery service, the details (e.g., chicken salad) are sent to the server. The server analyzes the details, retrieves calorie information, displays it to the user, and also provides suggestions for the next meal (e.g., yogurt and fruit) and a necessary exercise plan (e.g., a 30-minute jog).

[0683] Example of a prompt:

[0684] User ID: 12345

[0685] Image of the meal: path_to_image.jpg

[0686] Please retrieve calorie information and suggest the next meal and exercise plan.

[0687] This system makes it easy for users to easily understand calorie information based on their meals and manage their health appropriately. Furthermore, even for meals ordered through food delivery services, users can check calorie information, healthy meal suggestions for the next meal, and exercise plans, providing support for maintaining a healthy lifestyle.

[0688] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0689] Step 1:

[0690] The user takes a picture of their meal. They use a smartphone or tablet to take a photo of the meal and save this image data to the device. The input is the image of the meal, and the output is the image data saved on the device.

[0691] Step 2:

[0692] Images stored on the device are uploaded to the server. The user operates the application to select an image and sends it to the server along with metadata such as the user ID and timestamp. The input is the image data and metadata stored on the device, and the output is the image data sent to the server.

[0693] Step 3:

[0694] The server analyzes the uploaded image. It then calls an image analysis API (e.g., Google Vision API) to extract information about the ingredients. The input is the image data sent to the server, and the output is the list of ingredients returned by the image analysis API.

[0695] Step 4:

[0696] The server retrieves calorie information based on the food ingredients. It searches the database for the calorie information of each ingredient and calculates the total calorie count. The input is a list of ingredients obtained from an image analysis API, and the output is the calorie information of each ingredient and the total calorie count.

[0697] Step 5:

[0698] The server sends the calculated calorie information to the terminal. It generates calorie information in JSON format and sends it to the user's terminal. The input is the calorie information calculated on the server side, and the output is the JSON data sent to the terminal.

[0699] Step 6:

[0700] This application displays calorie information received by the device. It parses the received JSON data and displays the calorie information in an easy-to-understand format for the user. The input is the received JSON data, and the output is the calorie information displayed on the device screen.

[0701] Step 7:

[0702] The server suggests the next day's menu based on the user's meal history and health goals. It retrieves the user's meal history and health goal data from a database and uses a machine learning model to generate the next meal suggestion. The input is the user's meal history and health goals, and the output is the suggested menu information for the following day.

[0703] Step 8:

[0704] The server sends the next day's menu information to the user's device. It generates the menu information in JSON format and sends it to the user's device. The input is the next day's menu information generated by a machine learning model, and the output is the JSON data sent to the device.

[0705] Step 9:

[0706] This application displays menu information received by the device. It parses the received JSON data and displays menu suggestions for the following day to the user. The input is the received JSON data, and the output is the menu information displayed on the device screen.

[0707] Step 10:

[0708] The server calculates the required amount of exercise and sends an exercise plan to the user's device. It calculates the required amount of exercise based on the total calorie intake for the day and generates a specific exercise plan. The calculated exercise plan is sent to the user's device in JSON format. The input is the total calorie intake for the day, and the output is the exercise plan information sent to the device.

[0709] Step 11:

[0710] This program displays the exercise plan received by the device. It parses the received JSON data and displays the recommended exercise plan for the user. The input is the received JSON data, and the output is the exercise plan information displayed on the device screen.

[0711] Step 12:

[0712] This system sends food delivery service order details to a server. When a user places an order using a food delivery service, the order details are sent to the server. The input is the food delivery service order details, and the output is the order data sent to the server.

[0713] Step 13:

[0714] The server retrieves calorie information based on the order and provides suggestions for the next meal and exercise plan. It analyzes the order to obtain calorie information and generates suggestions for the next meal and exercise plan. The input is the submitted order, and the output is calorie information, suggestions for the next meal, and an exercise plan.

[0715] Step 14:

[0716] Calorie information, meal suggestions for the next meal, and exercise plans are sent to the user's device, which then displays them. The input is the calorie information, meal suggestions for the next meal, and exercise plan sent from the server, and the output is this information displayed on the device's screen.

[0717] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0718] This invention relates to a system in which a user takes a picture of their meal, which is then automatically calculated based on the image, and which also provides menu suggestions for the following days and guidance on the amount of exercise needed. This system supports the user's healthy eating habits through communication between the server and the user's terminal. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it can provide more personalized support.

[0719] overview

[0720] The user takes a photo of their meal using a device such as a smartphone or tablet. The device uploads this photo to a server. The server uses image analysis technology to recognize the ingredients in the image and obtain their respective calorie information. Furthermore, the server suggests an appropriate menu and exercise plan for the next day based on the user's health goals. In this invention, an emotion engine is used to recognize the user's emotions and adjust the meal and exercise plan based on the user's emotions and psychological state.

[0721] Program Description

[0722] 1. Take a photo of your meal and upload the image.

[0723] The user takes a photo of their meal with their device. For example, they might take a photo of their breakfast toast, eggs, and vegetable salad.

[0724] The device reads the captured image from local storage and uploads it to the server along with metadata such as the user's ID and timestamp.

[0725] 2. Image analysis and calorie calculation

[0726] The server receives the uploaded image and calls an image analysis API (e.g., Google Vision API) to obtain information about the ingredients.

[0727] Based on the list of ingredients returned from the API (e.g., "toast", "eggs", "vegetable salad"), the calorie information for each ingredient is retrieved from the database.

[0728] Add up the calories of each ingredient to calculate the total calories. For example, if you have toast (150kcal), an egg (70kcal), and a vegetable salad (30kcal), the total calories would be 250kcal.

[0729] 3. Display of calorie information and user sentiment.

[0730] Along with the calculated calorie information, the server uses an emotion engine to recognize the user's emotions from their facial expressions, voice, and entered text.

[0731] The results will be sent to the terminal in JSON format.

[0732] The device parses the received JSON data and displays calorie information and perceived emotion information to the user. Example: "Your breakfast this morning was 250kcal. You seem to be feeling a little tired right now."

[0733] 4. Menu suggestions for the following day

[0734] The server retrieves the user's meal history from the database based on their ID. Furthermore, it takes into account the user's health goals (e.g., dieting).

[0735] Using a machine learning model, the system generates a menu for the next day based on the user's past eating history and health goals. The model also reflects the user's emotional state.

[0736] The generated menu information for the following day is sent to the terminal in JSON format.

[0737] The terminal parses the received menu information and displays it to the user. Example: "For tomorrow's menu, we recommend yogurt and fruit for breakfast, chicken salad for lunch, and vegetable soup for dinner."

[0738] 5. Guidance on recommended exercise levels

[0739] The server retrieves the total calorie intake for the day from the database. For example, let's say the total calories from all meals for the day are 1800 kcal.

[0740] The server calculates the difference between the user's target calorie intake and their current intake. If the target calorie intake is 1500kcal, the difference will be 300kcal.

[0741] Based on the difference, calculate an exercise plan to burn 300kcal. For example, assume that 30 minutes of jogging and 60 minutes of walking will burn 300kcal.

[0742] The exercise plan is adjusted based on the user's emotional state, as determined by the emotion engine. For example, if the user is "tired," it will suggest light, relaxing exercises.

[0743] A recommended exercise plan will be sent to your device in JSON format.

[0744] The device parses the received exercise plan and displays it to the user. Example: "To burn off today's calorie intake, we recommend a 30-minute jog or a 60-minute relaxing walk."

[0745] This system allows users to obtain appropriate calorie information from their diet and also suggests menus and exercise plans for the next day based on their health goals and emotional state, enabling them to live a healthier and less mentally stressful life.

[0746] The following describes the processing flow.

[0747] Step 1:

[0748] The user takes a picture of the meal with their device. The user takes a picture of the meal using the device's camera function, launches the application, and selects the picture.

[0749] Step 2:

[0750] The device reads photos taken from local storage and generates metadata such as the user's ID and timestamp. This integrates the photos with related information.

[0751] Step 3:

[0752] The device sends the photo and metadata to the server as an HTTP POST request. The data is securely transferred to the server using Internet protocols.

[0753] Step 4:

[0754] The server receives the uploaded photos and metadata. The received data is prepared for the image analysis process.

[0755] Step 5:

[0756] The server calls an image analysis API to analyze the content of the photograph. General-purpose image analysis services such as Google Vision API and Amazon Rekognition are used for the analysis API.

[0757] Step 6:

[0758] The server parses the list of ingredients received as an API response. For example, let's assume the response contains ingredient names such as "toast", "eggs", and "vegetable salad".

[0759] Step 7:

[0760] The server queries the database to retrieve the calorie value of each food item. The database stores information on various food items and their calorie content.

[0761] Step 8:

[0762] The server adds up the calories of each ingredient to calculate the total calories. For example, if toast is 150kcal, eggs are 70kcal, and a vegetable salad is 30kcal, the total calories would be 250kcal.

[0763] Step 9:

[0764] To recognize the user's emotions, the device uses its camera and microphone to capture the user's facial expressions and voice. Alternatively, the user fills out a questionnaire about their emotions.

[0765] Step 10:

[0766] The device uses an emotion engine to analyze the user's emotions from the acquired data. For example, it can determine if the user is feeling tired or stressed.

[0767] Step 11:

[0768] The server sends calorie information and recognized emotion information to the terminal in JSON format. The JSON data includes the analysis results, detailed calorie information, and recognized emotion.

[0769] Step 12:

[0770] The terminal parses the JSON data received from the server. The parsed data is then converted into a format that can be displayed to the user within the application.

[0771] Step 13:

[0772] The device displays the calories for each food item, the total calories, and the user's mood information. For example, it might display in the format: "Toast: 150kcal, Egg: 70kcal, Vegetable salad: 30kcal. Total calories: 250kcal. You seem to be feeling a little tired right now."

[0773] Step 14:

[0774] The server retrieves the user's meal history from the database based on their ID. Furthermore, it takes into account the user's health goals (e.g., dieting).

[0775] Step 15:

[0776] The server uses a machine learning model to generate a menu for the next day based on the user's past eating history and health goals. The model also reflects the user's emotional state.

[0777] Step 16:

[0778] The server generates the next day's menu information in JSON format and sends it to the terminal. The generated menu includes recommended foods and their quantities.

[0779] Step 17:

[0780] The terminal parses the menu information it receives. The parsed menu information is then displayed to the user in a visually easy-to-understand format.

[0781] Step 18:

[0782] The device displays the user's menu for the next day. For example, it might display something like, "For tomorrow's menu, we recommend yogurt and fruit for breakfast, chicken salad for lunch, and vegetable soup for dinner."

[0783] Step 19:

[0784] The server retrieves the total calorie intake for the day from the database. For example, let's say the total calories from all meals for the day are 1800 kcal.

[0785] Step 20:

[0786] The server calculates the difference between the user's target calorie intake and their current intake. If the target calorie intake is 1500kcal, the difference will be 300kcal.

[0787] Step 21:

[0788] The server calculates an exercise plan to burn 300kcal. For example, 30 minutes of jogging and 60 minutes of walking are assumed to burn 300kcal.

[0789] Step 22:

[0790] The server adjusts the exercise plan based on the user's emotional state, as determined by the emotion engine. For example, if the user is "tired," it suggests light, relaxing exercise.

[0791] Step 23:

[0792] The server generates a recommended exercise plan in JSON format and sends it to the device. The generated exercise plan includes specific exercise types and durations.

[0793] Step 24:

[0794] The device parses the received exercise plan information. The parsed information is converted into a format that can be displayed to the user.

[0795] Step 25:

[0796] The device displays an exercise plan recommended to the user. For example, it might say, "To burn off today's calorie intake, we recommend a 30-minute jog or a 60-minute relaxing walk."

[0797] (Example 2)

[0798] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0799] In recent years, the importance of users accurately understanding their diet and calorie expenditure, and implementing appropriate meal and exercise plans, has increased in order to maintain and improve a healthy lifestyle. However, conventional methods require manual recording and calculation, which is time-consuming and laborious, and they have the problem of not being able to take into account the user's emotions and psychological state. Furthermore, calorie calculation and exercise plan suggestions are unified, and do not provide optimal support for the user. This invention aims to solve these problems and provide a health management system that is more convenient and effective for users.

[0800] The identification processing by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for the user to take an image of a meal, means for uploading the image to the server, means for the server to analyze the image and obtain ingredient information and calorie information, means for transmitting the calorie information to the user's terminal, means for displaying the calorie information on the terminal, means for the server to suggest the next day's menu based on the user's meal history and health goals, means for recognizing the user's emotional state and adjusting the meal and exercise plan, means for transmitting the next day's menu information to the user's terminal, means for displaying the menu information on the terminal, means for the server to calculate the required amount of exercise based on the day's calorie intake and health goals and transmit the exercise plan to the user's terminal, and means for displaying the exercise plan on the terminal. As a result, the user can not only receive calorie calculations for the contents of their meals and menu suggestions for the next day, but also receive suggestions for exercise plans that take their emotional state into consideration.

[0801] A "user" is an individual who uses this system to manage their own health.

[0802] A "device" refers to an electronic device operated by a user, such as a smartphone or tablet.

[0803] A "server" is a computer that analyzes data sent from a terminal and provides the necessary information.

[0804] An "image analysis API" is an application program interface for recognizing food information within an image.

[0805] "Calorie information" refers to data that shows the energy intake of each food item.

[0806] "User's meal history" refers to a record of the meals a user has eaten in the past.

[0807] "Health goals" refer to health-related objectives set by the user, such as weight loss or muscle building.

[0808] "Tomorrow's menu" refers to the meal plan for the following day that is suggested to the user.

[0809] "Emotional state" refers to information that indicates the user's psychological or emotional condition.

[0810] An "exercise plan" is a plan of exercises that a user uses to burn off the calories they have consumed.

[0811] "Metadata" refers to supplementary information added to data, such as user IDs and timestamps.

[0812] A "machine learning model" is an algorithm that analyzes user data and performs predictions and classifications.

[0813] "JSON format" refers to a lightweight text format used for exchanging data.

[0814] This invention relates to a system that allows users to take pictures of their meals, automatically calculates calories based on those images, and then suggests menus for the following days and provides guidance on the amount of exercise needed. Specific proper nouns, names of people, and company names are converted to common names, and nouns are replaced with broader terms for explanation.

[0815] First, the user takes a picture of their meal using a device such as a smartphone or tablet. For example, consider a case where the user takes a picture of toast, eggs, and a vegetable salad for breakfast. The device saves this photo to local storage, adds metadata such as the user's ID and timestamp, and uploads it to the server.

[0816] The server receives the uploaded image and uses an image analysis API such as the Google Vision API to obtain information about the ingredients in the image. Based on the list of ingredients returned from the API (e.g., "toast", "egg", "vegetable salad"), it retrieves the calorie information for each ingredient from its internal database and sums them up. For example, if toast is 150kcal, egg is 70kcal, and vegetable salad is 30kcal, the total calorie count would be 250kcal.

[0817] Next, the server uses an emotion engine to recognize the user's emotions from their facial expressions, voice, and entered text. This calorie and emotion information is sent to the device in JSON format, and the device displays it. For example, a message such as "Your breakfast this morning was 250kcal. You seem to be feeling a little tired right now." might be displayed.

[0818] Furthermore, the server retrieves the user's meal history from a database based on their ID and suggests the next day's menu, taking into account the user's health goals and emotional state. The menu information, generated using a machine learning model, is sent to the device in JSON format, which the device then displays to the user. For example, a suggestion might be, "For tomorrow's menu, we recommend yogurt and fruit for breakfast, chicken salad for lunch, and vegetable soup for dinner."

[0819] The server also calculates the difference between the user's total calorie intake for the day and their target calorie intake, and then calculates a recommended amount of exercise based on this difference. The emotion engine checks the user's emotional state and adjusts the exercise plan accordingly. This is sent to the device in JSON format, and the device displays a message such as, "To burn off today's calorie intake, we recommend a 30-minute jog or a 60-minute relaxing walk."

[0820] This system can automatically obtain appropriate calorie information from the user's diet and further supports a healthier and less mentally stressful lifestyle by suggesting menus and exercise plans for the next day based on health goals and emotional state.

[0821] Example: Example of a prompt message to input to a generative AI model

[0822] Example 1: Taking photos of food and uploading the images

[0823] Prompt: "Please upload a photo of toast, eggs, and a vegetable salad taken with your smartphone to the server."

[0824] Example 2: Image analysis and calorie calculation

[0825] Prompt: "Retrieve ingredient information from the uploaded breakfast image and calculate the calories of each ingredient."

[0826] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0827] Step 1:

[0828] The user takes a photo of their meal with their device. For example, the process of taking a photo of toast, eggs, and a vegetable salad for breakfast is described.

[0829] Input: The user launches the camera app and takes a picture of the food.

[0830] Output: Image files of the meal saved on the device.

[0831] Specific action: A photo of breakfast is taken using the smartphone's camera app, and that image is saved to the device's local storage.

[0832] Step 2:

[0833] The device reads the captured image from local storage and uploads it to the server along with metadata such as the user's ID and timestamp.

[0834] Input: Image file of the meal saved on the device, user ID, timestamp

[0835] Output: Image data and metadata transferred to the server

[0836] Specific action: The device sends the image and metadata to the server as an HTTP POST request.

[0837] Step 3:

[0838] The server receives the uploaded image and calls an image analysis API to retrieve information about the ingredients in the image.

[0839] Input: Image data sent from the device

[0840] Output: List of ingredients returned from the image analysis API

[0841] Specific operation: The server sends images to an image analysis API such as the Google Vision API and receives the analysis results.

[0842] Step 4:

[0843] Based on the acquired food information, the server retrieves calorie information for each food item from the database.

[0844] Input: List of ingredients obtained from an image analysis API

[0845] Output: Calorie information for each food item retrieved from the database.

[0846] Specific operation: The server sends a query to an internal database to retrieve calorie information, for example, toast (150kcal), eggs (70kcal), and a vegetable salad (30kcal).

[0847] Step 5:

[0848] The server adds up the calories of each ingredient and calculates the total calories.

[0849] Input: Calorie information for each ingredient

[0850] Output: Total calorie value

[0851] Specific operation: The server adds up the calories of each ingredient and calculates a total calorie count of 250kcal as an example.

[0852] Step 6:

[0853] The server uses an emotion engine to recognize emotions from the user's facial expressions, voice, and input text.

[0854] Input: User facial image, audio data, input text

[0855] Output: Emotional state data returned from the emotion engine.

[0856] Specific operation: The server sends user data to the emotion engine and receives the emotion analysis results.

[0857] Step 7:

[0858] The server sends calorie information and emotional information to the terminal in JSON format.

[0859] Input: Total calorie value, emotional state data

[0860] Output: Data converted to JSON format

[0861] Specific operation: The server converts calorie information and emotion information into JSON and sends it to the terminal as an HTTP response.

[0862] Step 8:

[0863] The device parses the received JSON data and displays calorie information and emotional information to the user.

[0864] Input: Calorie information and emotion information in JSON format

[0865] Output: Calorie information and emotional information displayed on the screen

[0866] Specific action: The terminal parses the JSON data and displays the message, "Your breakfast this morning was 250kcal. You seem to be feeling a little tired right now."

[0867] Step 9:

[0868] The server retrieves past meal history from a database based on the user's ID and suggests the next day's menu, taking into account the user's health goals and emotional state.

[0869] Input: User ID, health goal information, emotional state

[0870] Output: Suggested menu for the next day

[0871] Specific operation: The server issues SQL queries to retrieve meal history and health goals, and uses a machine learning model to generate the next day's menu.

[0872] Step 10:

[0873] The server sends the suggested menu information for the following day to the terminal in JSON format.

[0874] Input: Suggestion menu data

[0875] Output: Suggested menu converted to JSON format

[0876] Specific operation: The server converts the suggested menu into JSON format and sends it to the terminal as an HTTP response.

[0877] Step 11:

[0878] The terminal analyzes the received menu information and displays it to the user.

[0879] Input: Suggestion menu in JSON format

[0880] Output: Suggestion menu displayed on the screen

[0881] Specific action: The terminal parses the JSON data and displays the message, "For tomorrow's menu, we recommend yogurt and fruit for breakfast, chicken salad for lunch, and vegetable soup for dinner."

[0882] Step 12:

[0883] The server calculates the difference between the user's total calorie intake for the day and their target calorie intake, and then calculates the amount of exercise required based on that difference.

[0884] Input: Total calorie intake for the day, target calorie intake

[0885] Output: Recommended exercise plan

[0886] Specific operation: The server calculates an exercise plan that burns, for example, 300 kcal, based on the difference.

[0887] Step 13:

[0888] The server checks the user's emotional state based on the emotion engine and adjusts the exercise plan accordingly.

[0889] Input: Recommended exercise plan, emotional state

[0890] Output: Adjusted exercise plan

[0891] Specific action: The server suggests a relaxing exercise plan (e.g., a 60-minute walk) when the user is tired.

[0892] Step 14:

[0893] The server sends a recommended exercise plan to the device in JSON format.

[0894] Input: Data from the adjusted exercise plan

[0895] Output: Exercise plan converted to JSON format

[0896] Specific operation: The server converts the motion plan into JSON format and sends it to the terminal as an HTTP response.

[0897] Step 15:

[0898] The device analyzes the received exercise plan and displays it to the user.

[0899] Input: Exercise plan in JSON format

[0900] Output: Exercise plan displayed on the screen

[0901] Specific action: The device parses the JSON data and displays the message, "To burn off today's calorie intake, we recommend a 30-minute jog or a 60-minute relaxing walk."

[0902] (Application Example 2)

[0903] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0904] In modern society, it is crucial for users to manage their own health and implement appropriate dietary and exercise plans. However, there are limited ways for users to access real-time calorie information and health recommendations when choosing food. Furthermore, it is difficult to provide health recommendations that take into account the user's emotional state. Therefore, there is a need for a system that enables users to make appropriate dietary choices and implement exercise plans based on their emotional state and health goals.

[0905] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0906] In this invention, the server includes means for the user to take an image of a meal, means for uploading the image to the server, means for the server to analyze the image and obtain ingredient information and calorie information, means for transmitting the calorie information to the user's terminal, means for displaying the calorie information on the terminal, means for the server to suggest the next day's menu based on the user's meal history and health goals, means for transmitting the next day's menu information to the user's terminal, means for displaying the menu information on the terminal, means for the server to calculate the required amount of exercise and transmit an exercise plan to the user's terminal, means for displaying the exercise plan on the terminal, means for a head-mounted display worn by the user to recognize ingredients and food products and display calorie information in real time, and means for the head-mounted display to recognize the user's emotional state and adjust health suggestions accordingly. As a result, when the user selects food in a store, they can check calorie information in real time and receive health suggestions tailored to their emotional state.

[0907] A "user" refers to an individual who uses the system to take photos of their meals or manage their health.

[0908] "Food images" refer to images taken by users to record the contents of their meals.

[0909] A "server" refers to a device or computer system that processes data sent by a user, calculates ingredient information and calorie information, and transmits it to the user's terminal.

[0910] "Uploading" refers to the operation or process of sending data from a user's device to a server.

[0911] "Image analysis" refers to the process of extracting information about ingredients and calories from images of food that have been photographed.

[0912] "Ingredient information" refers to data about the types and characteristics of each ingredient included in a meal.

[0913] "Calorie information" refers to data that shows the energy content of food and ingredients.

[0914] "Device" refers to a smartphone, tablet, or other electronic device owned by the user.

[0915] "Meal history" refers to data that records the types of meals a user has eaten in the past.

[0916] "Health goals" refer to health-related achievement goals set by the user (e.g., weight loss, muscle gain, etc.).

[0917] "Menu suggestions" refer to meal menus that are suggested based on the user's health goals and eating history.

[0918] An "exercise plan" refers to an exercise plan proposed to help the user achieve their desired level of physical activity.

[0919] A "head-mounted display" refers to a display device that a user wears on their head (e.g., a head-mounted display).

[0920] "Emotional state" refers to the user's psychological or emotional state.

[0921] "Health suggestions" refer to suggestions regarding diet and exercise that are tailored based on the user's emotional state and health goals.

[0922] "Real-time" refers to operations and data processing being performed instantly.

[0923] "Recognition" refers to the process by which a system extracts and understands specific information from input data (e.g., images or audio).

[0924] This invention is a system that supports users' health management by utilizing a head-mounted display (HMD) that users use on a daily basis. This system consistently performs tasks such as capturing images of meals, calculating calories, recognizing emotional states, and providing health recommendations. The specific implementation of the system is described below.

[0925] Hardware and software to be used

[0926] 1. Head-mounted display (HMD):

[0927] It is used by users when selecting food items in a store. The HMD has a built-in camera and display, and displays calorie information in real time when the user takes a picture of the food.

[0928] A typical HMD device is used as a concrete example.

[0929] 2. Server:

[0930] The system performs image analysis and calorie calculation, manages a database of meal history, generates health recommendations, and recognizes the user's emotional state. The server utilizes image analysis APIs (e.g., general image analysis services) and the EmotionRecognition library.

[0931] 3. User terminal:

[0932] The system displays calorie information and health suggestions sent from the server on the user's smartphone, tablet, or other device.

[0933] System processing and functions

[0934] 1. Take and upload photos of your meal:

[0935] The user takes pictures of food using the HMD's camera. The captured images are temporarily stored in the HMD's local storage and then uploaded to the server.

[0936] 2. Image analysis and acquisition of calorie information:

[0937] The server sends the received image to an image analysis API to obtain ingredient and calorie information. This allows detailed nutritional data of the food photographed by the user to be stored on the server.

[0938] 3. Display of calorie information:

[0939] The acquired calorie information is displayed in real time on the user's HMD (Head-Mounted Display). This allows the user to check the calorie content of food on the spot.

[0940] 4. Recognition of emotional states:

[0941] The HMD's camera and microphone are used to capture the user's facial expressions and voice, and an emotion recognition library is used to analyze the user's emotional state. The results are also sent to the server.

[0942] 5. Generating and displaying health suggestions:

[0943] The server generates health suggestions, including menu suggestions for the next day and recommended exercise levels, based on the user's eating history, health goals, and current emotional state. The generated suggestions are sent to and displayed on the user's HMD and device.

[0944] Specific example

[0945] For example, when a user chooses a sandwich at a cafe, the HMD (Head-Mounted Display) takes a picture of the sandwich and instantly displays its calorie count. Furthermore, if the server determines that the user has been lacking exercise recently and is feeling a bit tired, it will suggest light walking or relaxing exercises.

[0946] Example of a prompt

[0947] "Create a model for an application that calculates calories based on food images taken by the user using an HMD (Head-Mounted Display) and displays health suggestions tailored to the user's emotional state. The model will use a common image analysis service to obtain calorie information from food ingredients and the EmotionRecognition library to recognize the user's emotions. Implement the generated program in Python and explain the overall flow."

[0948] This system allows users to check calorie information on food in real time and select appropriate meals and exercise based on their health and emotional state.

[0949] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0950] Step 1:

[0951] The user takes a picture of their meal using the camera on a head-mounted display (HMD). Specifically, the user points the HMD's camera at the meal and the image is saved to local storage. The input is the image of the meal, and the output is the image file saved to local storage.

[0952] Step 2:

[0953] The device uploads the saved image to the server. This operation uses a communication module to send the image file over the internet. The input is the image file in local storage, and the output is the image data sent to the server.

[0954] Step 3:

[0955] The server analyzes the received image to obtain information about the ingredients and calories. Specifically, the server calls an image analysis API to recognize the ingredients in the image and obtain the component data for each ingredient. The input is the uploaded image of the meal, and the output is the ingredient information and its corresponding calorie information.

[0956] Step 4:

[0957] The server sends the acquired calorie information to the user's terminal. The server packages the calorie information in JSON format and sends it to the terminal. The input is the calorie information and the user ID, and the output is the JSON data sent to the terminal.

[0958] Step 5:

[0959] The terminal analyzes and displays the received calorie information. The terminal parses the JSON data and displays the calorie information clearly on the screen. The input is the JSON data received from the server, and the output is the calorie information displayed on the terminal's screen.

[0960] Step 6:

[0961] The server analyzes the user's facial expressions and voice, captured by the HMD's camera and microphone, to recognize the user's emotional state. It uses an emotion recognition library to analyze the emotional state and obtain the results. The input is the user's facial expression and voice data sent from the HMD, and the output is emotional state data.

[0962] Step 7:

[0963] The server generates menu suggestions and exercise plans for the next day based on the user's meal history, health goals, and emotional state. It uses a machine learning model to create optimal suggestions considering each data point. Inputs are the user's meal history, health goals, and emotional state, while outputs are menu suggestions and exercise plans for the next day.

[0964] Step 8:

[0965] The server sends the generated menu suggestions and exercise plan to the user's device and HMD. The packaged data is sent in JSON format, and receipt is confirmed. The input is the menu suggestion and exercise plan data, and the output is the suggestion data sent to the device and HMD.

[0966] Step 9:

[0967] The terminal and HMD display the received menu suggestions and exercise plans. The user reviews the suggestions displayed on the screen and plans their next course of action. This allows the user to manage their health in real time. The input is the suggestion data sent to the terminal and HMD, and the output is the displayed menu suggestions and exercise plans.

[0968] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0969] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0970] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0971] [Third Embodiment]

[0972] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0973] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0974] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0975] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0976] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0977] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0978] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0979] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0980] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0981] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0982] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0983] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0984] This invention relates to a system in which a user takes a picture of their meal, which is then automatically calculated based on the image, and which also provides menu suggestions for the following days and guidance on the amount of exercise needed. This system supports the user's healthy eating habits through communication between a server and the user's terminal.

[0985] overview

[0986] Users take photos of their meals using devices such as smartphones or tablets. These photos are uploaded to a server. The server uses image analysis technology to recognize the ingredients in the images and obtain their calorie information. Furthermore, the server suggests appropriate menus and exercise plans for the following day based on the user's health goals.

[0987] Program Description

[0988] 1. Take a photo of your meal and upload the image.

[0989] The user takes a photo of their meal with their device. For example, they might take a photo of their breakfast toast, eggs, and vegetable salad.

[0990] The device reads the captured image from local storage and uploads it to the server along with metadata such as the user's ID and timestamp.

[0991] 2. Image analysis and calorie calculation

[0992] The server receives the uploaded image and calls an image analysis API (e.g., Google Vision API) to obtain information about the ingredients.

[0993] Based on the list of ingredients returned from the API (e.g., "toast", "eggs", "vegetable salad"), the calorie information for each ingredient is retrieved from the database.

[0994] Add up the calories of each ingredient to calculate the total calories. For example, if you have toast (150kcal), an egg (70kcal), and a vegetable salad (30kcal), the total calories would be 250kcal.

[0995] 3. Display of calorie information

[0996] The server sends the calculated calorie information to the terminal in JSON format.

[0997] The device parses the received JSON data and displays calorie information to the user. Example: "This morning's breakfast was 250 kcal."

[0998] 4. Menu suggestions for the following day

[0999] The server retrieves the user's meal history from the database based on their ID. Furthermore, it takes into account the user's health goals (e.g., dieting).

[1000] Using a machine learning model, the system generates a menu for the next day based on the user's past eating history and health goals. For example, it might suggest a low-calorie, high-protein meal.

[1001] The generated menu information for the following day is sent to the terminal in JSON format.

[1002] The terminal parses the received menu information and displays it to the user. Example: "For tomorrow's menu, we recommend yogurt and fruit for breakfast, chicken salad for lunch, and vegetable soup for dinner."

[1003] 5. Guidance on recommended exercise levels

[1004] The server retrieves the total calorie intake for the day from the database. For example, if the intake is 1800kcal, it calculates the difference of 300kcal between that and the target intake of 1500kcal.

[1005] Based on the difference, calculate an exercise plan to burn 300kcal. Example: 30 minutes of jogging, 60 minutes of walking.

[1006] A recommended exercise plan will be sent to your device in JSON format.

[1007] The device parses the received exercise plan and displays it to the user. Example: "To burn off today's calorie intake, we recommend 30 minutes of jogging or 60 minutes of walking."

[1008] This system allows users to obtain appropriate calorie information from their meals and, furthermore, suggests menus and exercise plans for the next day based on their health goals, enabling them to live a healthy lifestyle.

[1009] The following describes the processing flow.

[1010] Step 1:

[1011] The user takes a photo of the meal with their device. The user takes a photo of the meal using the device's camera function, launches the application, and selects the photo.

[1012] Step 2:

[1013] The device reads photos taken from local storage and generates metadata such as the user's ID and timestamp. This integrates the photos with related information.

[1014] Step 3:

[1015] The device sends the photo and metadata to the server as an HTTP POST request. The data is securely transferred to the server using Internet protocols.

[1016] Step 4:

[1017] The server receives the uploaded photos and metadata. The received data is prepared for the image analysis process.

[1018] Step 5:

[1019] The server calls an image analysis API to analyze the content of the photograph. The analysis API uses general-purpose image analysis services such as Google Vision API or Amazon Rekognition.

[1020] Step 6:

[1021] The server parses the list of ingredients received as an API response. For example, let's assume the response contains ingredient names such as "toast", "eggs", and "vegetable salad".

[1022] Step 7:

[1023] The server queries the database to retrieve the calorie value of each food item. The database stores information on various food items and their calorie content.

[1024] Step 8:

[1025] The server adds up the calories of each ingredient to calculate the total calories. For example, if toast is 150kcal, eggs are 70kcal, and a vegetable salad is 30kcal, the total calories would be 250kcal.

[1026] Step 9:

[1027] The server generates the calculation results (list of ingredients and calorie information) in JSON format and sends them back to the terminal. The JSON data includes the analysis results and detailed calorie information.

[1028] Step 10:

[1029] The terminal parses the JSON data received from the server. The parsed data is then converted into a format that can be displayed to the user within the application.

[1030] Step 11:

[1031] The device displays the calorie content of each food item and the total calorie content to the user. For example, it might display in the format: "Toast: 150kcal, Egg: 70kcal, Vegetable salad: 30kcal. Total calories: 250kcal."

[1032] Step 12:

[1033] The server retrieves the user's meal history and health goals from the database. Health goals include things like weight loss and muscle gain.

[1034] Step 13:

[1035] The server uses a machine learning model to generate a menu for the next day based on the user's past eating history and health goals. The model calculates the optimal meal for the user's health condition.

[1036] Step 14:

[1037] The server generates the next day's menu information in JSON format and sends it to the terminal. The generated menu includes recommended foods and their quantities.

[1038] Step 15:

[1039] The terminal parses the menu information it receives. The parsed menu information is then displayed to the user in a visually easy-to-understand format.

[1040] Step 16:

[1041] The device displays the user's menu for the next day. For example, it might display something like, "We recommend yogurt and fruit for breakfast, chicken salad for lunch, and vegetable soup for dinner."

[1042] Step 17:

[1043] The server retrieves the total calorie intake for the day from the database. For example, let's say the total calories from all meals for the day are 1800 kcal.

[1044] Step 18:

[1045] The server calculates the difference between the user's target calorie intake and their current intake. If the target calorie intake is 1500kcal, the difference will be 300kcal.

[1046] Step 19:

[1047] The server calculates an exercise plan to burn 300kcal. For example, 30 minutes of jogging or 60 minutes of walking might be considered the amount of exercise that burns 300kcal.

[1048] Step 20:

[1049] The server generates a recommended exercise plan in JSON format and sends it to the device. The generated exercise plan includes specific exercise types and durations.

[1050] Step 21:

[1051] The device parses the received exercise plan information. The parsed information is converted into a format that can be displayed to the user.

[1052] Step 22:

[1053] The device displays an exercise plan recommended to the user. For example, it might display something like, "To burn off today's calorie intake, we recommend a 30-minute jog or a 60-minute walk."

[1054] (Example 1)

[1055] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1056] In today's lifestyle, many people struggle to maintain a healthy diet. In particular, excessive calorie intake and insufficient exercise lead to obesity and other health problems. Traditional health management systems often require users to manually input calorie calculations and create exercise plans. Against this backdrop, there is a need for a system that allows users to easily record their meals and centrally manage calorie information, optimal menu suggestions, and exercise plans.

[1057] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1058] In this invention, the server includes means for the user to take an image of a meal, means for uploading the image to the server, means for the server to analyze the image and obtain ingredient information and calorie information, means for transmitting the calorie information to the user's terminal, means for displaying the calorie information on the terminal, means for the server to suggest the next day's menu based on the user's meal history and health goals, means for transmitting the next day's menu information to the user's terminal, means for displaying the menu information on the terminal, means for the server to calculate the required amount of exercise and transmit an exercise plan to the user's terminal, and means for displaying the exercise plan on the terminal. As a result, the user can easily upload an image to automatically obtain ingredient information and calorie information and receive menu suggestions and exercise plans tailored to their health goals.

[1059] 1. A "user" refers to an individual who uses the system to take pictures of their meals and receives calorie information, menu suggestions, and exercise plans.

[1060] 2. "Terminal" refers to a device such as a smartphone or tablet used by the user, which is used to take pictures of food or to receive and display information from a server.

[1061] 3. A "server" refers to a computer system that analyzes images sent by users, obtains information on ingredients and calories, and generates menu suggestions and exercise plans to send to the user's terminal.

[1062] 4. "Image analysis" is the process that a server performs to obtain information about ingredients from images of food, and is generally carried out using an image analysis API.

[1063] 5. "Calorie information" refers to the amount of energy calculated by the server based on ingredient information obtained from the image of the meal.

[1064] 6. "Menu information" refers to the meal plan for the following day that the server generates based on the user's meal history and health goals.

[1065] 7. "Exercise plan" refers to the type and amount of exercise suggested by the server based on the user's calorie intake.

[1066] 8. "Image analysis API" refers to an application programming interface used by a server to analyze images of food and obtain information about the ingredients.

[1067] 9. "JSON format" refers to the JavaScript Object Notation format used for lightweight and easy data exchange.

[1068] 10. A "machine learning model" refers to the algorithms and analytical models that the server uses to generate the next day's menu based on the user's past eating history and health goals.

[1069] 11. "Meal history" refers to data that records information about meals a user has eaten in the past.

[1070] 12. "Health goals" refer to health-related goals that users wish to achieve, such as weight loss or muscle building.

[1071] This invention relates to a system in which a user takes a picture of their meal, which is then automatically calculated based on the image, and which also provides menu suggestions for the following days and guidance on the amount of exercise needed. This system supports the user's healthy eating habits through communication between a server and the user's terminal.

[1072] overview

[1073] Taking photos of food and uploading the images

[1074] The user takes a photo of their meal using a device such as a smartphone or tablet. For example, they might take a photo of their breakfast toast, eggs, and vegetable salad.

[1075] The device reads this photo from local storage and uploads it to the server along with metadata such as the user's ID and timestamp.

[1076] Image analysis and calorie calculation

[1077] The server receives the uploaded image and calls an image analysis API (e.g., Google Vision API) to obtain information about the ingredients.

[1078] Based on the retrieved list of ingredients (e.g., "toast", "eggs", "vegetable salad"), the server retrieves calorie information for each ingredient from the database.

[1079] The server adds up the calories of each ingredient to calculate the total calorie count. For example, if you have toast (150kcal), an egg (70kcal), and a vegetable salad (30kcal), the total calorie count will be 250kcal.

[1080] Display of calorie information

[1081] The server sends the calculated calorie information to the terminal in JSON format.

[1082] The device parses the received JSON data and displays calorie information to the user. For example, it might display, "This morning's breakfast was 250 kcal."

[1083] Menu suggestions for the next day

[1084] The server retrieves the user's meal history and health goals from the database based on their ID. Furthermore, it takes into account the user's health goals (e.g., diet).

[1085] The server uses a machine learning model to generate a menu for the next day based on the user's past eating history and health goals. For example, it might suggest a low-calorie, high-protein meal.

[1086] The generated menu information for the following day is sent to the terminal in JSON format.

[1087] The terminal parses the received menu information and displays it to the user. For example, it might display, "For tomorrow's menu, we recommend yogurt and fruit for breakfast, chicken salad for lunch, and vegetable soup for dinner."

[1088] Guide to recommended exercise levels

[1089] The server retrieves the total calorie intake for the day from the database. For example, if the intake is 1800kcal, it calculates the difference of 300kcal between that and the target intake of 1500kcal.

[1090] The server calculates an exercise plan to burn 300kcal based on the difference. An example might include 30 minutes of jogging and 60 minutes of walking.

[1091] A recommended exercise plan will be sent to your device in JSON format.

[1092] The device parses the received exercise plan and displays it to the user. For example, it might display, "To burn off today's calorie intake, we recommend 30 minutes of jogging or 60 minutes of walking."

[1093] Specific example

[1094] For example, if a user takes a photo of "pancakes, bacon, and fruit" for breakfast, the process would proceed as follows:

[1095] The device uploads this photo to the server.

[1096] The server uses the Google Vision API to recognize these ingredients and retrieves their respective calorie counts from a database.

[1097] The total calories from the pancakes (200kcal), bacon (100kcal), and fruit (80kcal) amount to 380kcal. This is sent to the terminal in JSON format, and the user is shown "This morning's breakfast is 380kcal."

[1098] Furthermore, the server takes the user's past data and health goals into consideration and suggests "oatmeal, yogurt, and fruit" as breakfast for the next day. This is also sent to the terminal in JSON format and displays the message, "We recommend oatmeal, yogurt, and fruit for breakfast tomorrow."

[1099] Finally, the server calculates the calories to be burned and suggests "20 minutes of jogging or 50 minutes of walking," sending this information to the terminal. The terminal then displays a message to the user saying, "To burn off today's calorie intake, we recommend 20 minutes of jogging or 50 minutes of walking."

[1100] Example of a prompt

[1101] "A user has uploaded a photo of their breakfast (pancakes, bacon, fruit). Calculate the calories in these ingredients and display the total calories to the user. Also, suggest a healthy breakfast menu for the next day and display an optimal exercise plan to burn off the user's calories."

[1102] This system allows users to obtain appropriate calorie information from their meals and, furthermore, suggests menus and exercise plans for the next day based on their health goals, enabling them to live a healthy lifestyle.

[1103] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1104] Step 1:

[1105] Taking photos of food and uploading the images

[1106] The user takes a picture of their meal using a device such as a smartphone or tablet. For example, they might take a picture of toast, eggs, and a vegetable salad for breakfast. The captured image is saved to the device's local storage. The device then uploads this photo to the server along with metadata such as the user's ID and a timestamp. The input includes the image of the meal, the user's ID, and the timestamp, while the output is the image data sent to the server.

[1107] Step 2:

[1108] Image analysis and acquisition of food ingredient information

[1109] The server receives the uploaded image and saves it to the specified directory. Next, the server calls an image analysis API (e.g., an image analysis API) to obtain information about the ingredients in the image. Specifically, it sends the image data as input to the image analysis API and receives a list of identified ingredients (e.g., "toast", "egg", "vegetable salad") as output. In this step, the server passes the image data to the API and analyzes the data returned from the API.

[1110] Step 3:

[1111] Calorie calculation

[1112] The server retrieves calorie information for each ingredient from the database based on the retrieved ingredient list. Taking the ingredient list (e.g., "toast", "egg", "vegetable salad") as input, it retrieves the corresponding calorie values ​​from the database (e.g., toast is 150kcal, egg is 70kcal, vegetable salad is 30kcal) as output. The server then sums these calories to calculate the total calorie count (e.g., 250kcal). In this case, the input is the ingredient list, the calorie information is from the database, and the output is the total calorie count.

[1113] Step 4:

[1114] Display of calorie information

[1115] The server sends the calculated calorie information to the terminal in JSON format. It takes calorie information (e.g., {"Breakfast Calories": 250}) as input, converts it to JSON, and sends it to the terminal. The terminal parses the received JSON data and displays "This morning's breakfast was 250kcal." to the user. Here, the input is calorie information in JSON format, and the output is the display of the parsed calorie information.

[1116] Step 5:

[1117] Menu suggestions for the next day

[1118] The server retrieves the user's past meal history and health goals from a database based on their ID. It then inputs the meal history and health goals from the database and uses the output data to generate a menu for the next day based on a machine learning model. For example, it might suggest yogurt and fruit for breakfast, chicken salad for lunch, and vegetable soup for dinner. The generated menu information is then converted to JSON format and sent to the terminal. In this case, the input is the meal history and health goals, and the output is the generated menu information in JSON format.

[1119] Step 6:

[1120] Displaying menu information for the following day

[1121] The terminal parses the received menu information and displays to the user, "For tomorrow's menu, we recommend yogurt and fruit for breakfast, chicken salad for lunch, and vegetable soup for dinner." Here, the input is menu information in JSON format, and the output is a display of the parsed menu information.

[1122] Step 7:

[1123] Calculation of Recommended Exercise Amount

[1124] The server retrieves the total calorie intake for the day from the database. For example, if a user has consumed 1800 kcal and their target intake is 1500 kcal, the server calculates the difference of 300 kcal. Based on this difference, it generates an exercise plan to burn 300 kcal (e.g., 30 minutes of jogging, 60 minutes of walking). In this step, the input is the calorie intake information retrieved from the database, and the output is the generated exercise plan.

[1125] Step 8:

[1126] Displaying exercise plan

[1127] The server sends the exercise plan to the terminal in JSON format. The server takes the exercise plan (e.g., {"exercise plan": "30 minutes of jogging or 60 minutes of walking"}) as input, converts it to JSON, and sends it to the terminal. The terminal parses the received exercise plan and displays to the user, "To burn off today's calorie intake, we recommend 30 minutes of jogging or 60 minutes of walking." Here again, the input is the exercise plan in JSON format, and the output is a display of the parsed exercise plan.

[1128] (Application Example 1)

[1129] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1130] In modern society, there is a growing concern for health, and it is essential to understand the calorie content of meals and maintain a healthy diet. However, calculating calories and creating appropriate meal and exercise plans in daily life is time-consuming and difficult to maintain. In particular, users of food delivery services have a need to easily find out the calorie information of the meals they order, but current technology lacks a system that can provide this information automatically and accurately.

[1131] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1132] In this invention, the server includes means for the user to take an image of a meal, means for uploading the image to the server, means for the server to analyze the image and obtain ingredient information and calorie information, means for transmitting the calorie information to the user's terminal, means for displaying the calorie information on the terminal, means for the server to suggest the next day's menu based on the user's meal history and health goals, means for transmitting the next day's menu information to the user's terminal, means for displaying the menu information on the terminal, means for the server to calculate the required amount of exercise and transmit an exercise plan to the user's terminal, means for providing calorie information of the order and guiding the user to suggest the next meal and exercise plan when using the food delivery service, and means for displaying the exercise plan on the terminal. As a result, when a user uses the food delivery service, they can easily obtain calorie information of their order and check healthy meal suggestions and necessary exercise plans for the next meal.

[1133] "User" refers to any person who uses the system of the present invention.

[1134] "Food images" refers to photographic data of meals taken by users.

[1135] A "server" refers to a central computer system that performs image analysis, data processing, and the transmission and reception of various types of information.

[1136] "Device" refers to a device used by a user (e.g., smartphone, tablet).

[1137] "Method for uploading images" refers to the function that sends image data of a meal from the device to the server.

[1138] "Means for analyzing images" refers to a function that processes and extracts food ingredient information from uploaded images.

[1139] "Ingredient information" refers to the specific types and contents of ingredients obtained from the image analysis results.

[1140] "Calorie information" refers to data on the amount of energy calculated based on information about the ingredients.

[1141] "Meal history" refers to a record of meals a user has eaten in the past.

[1142] "Health goals" refer to objectives or targets set by the user for maintaining or improving their health (e.g., weight loss, muscle building).

[1143] "Next day's menu information" refers to the meal plan for the following day, suggested based on the user's health goals.

[1144] "Required exercise" refers to the amount of energy that should be expended, calculated based on the daily calorie intake.

[1145] An "exercise plan" refers to the specific exercises recommended to burn the necessary amount of energy.

[1146] "Food delivery service" refers to any service where users order food online and have it delivered.

[1147] "Order details" refers to the specific details of the meals and food items selected and confirmed by the user through the food delivery service.

[1148] "Healthy next meal suggestion" refers to the content of the next meal recommended based on the user's health goals and eating history.

[1149] This invention relates to a system that allows users to take pictures of their meals, calculate calories based on those images, and then provide suggestions for their next meal and necessary exercise plans. Users take pictures of their meals using a device such as a smartphone or tablet and upload those images to a server.

[1150] The server analyzes the received image and retrieves information about the ingredients within it. This analysis uses an image analysis API (such as the Google Vision API). After obtaining a list of ingredients from the analysis results, the server searches the database for the calorie information of each ingredient and calculates the total calories. The server sends this calculation result to the terminal in JSON format, and the terminal displays the calorie information to the user.

[1151] Furthermore, the server suggests the next day's menu based on the user's meal history and health goals. A machine learning model is used for this menu suggestion. The menu information is sent in JSON format, and the device displays the next meal suggestion to the user. The server also calculates the amount of exercise needed based on the total calorie intake for the day and sends a specific exercise plan to the device. The device also displays the recommended exercise plan to the user.

[1152] A key feature of this system is its ability to provide calorie information for food delivery orders and offer suggestions for future meals and exercise plans. When a user orders food delivery, the order details are sent to the server upon confirmation. The server retrieves calorie information based on the order and calculates and presents suggestions for healthy meals and exercise plans for the next order.

[1153] As a concrete example, suppose a user eats toast, eggs, and a vegetable salad for breakfast. The user takes pictures of these meals with their device and uploads them to the server. The server uses an image analysis API to obtain information about the ingredients, adds up the calories of each, and sends the total to the user's device. For example, if the toast is 150kcal, the eggs are 70kcal, and the vegetable salad is 30kcal, the total is 250kcal. This information is then displayed on the device.

[1154] Next, when a user orders lunch using a food delivery service, the details (e.g., chicken salad) are sent to the server. The server analyzes the details, retrieves calorie information, displays it to the user, and also provides suggestions for the next meal (e.g., yogurt and fruit) and a necessary exercise plan (e.g., a 30-minute jog).

[1155] Example of a prompt:

[1156] User ID: 12345

[1157] Image of the meal: path_to_image.jpg

[1158] Please retrieve calorie information and suggest the next meal and exercise plan.

[1159] This system makes it easy for users to easily understand calorie information based on their meals and manage their health appropriately. Furthermore, even for meals ordered through food delivery services, users can check calorie information, healthy meal suggestions for the next meal, and exercise plans, providing support for maintaining a healthy lifestyle.

[1160] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1161] Step 1:

[1162] The user takes a picture of their meal. They use a smartphone or tablet to take a photo of the meal and save this image data to the device. The input is the image of the meal, and the output is the image data saved on the device.

[1163] Step 2:

[1164] Images stored on the device are uploaded to the server. The user operates the application to select an image and sends it to the server along with metadata such as the user ID and timestamp. The input is the image data and metadata stored on the device, and the output is the image data sent to the server.

[1165] Step 3:

[1166] The server analyzes the uploaded image. It then calls an image analysis API (e.g., Google Vision API) to extract information about the ingredients. The input is the image data sent to the server, and the output is the list of ingredients returned by the image analysis API.

[1167] Step 4:

[1168] The server retrieves calorie information based on the food ingredients. It searches the database for the calorie information of each ingredient and calculates the total calorie count. The input is a list of ingredients obtained from an image analysis API, and the output is the calorie information of each ingredient and the total calorie count.

[1169] Step 5:

[1170] The server sends the calculated calorie information to the terminal. It generates calorie information in JSON format and sends it to the user's terminal. The input is the calorie information calculated on the server side, and the output is the JSON data sent to the terminal.

[1171] Step 6:

[1172] This application displays calorie information received by the device. It parses the received JSON data and displays the calorie information in an easy-to-understand format for the user. The input is the received JSON data, and the output is the calorie information displayed on the device screen.

[1173] Step 7:

[1174] The server suggests the next day's menu based on the user's meal history and health goals. It retrieves the user's meal history and health goal data from a database and uses a machine learning model to generate the next meal suggestion. The input is the user's meal history and health goals, and the output is the suggested menu information for the following day.

[1175] Step 8:

[1176] The server sends the next day's menu information to the user's device. It generates the menu information in JSON format and sends it to the user's device. The input is the next day's menu information generated by a machine learning model, and the output is the JSON data sent to the device.

[1177] Step 9:

[1178] This application displays menu information received by the device. It parses the received JSON data and displays menu suggestions for the following day to the user. The input is the received JSON data, and the output is the menu information displayed on the device screen.

[1179] Step 10:

[1180] The server calculates the required amount of exercise and sends an exercise plan to the user's device. It calculates the required amount of exercise based on the total calorie intake for the day and generates a specific exercise plan. The calculated exercise plan is sent to the user's device in JSON format. The input is the total calorie intake for the day, and the output is the exercise plan information sent to the device.

[1181] Step 11:

[1182] This program displays the exercise plan received by the device. It parses the received JSON data and displays the recommended exercise plan for the user. The input is the received JSON data, and the output is the exercise plan information displayed on the device screen.

[1183] Step 12:

[1184] This system sends food delivery service order details to a server. When a user places an order using a food delivery service, the order details are sent to the server. The input is the food delivery service order details, and the output is the order data sent to the server.

[1185] Step 13:

[1186] The server retrieves calorie information based on the order and provides suggestions for the next meal and exercise plan. It analyzes the order to obtain calorie information and generates suggestions for the next meal and exercise plan. The input is the submitted order, and the output is calorie information, suggestions for the next meal, and an exercise plan.

[1187] Step 14:

[1188] Calorie information, meal suggestions for the next meal, and exercise plans are sent to the user's device, which then displays them. The input is the calorie information, meal suggestions for the next meal, and exercise plan sent from the server, and the output is this information displayed on the device's screen.

[1189] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1190] This invention relates to a system in which a user takes a picture of their meal, which is then automatically calculated based on the image, and which also provides menu suggestions for the following days and guidance on the amount of exercise needed. This system supports the user's healthy eating habits through communication between the server and the user's terminal. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it can provide more personalized support.

[1191] overview

[1192] The user takes a photo of their meal using a device such as a smartphone or tablet. The device uploads this photo to a server. The server uses image analysis technology to recognize the ingredients in the image and obtain their respective calorie information. Furthermore, the server suggests an appropriate menu and exercise plan for the next day based on the user's health goals. In this invention, an emotion engine is used to recognize the user's emotions and adjust the meal and exercise plan based on the user's emotions and psychological state.

[1193] Program Description

[1194] 1. Take a photo of your meal and upload the image.

[1195] The user takes a photo of their meal with their device. For example, they might take a photo of their breakfast toast, eggs, and vegetable salad.

[1196] The device reads the captured image from local storage and uploads it to the server along with metadata such as the user's ID and timestamp.

[1197] 2. Image analysis and calorie calculation

[1198] The server receives the uploaded image and calls an image analysis API (e.g., Google Vision API) to obtain information about the ingredients.

[1199] Based on the list of ingredients returned from the API (e.g., "toast", "eggs", "vegetable salad"), the calorie information for each ingredient is retrieved from the database.

[1200] Add up the calories of each ingredient to calculate the total calories. For example, if you have toast (150kcal), an egg (70kcal), and a vegetable salad (30kcal), the total calories would be 250kcal.

[1201] 3. Display of calorie information and user sentiment.

[1202] Along with the calculated calorie information, the server uses an emotion engine to recognize the user's emotions from their facial expressions, voice, and entered text.

[1203] The results will be sent to the terminal in JSON format.

[1204] The device parses the received JSON data and displays calorie information and perceived emotion information to the user. Example: "Your breakfast this morning was 250kcal. You seem to be feeling a little tired right now."

[1205] 4. Menu suggestions for the following day

[1206] The server retrieves the user's meal history from the database based on their ID. Furthermore, it takes into account the user's health goals (e.g., dieting).

[1207] Using a machine learning model, the system generates a menu for the next day based on the user's past eating history and health goals. The model also reflects the user's emotional state.

[1208] The generated menu information for the following day is sent to the terminal in JSON format.

[1209] The terminal parses the received menu information and displays it to the user. Example: "For tomorrow's menu, we recommend yogurt and fruit for breakfast, chicken salad for lunch, and vegetable soup for dinner."

[1210] 5. Guidance on recommended exercise levels

[1211] The server retrieves the total calorie intake for the day from the database. For example, let's say the total calories from all meals for the day are 1800 kcal.

[1212] The server calculates the difference between the user's target calorie intake and their current intake. If the target calorie intake is 1500kcal, the difference will be 300kcal.

[1213] Based on the difference, calculate an exercise plan to burn 300kcal. For example, assume that 30 minutes of jogging and 60 minutes of walking will burn 300kcal.

[1214] The exercise plan is adjusted based on the user's emotional state, as determined by the emotion engine. For example, if the user is "tired," it will suggest light, relaxing exercises.

[1215] A recommended exercise plan will be sent to your device in JSON format.

[1216] The device parses the received exercise plan and displays it to the user. Example: "To burn off today's calorie intake, we recommend a 30-minute jog or a 60-minute relaxing walk."

[1217] This system allows users to obtain appropriate calorie information from their diet and also suggests menus and exercise plans for the next day based on their health goals and emotional state, enabling them to live a healthier and less mentally stressful life.

[1218] The following describes the processing flow.

[1219] Step 1:

[1220] The user takes a picture of the meal with their device. The user takes a picture of the meal using the device's camera function, launches the application, and selects the picture.

[1221] Step 2:

[1222] The device reads photos taken from local storage and generates metadata such as the user's ID and timestamp. This integrates the photos with related information.

[1223] Step 3:

[1224] The device sends the photo and metadata to the server as an HTTP POST request. The data is securely transferred to the server using Internet protocols.

[1225] Step 4:

[1226] The server receives the uploaded photos and metadata. The received data is prepared for the image analysis process.

[1227] Step 5:

[1228] The server calls an image analysis API to analyze the content of the photograph. General-purpose image analysis services such as Google Vision API and Amazon Rekognition are used for the analysis API.

[1229] Step 6:

[1230] The server parses the list of ingredients received as an API response. For example, let's assume the response contains ingredient names such as "toast", "eggs", and "vegetable salad".

[1231] Step 7:

[1232] The server queries the database to retrieve the calorie value of each food item. The database stores information on various food items and their calorie content.

[1233] Step 8:

[1234] The server adds up the calories of each ingredient to calculate the total calories. For example, if toast is 150kcal, eggs are 70kcal, and a vegetable salad is 30kcal, the total calories would be 250kcal.

[1235] Step 9:

[1236] To recognize the user's emotions, the device uses its camera and microphone to capture the user's facial expressions and voice. Alternatively, the user fills out a questionnaire about their emotions.

[1237] Step 10:

[1238] The device uses an emotion engine to analyze the user's emotions from the acquired data. For example, it can determine if the user is feeling tired or stressed.

[1239] Step 11:

[1240] The server sends calorie information and recognized emotion information to the terminal in JSON format. The JSON data includes the analysis results, detailed calorie information, and recognized emotion.

[1241] Step 12:

[1242] The terminal parses the JSON data received from the server. The parsed data is then converted into a format that can be displayed to the user within the application.

[1243] Step 13:

[1244] The device displays the calories for each food item, the total calories, and the user's mood information. For example, it might display in the format: "Toast: 150kcal, Egg: 70kcal, Vegetable salad: 30kcal. Total calories: 250kcal. You seem to be feeling a little tired right now."

[1245] Step 14:

[1246] The server retrieves the user's meal history from the database based on their ID. Furthermore, it takes into account the user's health goals (e.g., dieting).

[1247] Step 15:

[1248] The server uses a machine learning model to generate a menu for the next day based on the user's past eating history and health goals. The model also reflects the user's emotional state.

[1249] Step 16:

[1250] The server generates the next day's menu information in JSON format and sends it to the terminal. The generated menu includes recommended foods and their quantities.

[1251] Step 17:

[1252] The terminal parses the menu information it receives. The parsed menu information is then displayed to the user in a visually easy-to-understand format.

[1253] Step 18:

[1254] The device displays the user's menu for the next day. For example, it might display something like, "For tomorrow's menu, we recommend yogurt and fruit for breakfast, chicken salad for lunch, and vegetable soup for dinner."

[1255] Step 19:

[1256] The server retrieves the total calorie intake for the day from the database. For example, let's say the total calories from all meals for the day are 1800 kcal.

[1257] Step 20:

[1258] The server calculates the difference between the user's target calorie intake and their current intake. If the target calorie intake is 1500kcal, the difference will be 300kcal.

[1259] Step 21:

[1260] The server calculates an exercise plan to burn 300kcal. For example, 30 minutes of jogging and 60 minutes of walking are assumed to burn 300kcal.

[1261] Step 22:

[1262] The server adjusts the exercise plan based on the user's emotional state, as determined by the emotion engine. For example, if the user is "tired," it suggests light, relaxing exercise.

[1263] Step 23:

[1264] The server generates a recommended exercise plan in JSON format and sends it to the device. The generated exercise plan includes specific exercise types and durations.

[1265] Step 24:

[1266] The device parses the received exercise plan information. The parsed information is converted into a format that can be displayed to the user.

[1267] Step 25:

[1268] The device displays an exercise plan recommended to the user. For example, it might say, "To burn off today's calorie intake, we recommend a 30-minute jog or a 60-minute relaxing walk."

[1269] (Example 2)

[1270] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1271] In recent years, the importance of users accurately understanding their diet and calorie expenditure, and implementing appropriate meal and exercise plans, has increased in order to maintain and improve a healthy lifestyle. However, conventional methods require manual recording and calculation, which is time-consuming and laborious, and they have the problem of not being able to take into account the user's emotions and psychological state. Furthermore, calorie calculation and exercise plan suggestions are unified, and do not provide optimal support for the user. This invention aims to solve these problems and provide a health management system that is more convenient and effective for users.

[1272] The identification processing by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for the user to take an image of a meal, means for uploading the image to the server, means for the server to analyze the image and obtain ingredient information and calorie information, means for transmitting the calorie information to the user's terminal, means for displaying the calorie information on the terminal, means for the server to suggest the next day's menu based on the user's meal history and health goals, means for recognizing the user's emotional state and adjusting the meal and exercise plan, means for transmitting the next day's menu information to the user's terminal, means for displaying the menu information on the terminal, means for the server to calculate the required amount of exercise based on the day's calorie intake and health goals and transmit the exercise plan to the user's terminal, and means for displaying the exercise plan on the terminal. As a result, the user can not only receive calorie calculations for the contents of their meals and menu suggestions for the next day, but also receive suggestions for exercise plans that take their emotional state into consideration.

[1273] A "user" is an individual who uses this system to manage their own health.

[1274] A "device" refers to an electronic device operated by a user, such as a smartphone or tablet.

[1275] A "server" is a computer that analyzes data sent from a terminal and provides the necessary information.

[1276] An "image analysis API" is an application program interface for recognizing food information within an image.

[1277] "Calorie information" refers to data that shows the energy intake of each food item.

[1278] "User's meal history" refers to a record of the meals a user has eaten in the past.

[1279] "Health goals" refer to health-related objectives set by the user, such as weight loss or muscle building.

[1280] "Tomorrow's menu" refers to the meal plan for the following day that is suggested to the user.

[1281] "Emotional state" refers to information that indicates the user's psychological or emotional condition.

[1282] An "exercise plan" is a plan of exercises that a user uses to burn off the calories they have consumed.

[1283] "Metadata" refers to supplementary information added to data, such as user IDs and timestamps.

[1284] A "machine learning model" is an algorithm that analyzes user data and performs predictions and classifications.

[1285] "JSON format" refers to a lightweight text format used for exchanging data.

[1286] This invention relates to a system that allows users to take pictures of their meals, automatically calculates calories based on those images, and then suggests menus for the following days and provides guidance on the amount of exercise needed. Specific proper nouns, names of people, and company names are converted to common names, and nouns are replaced with broader terms for explanation.

[1287] First, the user takes a picture of their meal using a device such as a smartphone or tablet. For example, consider a case where the user takes a picture of toast, eggs, and a vegetable salad for breakfast. The device saves this photo to local storage, adds metadata such as the user's ID and timestamp, and uploads it to the server.

[1288] The server receives the uploaded image and uses an image analysis API such as the Google Vision API to obtain information about the ingredients in the image. Based on the list of ingredients returned from the API (e.g., "toast", "egg", "vegetable salad"), it retrieves the calorie information for each ingredient from its internal database and sums them up. For example, if toast is 150kcal, egg is 70kcal, and vegetable salad is 30kcal, the total calorie count would be 250kcal.

[1289] Next, the server uses an emotion engine to recognize the user's emotions from their facial expressions, voice, and entered text. This calorie and emotion information is sent to the device in JSON format, and the device displays it. For example, a message such as "Your breakfast this morning was 250kcal. You seem to be feeling a little tired right now." might be displayed.

[1290] Furthermore, the server retrieves the user's meal history from a database based on their ID and suggests the next day's menu, taking into account the user's health goals and emotional state. The menu information, generated using a machine learning model, is sent to the device in JSON format, which the device then displays to the user. For example, a suggestion might be, "For tomorrow's menu, we recommend yogurt and fruit for breakfast, chicken salad for lunch, and vegetable soup for dinner."

[1291] The server also calculates the difference between the user's total calorie intake for the day and their target calorie intake, and then calculates a recommended amount of exercise based on this difference. The emotion engine checks the user's emotional state and adjusts the exercise plan accordingly. This is sent to the device in JSON format, and the device displays a message such as, "To burn off today's calorie intake, we recommend a 30-minute jog or a 60-minute relaxing walk."

[1292] This system can automatically obtain appropriate calorie information from the user's diet and further supports a healthier and less mentally stressful lifestyle by suggesting menus and exercise plans for the next day based on health goals and emotional state.

[1293] Example: Example of a prompt message to input to a generative AI model

[1294] Example 1: Taking photos of food and uploading the images

[1295] Prompt: "Please upload a photo of toast, eggs, and a vegetable salad taken with your smartphone to the server."

[1296] Example 2: Image analysis and calorie calculation

[1297] Prompt: "Retrieve ingredient information from the uploaded breakfast image and calculate the calories of each ingredient."

[1298] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1299] Step 1:

[1300] The user takes a photo of their meal with their device. For example, the process of taking a photo of toast, eggs, and a vegetable salad for breakfast is described.

[1301] Input: The user launches the camera app and takes a picture of the food.

[1302] Output: Image files of the meal saved on the device.

[1303] Specific action: A photo of breakfast is taken using the smartphone's camera app, and that image is saved to the device's local storage.

[1304] Step 2:

[1305] The device reads the captured image from local storage and uploads it to the server along with metadata such as the user's ID and timestamp.

[1306] Input: Image file of the meal saved on the device, user ID, timestamp

[1307] Output: Image data and metadata transferred to the server

[1308] Specific action: The device sends the image and metadata to the server as an HTTP POST request.

[1309] Step 3:

[1310] The server receives the uploaded image and calls an image analysis API to retrieve information about the ingredients in the image.

[1311] Input: Image data sent from the device

[1312] Output: List of ingredients returned from the image analysis API

[1313] Specific operation: The server sends images to an image analysis API such as the Google Vision API and receives the analysis results.

[1314] Step 4:

[1315] Based on the acquired food information, the server retrieves calorie information for each food item from the database.

[1316] Input: List of ingredients obtained from an image analysis API

[1317] Output: Calorie information for each food item retrieved from the database.

[1318] Specific operation: The server sends a query to an internal database to retrieve calorie information, for example, toast (150kcal), eggs (70kcal), and a vegetable salad (30kcal).

[1319] Step 5:

[1320] The server adds up the calories of each ingredient and calculates the total calories.

[1321] Input: Calorie information for each ingredient

[1322] Output: Total calorie value

[1323] Specific operation: The server adds up the calories of each ingredient and calculates a total calorie count of 250kcal as an example.

[1324] Step 6:

[1325] The server uses an emotion engine to recognize emotions from the user's facial expressions, voice, and input text.

[1326] Input: User facial image, audio data, input text

[1327] Output: Emotional state data returned from the emotion engine.

[1328] Specific operation: The server sends user data to the emotion engine and receives the emotion analysis results.

[1329] Step 7:

[1330] The server sends calorie information and emotional information to the terminal in JSON format.

[1331] Input: Total calorie value, emotional state data

[1332] Output: Data converted to JSON format

[1333] Specific operation: The server converts calorie information and emotion information into JSON and sends it to the terminal as an HTTP response.

[1334] Step 8:

[1335] The device parses the received JSON data and displays calorie information and emotional information to the user.

[1336] Input: Calorie information and emotion information in JSON format

[1337] Output: Calorie information and emotional information displayed on the screen

[1338] Specific action: The terminal parses the JSON data and displays the message, "Your breakfast this morning was 250kcal. You seem to be feeling a little tired right now."

[1339] Step 9:

[1340] The server retrieves past meal history from a database based on the user's ID and suggests the next day's menu, taking into account the user's health goals and emotional state.

[1341] Input: User ID, health goal information, emotional state

[1342] Output: Suggested menu for the next day

[1343] Specific operation: The server issues SQL queries to retrieve meal history and health goals, and uses a machine learning model to generate the next day's menu.

[1344] Step 10:

[1345] The server sends the suggested menu information for the following day to the terminal in JSON format.

[1346] Input: Suggestion menu data

[1347] Output: Suggested menu converted to JSON format

[1348] Specific operation: The server converts the suggested menu into JSON format and sends it to the terminal as an HTTP response.

[1349] Step 11:

[1350] The terminal analyzes the received menu information and displays it to the user.

[1351] Input: Suggestion menu in JSON format

[1352] Output: Suggestion menu displayed on the screen

[1353] Specific action: The terminal parses the JSON data and displays the message, "For tomorrow's menu, we recommend yogurt and fruit for breakfast, chicken salad for lunch, and vegetable soup for dinner."

[1354] Step 12:

[1355] The server calculates the difference between the user's total calorie intake for the day and their target calorie intake, and then calculates the amount of exercise required based on that difference.

[1356] Input: Total calorie intake for the day, target calorie intake

[1357] Output: Recommended exercise plan

[1358] Specific operation: The server calculates an exercise plan that burns, for example, 300 kcal, based on the difference.

[1359] Step 13:

[1360] The server checks the user's emotional state based on the emotion engine and adjusts the exercise plan accordingly.

[1361] Input: Recommended exercise plan, emotional state

[1362] Output: Adjusted exercise plan

[1363] Specific action: The server suggests a relaxing exercise plan (e.g., a 60-minute walk) when the user is tired.

[1364] Step 14:

[1365] The server sends a recommended exercise plan to the device in JSON format.

[1366] Input: Data from the adjusted exercise plan

[1367] Output: Exercise plan converted to JSON format

[1368] Specific operation: The server converts the motion plan into JSON format and sends it to the terminal as an HTTP response.

[1369] Step 15:

[1370] The device analyzes the received exercise plan and displays it to the user.

[1371] Input: Exercise plan in JSON format

[1372] Output: Exercise plan displayed on the screen

[1373] Specific action: The device parses the JSON data and displays the message, "To burn off today's calorie intake, we recommend a 30-minute jog or a 60-minute relaxing walk."

[1374] (Application Example 2)

[1375] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1376] In modern society, it is crucial for users to manage their own health and implement appropriate dietary and exercise plans. However, there are limited ways for users to access real-time calorie information and health recommendations when choosing food. Furthermore, it is difficult to provide health recommendations that take into account the user's emotional state. Therefore, there is a need for a system that enables users to make appropriate dietary choices and implement exercise plans based on their emotional state and health goals.

[1377] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[1378] In this invention, the server includes means for the user to take an image of a meal, means for uploading the image to the server, means for the server to analyze the image and obtain ingredient information and calorie information, means for transmitting the calorie information to the user's terminal, means for displaying the calorie information on the terminal, means for the server to suggest the next day's menu based on the user's meal history and health goals, means for transmitting the next day's menu information to the user's terminal, means for displaying the menu information on the terminal, means for the server to calculate the required amount of exercise and transmit an exercise plan to the user's terminal, means for displaying the exercise plan on the terminal, means for a head-mounted display worn by the user to recognize ingredients and food products and display calorie information in real time, and means for the head-mounted display to recognize the user's emotional state and adjust health suggestions accordingly. As a result, when the user selects food in a store, they can check calorie information in real time and receive health suggestions tailored to their emotional state.

[1379] A "user" refers to an individual who uses the system to take photos of their meals or manage their health.

[1380] "Food images" refer to images taken by users to record the contents of their meals.

[1381] A "server" refers to a device or computer system that processes data sent by a user, calculates ingredient information and calorie information, and transmits it to the user's terminal.

[1382] "Uploading" refers to the operation or process of sending data from a user's device to a server.

[1383] "Image analysis" refers to the process of extracting information about ingredients and calories from images of food that have been photographed.

[1384] "Ingredient information" refers to data about the types and characteristics of each ingredient included in a meal.

[1385] "Calorie information" refers to data that shows the energy content of food and ingredients.

[1386] "Device" refers to a smartphone, tablet, or other electronic device owned by the user.

[1387] "Meal history" refers to data that records the types of meals a user has eaten in the past.

[1388] "Health goals" refer to health-related achievement goals set by the user (e.g., weight loss, muscle gain, etc.).

[1389] "Menu suggestions" refer to meal menus that are suggested based on the user's health goals and eating history.

[1390] An "exercise plan" refers to an exercise plan proposed to help the user achieve their desired level of physical activity.

[1391] A "head-mounted display" refers to a display device that a user wears on their head (e.g., a head-mounted display).

[1392] "Emotional state" refers to the user's psychological or emotional state.

[1393] "Health suggestions" refer to suggestions regarding diet and exercise that are tailored based on the user's emotional state and health goals.

[1394] "Real-time" refers to operations and data processing being performed instantly.

[1395] "Recognition" refers to the process by which a system extracts and understands specific information from input data (e.g., images or audio).

[1396] This invention is a system that supports users' health management by utilizing a head-mounted display (HMD) that users use on a daily basis. This system consistently performs tasks such as capturing images of meals, calculating calories, recognizing emotional states, and providing health recommendations. The specific implementation of the system is described below.

[1397] Hardware and software to be used

[1398] 1. Head-mounted display (HMD):

[1399] It is used by users when selecting food items in a store. The HMD has a built-in camera and display, and displays calorie information in real time when the user takes a picture of the food.

[1400] A typical HMD device is used as a concrete example.

[1401] 2. Server:

[1402] The system performs image analysis and calorie calculation, manages a database of meal history, generates health recommendations, and recognizes the user's emotional state. The server utilizes image analysis APIs (e.g., general image analysis services) and the EmotionRecognition library.

[1403] 3. User terminal:

[1404] The system displays calorie information and health suggestions sent from the server on the user's smartphone, tablet, or other device.

[1405] System processing and functions

[1406] 1. Take and upload photos of your meal:

[1407] The user takes pictures of food using the HMD's camera. The captured images are temporarily stored in the HMD's local storage and then uploaded to the server.

[1408] 2. Image analysis and acquisition of calorie information:

[1409] The server sends the received image to an image analysis API to obtain ingredient and calorie information. This allows detailed nutritional data of the food photographed by the user to be stored on the server.

[1410] 3. Display of calorie information:

[1411] The acquired calorie information is displayed in real time on the user's HMD (Head-Mounted Display). This allows the user to check the calorie content of food on the spot.

[1412] 4. Recognition of emotional states:

[1413] The HMD's camera and microphone are used to capture the user's facial expressions and voice, and an emotion recognition library is used to analyze the user's emotional state. The results are also sent to the server.

[1414] 5. Generating and displaying health suggestions:

[1415] The server generates health suggestions, including menu suggestions for the next day and recommended exercise levels, based on the user's eating history, health goals, and current emotional state. The generated suggestions are sent to and displayed on the user's HMD and device.

[1416] Specific example

[1417] For example, when a user chooses a sandwich at a cafe, the HMD (Head-Mounted Display) takes a picture of the sandwich and instantly displays its calorie count. Furthermore, if the server determines that the user has been lacking exercise recently and is feeling a bit tired, it will suggest light walking or relaxing exercises.

[1418] Example of a prompt

[1419] "Create a model for an application that calculates calories based on food images taken by the user using an HMD (Head-Mounted Display) and displays health suggestions tailored to the user's emotional state. The model will use a common image analysis service to obtain calorie information from food ingredients and the EmotionRecognition library to recognize the user's emotions. Implement the generated program in Python and explain the overall flow."

[1420] This system allows users to check calorie information on food in real time and select appropriate meals and exercise based on their health and emotional state.

[1421] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1422] Step 1:

[1423] The user takes a picture of their meal using the camera on a head-mounted display (HMD). Specifically, the user points the HMD's camera at the meal and the image is saved to local storage. The input is the image of the meal, and the output is the image file saved to local storage.

[1424] Step 2:

[1425] The device uploads the saved image to the server. This operation uses a communication module to send the image file over the internet. The input is the image file in local storage, and the output is the image data sent to the server.

[1426] Step 3:

[1427] The server analyzes the received image to obtain information about the ingredients and calories. Specifically, the server calls an image analysis API to recognize the ingredients in the image and obtain the component data for each ingredient. The input is the uploaded image of the meal, and the output is the ingredient information and its corresponding calorie information.

[1428] Step 4:

[1429] The server sends the acquired calorie information to the user's terminal. The server packages the calorie information in JSON format and sends it to the terminal. The input is the calorie information and the user ID, and the output is the JSON data sent to the terminal.

[1430] Step 5:

[1431] The terminal analyzes and displays the received calorie information. The terminal parses the JSON data and displays the calorie information clearly on the screen. The input is the JSON data received from the server, and the output is the calorie information displayed on the terminal's screen.

[1432] Step 6:

[1433] The server analyzes the user's facial expressions and voice, captured by the HMD's camera and microphone, to recognize the user's emotional state. It uses an emotion recognition library to analyze the emotional state and obtain the results. The input is the user's facial expression and voice data sent from the HMD, and the output is emotional state data.

[1434] Step 7:

[1435] The server generates menu suggestions and exercise plans for the next day based on the user's meal history, health goals, and emotional state. It uses a machine learning model to create optimal suggestions considering each data point. Inputs are the user's meal history, health goals, and emotional state, while outputs are menu suggestions and exercise plans for the next day.

[1436] Step 8:

[1437] The server sends the generated menu suggestions and exercise plan to the user's device and HMD. The packaged data is sent in JSON format, and receipt is confirmed. The input is the menu suggestion and exercise plan data, and the output is the suggestion data sent to the device and HMD.

[1438] Step 9:

[1439] The terminal and HMD display the received menu suggestions and exercise plans. The user reviews the suggestions displayed on the screen and plans their next course of action. This allows the user to manage their health in real time. The input is the suggestion data sent to the terminal and HMD, and the output is the displayed menu suggestions and exercise plans.

[1440] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1441] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1442] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[1443] [Fourth Embodiment]

[1444] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[1445] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1446] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1447] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[1448] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1449] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1450] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1451] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[1452] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1453] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1454] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1455] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1456] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1457] This invention relates to a system in which a user takes a picture of their meal, which is then automatically calculated based on the image, and which also provides menu suggestions for the following days and guidance on the amount of exercise needed. This system supports the user's healthy eating habits through communication between a server and the user's terminal.

[1458] overview

[1459] Users take photos of their meals using devices such as smartphones or tablets. These photos are uploaded to a server. The server uses image analysis technology to recognize the ingredients in the images and obtain their calorie information. Furthermore, the server suggests appropriate menus and exercise plans for the following day based on the user's health goals.

[1460] Program Description

[1461] 1. Take a photo of your meal and upload the image.

[1462] The user takes a photo of their meal with their device. For example, they might take a photo of their breakfast toast, eggs, and vegetable salad.

[1463] The device reads the captured image from local storage and uploads it to the server along with metadata such as the user's ID and timestamp.

[1464] 2. Image analysis and calorie calculation

[1465] The server receives the uploaded image and calls an image analysis API (e.g., Google Vision API) to obtain information about the ingredients.

[1466] Based on the list of ingredients returned from the API (e.g., "toast", "eggs", "vegetable salad"), the calorie information for each ingredient is retrieved from the database.

[1467] Add up the calories of each ingredient to calculate the total calories. For example, if you have toast (150kcal), an egg (70kcal), and a vegetable salad (30kcal), the total calories would be 250kcal.

[1468] 3. Display of calorie information

[1469] The server sends the calculated calorie information to the terminal in JSON format.

[1470] The device parses the received JSON data and displays calorie information to the user. Example: "This morning's breakfast was 250 kcal."

[1471] 4. Menu suggestions for the following day

[1472] The server retrieves the user's meal history from the database based on their ID. Furthermore, it takes into account the user's health goals (e.g., dieting).

[1473] Using a machine learning model, the system generates a menu for the next day based on the user's past eating history and health goals. For example, it might suggest a low-calorie, high-protein meal.

[1474] The generated menu information for the following day is sent to the terminal in JSON format.

[1475] The terminal parses the received menu information and displays it to the user. Example: "For tomorrow's menu, we recommend yogurt and fruit for breakfast, chicken salad for lunch, and vegetable soup for dinner."

[1476] 5. Guidance on recommended exercise levels

[1477] The server retrieves the total calorie intake for the day from the database. For example, if the intake is 1800kcal, it calculates the difference of 300kcal between that and the target intake of 1500kcal.

[1478] Based on the difference, calculate an exercise plan to burn 300kcal. Example: 30 minutes of jogging, 60 minutes of walking.

[1479] A recommended exercise plan will be sent to your device in JSON format.

[1480] The device parses the received exercise plan and displays it to the user. Example: "To burn off today's calorie intake, we recommend 30 minutes of jogging or 60 minutes of walking."

[1481] This system allows users to obtain appropriate calorie information from their meals and, furthermore, suggests menus and exercise plans for the next day based on their health goals, enabling them to live a healthy lifestyle.

[1482] The following describes the processing flow.

[1483] Step 1:

[1484] The user takes a photo of the meal with their device. The user takes a photo of the meal using the device's camera function, launches the application, and selects the photo.

[1485] Step 2:

[1486] The device reads photos taken from local storage and generates metadata such as the user's ID and timestamp. This integrates the photos with related information.

[1487] Step 3:

[1488] The device sends the photo and metadata to the server as an HTTP POST request. The data is securely transferred to the server using Internet protocols.

[1489] Step 4:

[1490] The server receives the uploaded photos and metadata. The received data is prepared for the image analysis process.

[1491] Step 5:

[1492] The server calls an image analysis API to analyze the content of the photograph. The analysis API uses general-purpose image analysis services such as Google Vision API or Amazon Rekognition.

[1493] Step 6:

[1494] The server parses the list of ingredients received as an API response. For example, let's assume the response contains ingredient names such as "toast", "eggs", and "vegetable salad".

[1495] Step 7:

[1496] The server queries the database to retrieve the calorie value of each food item. The database stores information on various food items and their calorie content.

[1497] Step 8:

[1498] The server adds up the calories of each ingredient to calculate the total calories. For example, if toast is 150kcal, eggs are 70kcal, and a vegetable salad is 30kcal, the total calories would be 250kcal.

[1499] Step 9:

[1500] The server generates the calculation results (list of ingredients and calorie information) in JSON format and sends them back to the terminal. The JSON data includes the analysis results and detailed calorie information.

[1501] Step 10:

[1502] The terminal parses the JSON data received from the server. The parsed data is then converted into a format that can be displayed to the user within the application.

[1503] Step 11:

[1504] The device displays the calorie content of each food item and the total calorie content to the user. For example, it might display in the format: "Toast: 150kcal, Egg: 70kcal, Vegetable salad: 30kcal. Total calories: 250kcal."

[1505] Step 12:

[1506] The server retrieves the user's meal history and health goals from the database. Health goals include things like weight loss and muscle gain.

[1507] Step 13:

[1508] The server uses a machine learning model to generate a menu for the next day based on the user's past eating history and health goals. The model calculates the optimal meal for the user's health condition.

[1509] Step 14:

[1510] The server generates the next day's menu information in JSON format and sends it to the terminal. The generated menu includes recommended foods and their quantities.

[1511] Step 15:

[1512] The terminal parses the menu information it receives. The parsed menu information is then displayed to the user in a visually easy-to-understand format.

[1513] Step 16:

[1514] The device displays the user's menu for the next day. For example, it might display something like, "We recommend yogurt and fruit for breakfast, chicken salad for lunch, and vegetable soup for dinner."

[1515] Step 17:

[1516] The server retrieves the total calorie intake for the day from the database. For example, let's say the total calories from all meals for the day are 1800 kcal.

[1517] Step 18:

[1518] The server calculates the difference between the user's target calorie intake and their current intake. If the target calorie intake is 1500kcal, the difference will be 300kcal.

[1519] Step 19:

[1520] The server calculates an exercise plan to burn 300kcal. For example, 30 minutes of jogging or 60 minutes of walking might be considered the amount of exercise that burns 300kcal.

[1521] Step 20:

[1522] The server generates a recommended exercise plan in JSON format and sends it to the device. The generated exercise plan includes specific exercise types and durations.

[1523] Step 21:

[1524] The device parses the received exercise plan information. The parsed information is converted into a format that can be displayed to the user.

[1525] Step 22:

[1526] The device displays an exercise plan recommended to the user. For example, it might display something like, "To burn off today's calorie intake, we recommend a 30-minute jog or a 60-minute walk."

[1527] (Example 1)

[1528] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1529] In today's lifestyle, many people struggle to maintain a healthy diet. In particular, excessive calorie intake and insufficient exercise lead to obesity and other health problems. Traditional health management systems often require users to manually input calorie calculations and create exercise plans. Against this backdrop, there is a need for a system that allows users to easily record their meals and centrally manage calorie information, optimal menu suggestions, and exercise plans.

[1530] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1531] In this invention, the server includes means for the user to take an image of a meal, means for uploading the image to the server, means for the server to analyze the image and obtain ingredient information and calorie information, means for transmitting the calorie information to the user's terminal, means for displaying the calorie information on the terminal, means for the server to suggest the next day's menu based on the user's meal history and health goals, means for transmitting the next day's menu information to the user's terminal, means for displaying the menu information on the terminal, means for the server to calculate the required amount of exercise and transmit an exercise plan to the user's terminal, and means for displaying the exercise plan on the terminal. As a result, the user can easily upload an image to automatically obtain ingredient information and calorie information and receive menu suggestions and exercise plans tailored to their health goals.

[1532] 1. A "user" refers to an individual who uses the system to take pictures of their meals and receives calorie information, menu suggestions, and exercise plans.

[1533] 2. "Terminal" refers to a device such as a smartphone or tablet used by the user, which is used to take pictures of food or to receive and display information from a server.

[1534] 3. A "server" refers to a computer system that analyzes images sent by users, obtains information on ingredients and calories, and generates menu suggestions and exercise plans to send to the user's terminal.

[1535] 4. "Image analysis" is the process that a server performs to obtain information about ingredients from images of food, and is generally carried out using an image analysis API.

[1536] 5. "Calorie information" refers to the amount of energy calculated by the server based on ingredient information obtained from the image of the meal.

[1537] 6. "Menu information" refers to the meal plan for the following day that the server generates based on the user's meal history and health goals.

[1538] 7. "Exercise plan" refers to the type and amount of exercise suggested by the server based on the user's calorie intake.

[1539] 8. "Image analysis API" refers to an application programming interface used by a server to analyze images of food and obtain information about the ingredients.

[1540] 9. "JSON format" refers to the JavaScript Object Notation format used for lightweight and easy data exchange.

[1541] 10. A "machine learning model" refers to the algorithms and analytical models that the server uses to generate the next day's menu based on the user's past eating history and health goals.

[1542] 11. "Meal history" refers to data that records information about meals a user has eaten in the past.

[1543] 12. "Health goals" refer to health-related goals that users wish to achieve, such as weight loss or muscle building.

[1544] This invention relates to a system in which a user takes a picture of their meal, which is then automatically calculated based on the image, and which also provides menu suggestions for the following days and guidance on the amount of exercise needed. This system supports the user's healthy eating habits through communication between a server and the user's terminal.

[1545] overview

[1546] Taking photos of food and uploading the images

[1547] The user takes a photo of their meal using a device such as a smartphone or tablet. For example, they might take a photo of their breakfast toast, eggs, and vegetable salad.

[1548] The device reads this photo from local storage and uploads it to the server along with metadata such as the user's ID and timestamp.

[1549] Image analysis and calorie calculation

[1550] The server receives the uploaded image and calls an image analysis API (e.g., Google Vision API) to obtain information about the ingredients.

[1551] Based on the retrieved list of ingredients (e.g., "toast", "eggs", "vegetable salad"), the server retrieves calorie information for each ingredient from the database.

[1552] The server adds up the calories of each ingredient to calculate the total calorie count. For example, if you have toast (150kcal), an egg (70kcal), and a vegetable salad (30kcal), the total calorie count will be 250kcal.

[1553] Display of calorie information

[1554] The server sends the calculated calorie information to the terminal in JSON format.

[1555] The device parses the received JSON data and displays calorie information to the user. For example, it might display, "This morning's breakfast was 250 kcal."

[1556] Menu suggestions for the next day

[1557] The server retrieves the user's meal history and health goals from the database based on their ID. Furthermore, it takes into account the user's health goals (e.g., diet).

[1558] The server uses a machine learning model to generate a menu for the next day based on the user's past eating history and health goals. For example, it might suggest a low-calorie, high-protein meal.

[1559] The generated menu information for the following day is sent to the terminal in JSON format.

[1560] The terminal parses the received menu information and displays it to the user. For example, it might display, "For tomorrow's menu, we recommend yogurt and fruit for breakfast, chicken salad for lunch, and vegetable soup for dinner."

[1561] Guide to recommended exercise levels

[1562] The server retrieves the total calorie intake for the day from the database. For example, if the intake is 1800kcal, it calculates the difference of 300kcal between that and the target intake of 1500kcal.

[1563] The server calculates an exercise plan to burn 300kcal based on the difference. An example might include 30 minutes of jogging and 60 minutes of walking.

[1564] A recommended exercise plan will be sent to your device in JSON format.

[1565] The device parses the received exercise plan and displays it to the user. For example, it might display, "To burn off today's calorie intake, we recommend 30 minutes of jogging or 60 minutes of walking."

[1566] Specific example

[1567] For example, if a user takes a photo of "pancakes, bacon, and fruit" for breakfast, the process would proceed as follows:

[1568] The device uploads this photo to the server.

[1569] The server uses the Google Vision API to recognize these ingredients and retrieves their respective calorie counts from a database.

[1570] The total calories from the pancakes (200kcal), bacon (100kcal), and fruit (80kcal) amount to 380kcal. This is sent to the terminal in JSON format, and the user is shown "This morning's breakfast is 380kcal."

[1571] Furthermore, the server takes the user's past data and health goals into consideration and suggests "oatmeal, yogurt, and fruit" as breakfast for the next day. This is also sent to the terminal in JSON format and displays the message, "We recommend oatmeal, yogurt, and fruit for breakfast tomorrow."

[1572] Finally, the server calculates the calories to be burned and suggests "20 minutes of jogging or 50 minutes of walking," sending this information to the terminal. The terminal then displays a message to the user saying, "To burn off today's calorie intake, we recommend 20 minutes of jogging or 50 minutes of walking."

[1573] Example of a prompt

[1574] "A user has uploaded a photo of their breakfast (pancakes, bacon, fruit). Calculate the calories in these ingredients and display the total calories to the user. Also, suggest a healthy breakfast menu for the next day and display an optimal exercise plan to burn off the user's calories."

[1575] This system allows users to obtain appropriate calorie information from their meals and, furthermore, suggests menus and exercise plans for the next day based on their health goals, enabling them to live a healthy lifestyle.

[1576] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1577] Step 1:

[1578] Taking photos of food and uploading the images

[1579] The user takes a picture of their meal using a device such as a smartphone or tablet. For example, they might take a picture of toast, eggs, and a vegetable salad for breakfast. The captured image is saved to the device's local storage. The device then uploads this photo to the server along with metadata such as the user's ID and a timestamp. The input includes the image of the meal, the user's ID, and the timestamp, while the output is the image data sent to the server.

[1580] Step 2:

[1581] Image analysis and acquisition of food ingredient information

[1582] The server receives the uploaded image and saves it to the specified directory. Next, the server calls an image analysis API (e.g., an image analysis API) to obtain information about the ingredients in the image. Specifically, it sends the image data as input to the image analysis API and receives a list of identified ingredients (e.g., "toast", "egg", "vegetable salad") as output. In this step, the server passes the image data to the API and analyzes the data returned from the API.

[1583] Step 3:

[1584] Calorie calculation

[1585] The server retrieves calorie information for each ingredient from the database based on the retrieved ingredient list. Taking the ingredient list (e.g., "toast", "egg", "vegetable salad") as input, it retrieves the corresponding calorie values ​​from the database (e.g., toast is 150kcal, egg is 70kcal, vegetable salad is 30kcal) as output. The server then sums these calories to calculate the total calorie count (e.g., 250kcal). In this case, the input is the ingredient list, the calorie information is from the database, and the output is the total calorie count.

[1586] Step 4:

[1587] Display of calorie information

[1588] The server sends the calculated calorie information to the terminal in JSON format. It takes calorie information (e.g., {"Breakfast Calories": 250}) as input, converts it to JSON, and sends it to the terminal. The terminal parses the received JSON data and displays "This morning's breakfast was 250kcal." to the user. Here, the input is calorie information in JSON format, and the output is the display of the parsed calorie information.

[1589] Step 5:

[1590] Menu suggestions for the next day

[1591] The server retrieves the user's past meal history and health goals from a database based on their ID. It then inputs the meal history and health goals from the database and uses the output data to generate a menu for the next day based on a machine learning model. For example, it might suggest yogurt and fruit for breakfast, chicken salad for lunch, and vegetable soup for dinner. The generated menu information is then converted to JSON format and sent to the terminal. In this case, the input is the meal history and health goals, and the output is the generated menu information in JSON format.

[1592] Step 6:

[1593] Displaying menu information for the following day

[1594] The terminal parses the received menu information and displays to the user, "For tomorrow's menu, we recommend yogurt and fruit for breakfast, chicken salad for lunch, and vegetable soup for dinner." Here, the input is menu information in JSON format, and the output is a display of the parsed menu information.

[1595] Step 7:

[1596] Calculation of Recommended Exercise Amount

[1597] The server retrieves the total calorie intake for the day from the database. For example, if a user has consumed 1800 kcal and their target intake is 1500 kcal, the server calculates the difference of 300 kcal. Based on this difference, it generates an exercise plan to burn 300 kcal (e.g., 30 minutes of jogging, 60 minutes of walking). In this step, the input is the calorie intake information retrieved from the database, and the output is the generated exercise plan.

[1598] Step 8:

[1599] Displaying exercise plan

[1600] The server sends the exercise plan to the terminal in JSON format. The server takes the exercise plan (e.g., {"exercise plan": "30 minutes of jogging or 60 minutes of walking"}) as input, converts it to JSON, and sends it to the terminal. The terminal parses the received exercise plan and displays to the user, "To burn off today's calorie intake, we recommend 30 minutes of jogging or 60 minutes of walking." Here again, the input is the exercise plan in JSON format, and the output is a display of the parsed exercise plan.

[1601] (Application Example 1)

[1602] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1603] In modern society, there is a growing concern for health, and it is essential to understand the calorie content of meals and maintain a healthy diet. However, calculating calories and creating appropriate meal and exercise plans in daily life is time-consuming and difficult to maintain. In particular, users of food delivery services have a need to easily find out the calorie information of the meals they order, but current technology lacks a system that can provide this information automatically and accurately.

[1604] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1605] In this invention, the server includes means for the user to take an image of a meal, means for uploading the image to the server, means for the server to analyze the image and obtain ingredient information and calorie information, means for transmitting the calorie information to the user's terminal, means for displaying the calorie information on the terminal, means for the server to suggest the next day's menu based on the user's meal history and health goals, means for transmitting the next day's menu information to the user's terminal, means for displaying the menu information on the terminal, means for the server to calculate the required amount of exercise and transmit an exercise plan to the user's terminal, means for providing calorie information of the order and guiding the user to suggest the next meal and exercise plan when using the food delivery service, and means for displaying the exercise plan on the terminal. As a result, when a user uses the food delivery service, they can easily obtain calorie information of their order and check healthy meal suggestions and necessary exercise plans for the next meal.

[1606] "User" refers to any person who uses the system of the present invention.

[1607] "Food images" refers to photographic data of meals taken by users.

[1608] A "server" refers to a central computer system that performs image analysis, data processing, and the transmission and reception of various types of information.

[1609] "Device" refers to a device used by a user (e.g., smartphone, tablet).

[1610] "Method for uploading images" refers to the function that sends image data of a meal from the device to the server.

[1611] "Means for analyzing images" refers to a function that processes and extracts food ingredient information from uploaded images.

[1612] "Ingredient information" refers to the specific types and contents of ingredients obtained from the image analysis results.

[1613] "Calorie information" refers to data on the amount of energy calculated based on information about the ingredients.

[1614] "Meal history" refers to a record of meals a user has eaten in the past.

[1615] "Health goals" refer to objectives or targets set by the user for maintaining or improving their health (e.g., weight loss, muscle building).

[1616] "Next day's menu information" refers to the meal plan for the following day, suggested based on the user's health goals.

[1617] "Required exercise" refers to the amount of energy that should be expended, calculated based on the daily calorie intake.

[1618] An "exercise plan" refers to the specific exercises recommended to burn the necessary amount of energy.

[1619] "Food delivery service" refers to any service where users order food online and have it delivered.

[1620] "Order details" refers to the specific details of the meals and food items selected and confirmed by the user through the food delivery service.

[1621] "Healthy next meal suggestion" refers to the content of the next meal recommended based on the user's health goals and eating history.

[1622] This invention relates to a system that allows users to take pictures of their meals, calculate calories based on those images, and then provide suggestions for their next meal and necessary exercise plans. Users take pictures of their meals using a device such as a smartphone or tablet and upload those images to a server.

[1623] The server analyzes the received image and retrieves information about the ingredients within it. This analysis uses an image analysis API (such as the Google Vision API). After obtaining a list of ingredients from the analysis results, the server searches the database for the calorie information of each ingredient and calculates the total calories. The server sends this calculation result to the terminal in JSON format, and the terminal displays the calorie information to the user.

[1624] Furthermore, the server suggests the next day's menu based on the user's meal history and health goals. A machine learning model is used for this menu suggestion. The menu information is sent in JSON format, and the device displays the next meal suggestion to the user. The server also calculates the amount of exercise needed based on the total calorie intake for the day and sends a specific exercise plan to the device. The device also displays the recommended exercise plan to the user.

[1625] A key feature of this system is its ability to provide calorie information for food delivery orders and offer suggestions for future meals and exercise plans. When a user orders food delivery, the order details are sent to the server upon confirmation. The server retrieves calorie information based on the order and calculates and presents suggestions for healthy meals and exercise plans for the next order.

[1626] As a concrete example, suppose a user eats toast, eggs, and a vegetable salad for breakfast. The user takes pictures of these meals with their device and uploads them to the server. The server uses an image analysis API to obtain information about the ingredients, adds up the calories of each, and sends the total to the user's device. For example, if the toast is 150kcal, the eggs are 70kcal, and the vegetable salad is 30kcal, the total is 250kcal. This information is then displayed on the device.

[1627] Next, when a user orders lunch using a food delivery service, the details (e.g., chicken salad) are sent to the server. The server analyzes the details, retrieves calorie information, displays it to the user, and also provides suggestions for the next meal (e.g., yogurt and fruit) and a necessary exercise plan (e.g., a 30-minute jog).

[1628] Example of a prompt:

[1629] User ID: 12345

[1630] Image of the meal: path_to_image.jpg

[1631] Please retrieve calorie information and suggest the next meal and exercise plan.

[1632] This system makes it easy for users to easily understand calorie information based on their meals and manage their health appropriately. Furthermore, even for meals ordered through food delivery services, users can check calorie information, healthy meal suggestions for the next meal, and exercise plans, providing support for maintaining a healthy lifestyle.

[1633] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1634] Step 1:

[1635] The user takes a picture of their meal. They use a smartphone or tablet to take a photo of the meal and save this image data to the device. The input is the image of the meal, and the output is the image data saved on the device.

[1636] Step 2:

[1637] Images stored on the device are uploaded to the server. The user operates the application to select an image and sends it to the server along with metadata such as the user ID and timestamp. The input is the image data and metadata stored on the device, and the output is the image data sent to the server.

[1638] Step 3:

[1639] The server analyzes the uploaded image. It then calls an image analysis API (e.g., Google Vision API) to extract information about the ingredients. The input is the image data sent to the server, and the output is the list of ingredients returned by the image analysis API.

[1640] Step 4:

[1641] The server retrieves calorie information based on the food ingredients. It searches the database for the calorie information of each ingredient and calculates the total calorie count. The input is a list of ingredients obtained from an image analysis API, and the output is the calorie information of each ingredient and the total calorie count.

[1642] Step 5:

[1643] The server sends the calculated calorie information to the terminal. It generates calorie information in JSON format and sends it to the user's terminal. The input is the calorie information calculated on the server side, and the output is the JSON data sent to the terminal.

[1644] Step 6:

[1645] This application displays calorie information received by the device. It parses the received JSON data and displays the calorie information in an easy-to-understand format for the user. The input is the received JSON data, and the output is the calorie information displayed on the device screen.

[1646] Step 7:

[1647] The server suggests the next day's menu based on the user's meal history and health goals. It retrieves the user's meal history and health goal data from a database and uses a machine learning model to generate the next meal suggestion. The input is the user's meal history and health goals, and the output is the suggested menu information for the following day.

[1648] Step 8:

[1649] The server sends the next day's menu information to the user's device. It generates the menu information in JSON format and sends it to the user's device. The input is the next day's menu information generated by a machine learning model, and the output is the JSON data sent to the device.

[1650] Step 9:

[1651] This application displays menu information received by the device. It parses the received JSON data and displays menu suggestions for the following day to the user. The input is the received JSON data, and the output is the menu information displayed on the device screen.

[1652] Step 10:

[1653] The server calculates the required amount of exercise and sends an exercise plan to the user's device. It calculates the required amount of exercise based on the total calorie intake for the day and generates a specific exercise plan. The calculated exercise plan is sent to the user's device in JSON format. The input is the total calorie intake for the day, and the output is the exercise plan information sent to the device.

[1654] Step 11:

[1655] This program displays the exercise plan received by the device. It parses the received JSON data and displays the recommended exercise plan for the user. The input is the received JSON data, and the output is the exercise plan information displayed on the device screen.

[1656] Step 12:

[1657] This system sends food delivery service order details to a server. When a user places an order using a food delivery service, the order details are sent to the server. The input is the food delivery service order details, and the output is the order data sent to the server.

[1658] Step 13:

[1659] The server retrieves calorie information based on the order and provides suggestions for the next meal and exercise plan. It analyzes the order to obtain calorie information and generates suggestions for the next meal and exercise plan. The input is the submitted order, and the output is calorie information, suggestions for the next meal, and an exercise plan.

[1660] Step 14:

[1661] Calorie information, meal suggestions for the next meal, and exercise plans are sent to the user's device, which then displays them. The input is the calorie information, meal suggestions for the next meal, and exercise plan sent from the server, and the output is this information displayed on the device's screen.

[1662] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1663] This invention relates to a system in which a user takes a picture of their meal, which is then automatically calculated based on the image, and which also provides menu suggestions for the following days and guidance on the amount of exercise needed. This system supports the user's healthy eating habits through communication between the server and the user's terminal. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it can provide more personalized support.

[1664] overview

[1665] The user takes a photo of their meal using a device such as a smartphone or tablet. The device uploads this photo to a server. The server uses image analysis technology to recognize the ingredients in the image and obtain their respective calorie information. Furthermore, the server suggests an appropriate menu and exercise plan for the next day based on the user's health goals. In this invention, an emotion engine is used to recognize the user's emotions and adjust the meal and exercise plan based on the user's emotions and psychological state.

[1666] Program Description

[1667] 1. Take a photo of your meal and upload the image.

[1668] The user takes a photo of their meal with their device. For example, they might take a photo of their breakfast toast, eggs, and vegetable salad.

[1669] The device reads the captured image from local storage and uploads it to the server along with metadata such as the user's ID and timestamp.

[1670] 2. Image analysis and calorie calculation

[1671] The server receives the uploaded image and calls an image analysis API (e.g., Google Vision API) to obtain information about the ingredients.

[1672] Based on the list of ingredients returned from the API (e.g., "toast", "eggs", "vegetable salad"), the calorie information for each ingredient is retrieved from the database.

[1673] Add up the calories of each ingredient to calculate the total calories. For example, if you have toast (150kcal), an egg (70kcal), and a vegetable salad (30kcal), the total calories would be 250kcal.

[1674] 3. Display of calorie information and user sentiment.

[1675] Along with the calculated calorie information, the server uses an emotion engine to recognize the user's emotions from their facial expressions, voice, and entered text.

[1676] The results will be sent to the terminal in JSON format.

[1677] The device parses the received JSON data and displays calorie information and perceived emotion information to the user. Example: "Your breakfast this morning was 250kcal. You seem to be feeling a little tired right now."

[1678] 4. Menu suggestions for the following day

[1679] The server retrieves the user's meal history from the database based on their ID. Furthermore, it takes into account the user's health goals (e.g., dieting).

[1680] Using a machine learning model, the system generates a menu for the next day based on the user's past eating history and health goals. The model also reflects the user's emotional state.

[1681] The generated menu information for the following day is sent to the terminal in JSON format.

[1682] The terminal parses the received menu information and displays it to the user. Example: "For tomorrow's menu, we recommend yogurt and fruit for breakfast, chicken salad for lunch, and vegetable soup for dinner."

[1683] 5. Guidance on recommended exercise levels

[1684] The server retrieves the total calorie intake for the day from the database. For example, let's say the total calories from all meals for the day are 1800 kcal.

[1685] The server calculates the difference between the user's target calorie intake and their current intake. If the target calorie intake is 1500kcal, the difference will be 300kcal.

[1686] Based on the difference, calculate an exercise plan to burn 300kcal. For example, assume that 30 minutes of jogging and 60 minutes of walking will burn 300kcal.

[1687] The exercise plan is adjusted based on the user's emotional state, as determined by the emotion engine. For example, if the user is "tired," it will suggest light, relaxing exercises.

[1688] A recommended exercise plan will be sent to your device in JSON format.

[1689] The device parses the received exercise plan and displays it to the user. Example: "To burn off today's calorie intake, we recommend a 30-minute jog or a 60-minute relaxing walk."

[1690] This system allows users to obtain appropriate calorie information from their diet and also suggests menus and exercise plans for the next day based on their health goals and emotional state, enabling them to live a healthier and less mentally stressful life.

[1691] The following describes the processing flow.

[1692] Step 1:

[1693] The user takes a picture of the meal with their device. The user takes a picture of the meal using the device's camera function, launches the application, and selects the picture.

[1694] Step 2:

[1695] The device reads photos taken from local storage and generates metadata such as the user's ID and timestamp. This integrates the photos with related information.

[1696] Step 3:

[1697] The device sends the photo and metadata to the server as an HTTP POST request. The data is securely transferred to the server using Internet protocols.

[1698] Step 4:

[1699] The server receives the uploaded photos and metadata. The received data is prepared for the image analysis process.

[1700] Step 5:

[1701] The server calls an image analysis API to analyze the content of the photograph. General-purpose image analysis services such as Google Vision API and Amazon Rekognition are used for the analysis API.

[1702] Step 6:

[1703] The server parses the list of ingredients received as an API response. For example, let's assume the response contains ingredient names such as "toast", "eggs", and "vegetable salad".

[1704] Step 7:

[1705] The server queries the database to retrieve the calorie value of each food item. The database stores information on various food items and their calorie content.

[1706] Step 8:

[1707] The server adds up the calories of each ingredient to calculate the total calories. For example, if toast is 150kcal, eggs are 70kcal, and a vegetable salad is 30kcal, the total calories would be 250kcal.

[1708] Step 9:

[1709] To recognize the user's emotions, the device uses its camera and microphone to capture the user's facial expressions and voice. Alternatively, the user fills out a questionnaire about their emotions.

[1710] Step 10:

[1711] The device uses an emotion engine to analyze the user's emotions from the acquired data. For example, it can determine if the user is feeling tired or stressed.

[1712] Step 11:

[1713] The server sends calorie information and recognized emotion information to the terminal in JSON format. The JSON data includes the analysis results, detailed calorie information, and recognized emotion.

[1714] Step 12:

[1715] The terminal parses the JSON data received from the server. The parsed data is then converted into a format that can be displayed to the user within the application.

[1716] Step 13:

[1717] The device displays the calories for each food item, the total calories, and the user's mood information. For example, it might display in the format: "Toast: 150kcal, Egg: 70kcal, Vegetable salad: 30kcal. Total calories: 250kcal. You seem to be feeling a little tired right now."

[1718] Step 14:

[1719] The server retrieves the user's meal history from the database based on their ID. Furthermore, it takes into account the user's health goals (e.g., dieting).

[1720] Step 15:

[1721] The server uses a machine learning model to generate a menu for the next day based on the user's past eating history and health goals. The model also reflects the user's emotional state.

[1722] Step 16:

[1723] The server generates the next day's menu information in JSON format and sends it to the terminal. The generated menu includes recommended foods and their quantities.

[1724] Step 17:

[1725] The terminal parses the menu information it receives. The parsed menu information is then displayed to the user in a visually easy-to-understand format.

[1726] Step 18:

[1727] The device displays the user's menu for the next day. For example, it might display something like, "For tomorrow's menu, we recommend yogurt and fruit for breakfast, chicken salad for lunch, and vegetable soup for dinner."

[1728] Step 19:

[1729] The server retrieves the total calorie intake for the day from the database. For example, let's say the total calories from all meals for the day are 1800 kcal.

[1730] Step 20:

[1731] The server calculates the difference between the user's target calorie intake and their current intake. If the target calorie intake is 1500kcal, the difference will be 300kcal.

[1732] Step 21:

[1733] The server calculates an exercise plan to burn 300kcal. For example, 30 minutes of jogging and 60 minutes of walking are assumed to burn 300kcal.

[1734] Step 22:

[1735] The server adjusts the exercise plan based on the user's emotional state, as determined by the emotion engine. For example, if the user is "tired," it suggests light, relaxing exercise.

[1736] Step 23:

[1737] The server generates a recommended exercise plan in JSON format and sends it to the device. The generated exercise plan includes specific exercise types and durations.

[1738] Step 24:

[1739] The device parses the received exercise plan information. The parsed information is converted into a format that can be displayed to the user.

[1740] Step 25:

[1741] The device displays an exercise plan recommended to the user. For example, it might say, "To burn off today's calorie intake, we recommend a 30-minute jog or a 60-minute relaxing walk."

[1742] (Example 2)

[1743] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1744] In recent years, the importance of users accurately understanding their diet and calorie expenditure, and implementing appropriate meal and exercise plans, has increased in order to maintain and improve a healthy lifestyle. However, conventional methods require manual recording and calculation, which is time-consuming and laborious, and they have the problem of not being able to take into account the user's emotions and psychological state. Furthermore, calorie calculation and exercise plan suggestions are unified, and do not provide optimal support for the user. This invention aims to solve these problems and provide a health management system that is more convenient and effective for users.

[1745] The identification processing by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for the user to take an image of a meal, means for uploading the image to the server, means for the server to analyze the image and obtain ingredient information and calorie information, means for transmitting the calorie information to the user's terminal, means for displaying the calorie information on the terminal, means for the server to suggest the next day's menu based on the user's meal history and health goals, means for recognizing the user's emotional state and adjusting the meal and exercise plan, means for transmitting the next day's menu information to the user's terminal, means for displaying the menu information on the terminal, means for the server to calculate the required amount of exercise based on the day's calorie intake and health goals and transmit the exercise plan to the user's terminal, and means for displaying the exercise plan on the terminal. As a result, the user can not only receive calorie calculations for the contents of their meals and menu suggestions for the next day, but also receive suggestions for exercise plans that take their emotional state into consideration.

[1746] A "user" is an individual who uses this system to manage their own health.

[1747] A "device" refers to an electronic device operated by a user, such as a smartphone or tablet.

[1748] A "server" is a computer that analyzes data sent from a terminal and provides the necessary information.

[1749] An "image analysis API" is an application program interface for recognizing food information within an image.

[1750] "Calorie information" refers to data that shows the energy intake of each food item.

[1751] "User's meal history" refers to a record of the meals a user has eaten in the past.

[1752] "Health goals" refer to health-related objectives set by the user, such as weight loss or muscle building.

[1753] "Tomorrow's menu" refers to the meal plan for the following day that is suggested to the user.

[1754] "Emotional state" refers to information that indicates the user's psychological or emotional condition.

[1755] An "exercise plan" is a plan of exercises that a user uses to burn off the calories they have consumed.

[1756] "Metadata" refers to supplementary information added to data, such as user IDs and timestamps.

[1757] A "machine learning model" is an algorithm that analyzes user data and performs predictions and classifications.

[1758] "JSON format" refers to a lightweight text format used for exchanging data.

[1759] This invention relates to a system that allows users to take pictures of their meals, automatically calculates calories based on those images, and then suggests menus for the following days and provides guidance on the amount of exercise needed. Specific proper nouns, names of people, and company names are converted to common names, and nouns are replaced with broader terms for explanation.

[1760] First, the user takes a picture of their meal using a device such as a smartphone or tablet. For example, consider a case where the user takes a picture of toast, eggs, and a vegetable salad for breakfast. The device saves this photo to local storage, adds metadata such as the user's ID and timestamp, and uploads it to the server.

[1761] The server receives the uploaded image and uses an image analysis API such as the Google Vision API to obtain information about the ingredients in the image. Based on the list of ingredients returned from the API (e.g., "toast", "egg", "vegetable salad"), it retrieves the calorie information for each ingredient from its internal database and sums them up. For example, if toast is 150kcal, egg is 70kcal, and vegetable salad is 30kcal, the total calorie count would be 250kcal.

[1762] Next, the server uses an emotion engine to recognize the user's emotions from their facial expressions, voice, and entered text. This calorie and emotion information is sent to the device in JSON format, and the device displays it. For example, a message such as "Your breakfast this morning was 250kcal. You seem to be feeling a little tired right now." might be displayed.

[1763] Furthermore, the server retrieves the user's meal history from a database based on their ID and suggests the next day's menu, taking into account the user's health goals and emotional state. The menu information, generated using a machine learning model, is sent to the device in JSON format, which the device then displays to the user. For example, a suggestion might be, "For tomorrow's menu, we recommend yogurt and fruit for breakfast, chicken salad for lunch, and vegetable soup for dinner."

[1764] The server also calculates the difference between the user's total calorie intake for the day and their target calorie intake, and then calculates a recommended amount of exercise based on this difference. The emotion engine checks the user's emotional state and adjusts the exercise plan accordingly. This is sent to the device in JSON format, and the device displays a message such as, "To burn off today's calorie intake, we recommend a 30-minute jog or a 60-minute relaxing walk."

[1765] This system can automatically obtain appropriate calorie information from the user's diet and further supports a healthier and less mentally stressful lifestyle by suggesting menus and exercise plans for the next day based on health goals and emotional state.

[1766] Example: Example of a prompt message to input to a generative AI model

[1767] Example 1: Taking photos of food and uploading the images

[1768] Prompt: "Please upload a photo of toast, eggs, and a vegetable salad taken with your smartphone to the server."

[1769] Example 2: Image analysis and calorie calculation

[1770] Prompt: "Retrieve ingredient information from the uploaded breakfast image and calculate the calories of each ingredient."

[1771] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1772] Step 1:

[1773] The user takes a photo of their meal with their device. For example, the process of taking a photo of toast, eggs, and a vegetable salad for breakfast is described.

[1774] Input: The user launches the camera app and takes a picture of the food.

[1775] Output: Image files of the meal saved on the device.

[1776] Specific action: A photo of breakfast is taken using the smartphone's camera app, and that image is saved to the device's local storage.

[1777] Step 2:

[1778] The device reads the captured image from local storage and uploads it to the server along with metadata such as the user's ID and timestamp.

[1779] Input: Image file of the meal saved on the device, user ID, timestamp

[1780] Output: Image data and metadata transferred to the server

[1781] Specific action: The device sends the image and metadata to the server as an HTTP POST request.

[1782] Step 3:

[1783] The server receives the uploaded image and calls an image analysis API to retrieve information about the ingredients in the image.

[1784] Input: Image data sent from the device

[1785] Output: List of ingredients returned from the image analysis API

[1786] Specific operation: The server sends images to an image analysis API such as the Google Vision API and receives the analysis results.

[1787] Step 4:

[1788] Based on the acquired food information, the server retrieves calorie information for each food item from the database.

[1789] Input: List of ingredients obtained from an image analysis API

[1790] Output: Calorie information for each food item retrieved from the database.

[1791] Specific operation: The server sends a query to an internal database to retrieve calorie information, for example, toast (150kcal), eggs (70kcal), and a vegetable salad (30kcal).

[1792] Step 5:

[1793] The server adds up the calories of each ingredient and calculates the total calories.

[1794] Input: Calorie information for each ingredient

[1795] Output: Total calorie value

[1796] Specific operation: The server adds up the calories of each ingredient and calculates a total calorie count of 250kcal as an example.

[1797] Step 6:

[1798] The server uses an emotion engine to recognize emotions from the user's facial expressions, voice, and input text.

[1799] Input: User facial image, audio data, input text

[1800] Output: Emotional state data returned from the emotion engine.

[1801] Specific operation: The server sends user data to the emotion engine and receives the emotion analysis results.

[1802] Step 7:

[1803] The server sends calorie information and emotional information to the terminal in JSON format.

[1804] Input: Total calorie value, emotional state data

[1805] Output: Data converted to JSON format

[1806] Specific operation: The server converts calorie information and emotion information into JSON and sends it to the terminal as an HTTP response.

[1807] Step 8:

[1808] The device parses the received JSON data and displays calorie information and emotional information to the user.

[1809] Input: Calorie information and emotion information in JSON format

[1810] Output: Calorie information and emotional information displayed on the screen

[1811] Specific action: The terminal parses the JSON data and displays the message, "Your breakfast this morning was 250kcal. You seem to be feeling a little tired right now."

[1812] Step 9:

[1813] The server retrieves past meal history from a database based on the user's ID and suggests the next day's menu, taking into account the user's health goals and emotional state.

[1814] Input: User ID, health goal information, emotional state

[1815] Output: Suggested menu for the next day

[1816] Specific operation: The server issues SQL queries to retrieve meal history and health goals, and uses a machine learning model to generate the next day's menu.

[1817] Step 10:

[1818] The server sends the suggested menu information for the following day to the terminal in JSON format.

[1819] Input: Suggestion menu data

[1820] Output: Suggested menu converted to JSON format

[1821] Specific operation: The server converts the suggested menu into JSON format and sends it to the terminal as an HTTP response.

[1822] Step 11:

[1823] The terminal analyzes the received menu information and displays it to the user.

[1824] Input: Suggestion menu in JSON format

[1825] Output: Suggestion menu displayed on the screen

[1826] Specific action: The terminal parses the JSON data and displays the message, "For tomorrow's menu, we recommend yogurt and fruit for breakfast, chicken salad for lunch, and vegetable soup for dinner."

[1827] Step 12:

[1828] The server calculates the difference between the user's total calorie intake for the day and their target calorie intake, and then calculates the amount of exercise required based on that difference.

[1829] Input: Total calorie intake for the day, target calorie intake

[1830] Output: Recommended exercise plan

[1831] Specific operation: The server calculates an exercise plan that burns, for example, 300 kcal, based on the difference.

[1832] Step 13:

[1833] The server checks the user's emotional state based on the emotion engine and adjusts the exercise plan accordingly.

[1834] Input: Recommended exercise plan, emotional state

[1835] Output: Adjusted exercise plan

[1836] Specific action: The server suggests a relaxing exercise plan (e.g., a 60-minute walk) when the user is tired.

[1837] Step 14:

[1838] The server sends a recommended exercise plan to the devi...

Claims

1. The means by which users take pictures of their food, A means for uploading the aforementioned image to a server, A means for analyzing the aforementioned image on a server and obtaining ingredient information and calorie information, Means for transmitting the aforementioned calorie information to the user's terminal, The terminal includes means for displaying the calorie information, The server provides means for suggesting the next day's menu based on the user's meal history and health goals, A means for sending the menu information for the following day to the user's terminal, The terminal includes means for displaying the menu information, The server calculates the required amount of exercise and transmits the exercise plan to the user's terminal. Means for displaying the exercise plan in the terminal A system that includes this.

2. The system according to claim 1, further comprising means for optimizing the calorie information and the next day's menu based on the user's health goals.

3. The system according to claim 1, further comprising means for using an image analysis API to analyze the aforementioned image.

Images