System

Abstract

A system is provided.SOLUTION: A system comprising: means for inputting information such as a user's age, gender, height, weight, health condition, etc.; means for determining calories and nutrients in a meal using an image analysis engine; means for estimating a user's energy expenditure; means for generating a meal suggestion based on the user's caloric and nutritional needs using a generative AI model; and means for notifying the user of the generated meal suggestion.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In today's busy daily lives, it is difficult to manage one's diet and maintain a balanced nutritional intake. Therefore, there is a need for a system that can provide customized meal recommendations tailored to each individual's lifestyle and health condition. However, current dietary management applications often do not fully address the individual needs of users and do not provide effective meal recommendations. Furthermore, these applications often lack the ability to accurately analyze calorie and nutrient intake, resulting in a lack of accuracy in supporting users' health management. [Means for solving the problem]

[0005] The present invention solves the above-mentioned problems by providing a system that includes a means for inputting information such as a user's age, gender, height, weight, and health status; a means for uploading image data of the user's meals; a means for identifying the calories and nutrients of the meals using an image analysis engine; a means for estimating the user's energy expenditure; a means for generating meal suggestions based on the user's calorie and nutritional needs using a generative AI model; and a means for notifying the user of the generated meal suggestions. This system enables accurate meal suggestions tailored to each individual user's lifestyle and health status, providing support for maintaining an appropriate nutritional balance. The system also has a function for collecting user feedback to improve the accuracy of future suggestions, thereby enabling more effective health management for users.

[0006] "User information" refers to personal data of system users, such as their age, gender, height, weight, and health status.

[0007] "Image data" refers to digital data including photos and videos of the meals consumed by the user.

[0008] "Image analysis engine" refers to analytical software that identifies the calories and major nutrients of food from uploaded image data.

[0009] "Energy expenditure" refers to data used to estimate total daily calories burned based on a user's activity level.

[0010] "Generative AI Model" refers to the artificial intelligence algorithm used to generate meal suggestions based on a user's calorie and nutritional needs.

[0011] "Meal Suggestions" refers to specific meal menus, recipes, ingredients, serving sizes, and cooking methods provided to users.

[0012] "Feedback" refers to information about the results and impressions provided by users after consuming the suggested diet.

[0013] "Server" refers to the central processing unit that receives, stores, analyzes data from users, and generates appropriate meal recommendations.

[0014] "Terminal" refers to a device (such as a smartphone, tablet, or PC) through which a user accesses the system and inputs and receives information. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0016] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0019] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0020] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0021] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0023] [First embodiment]

[0024] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0025] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0028] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0031] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0035] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0036] A specific embodiment of a system for implementing the present invention will be described below. This system makes appropriate meal suggestions based on the user's individual health information and dietary data. The overall configuration of the system consists of steps to process the user's input information, make meal suggestions using a generative AI model based on the analysis results, and notify the user and collect feedback.

[0037] System Overview

[0038] Registering user information

[0039] Users log in to the system and enter personal data such as age, gender, height, weight, and health status. The terminal receives this information and sends it to the server, which then stores the received user information in a database.

[0040] Uploading meal information

[0041] Users take photos of the food they eat before and after eating and upload the image data to the system via their device. The device sends these images to the server, which receives the image data and transfers them to the image analysis engine.

[0042] Image analysis and calorie counting

[0043] The server uses an image analysis engine to identify the calories and macronutrients of the food from the uploaded image. The server then aggregates the calorie and nutrient information to calculate the total calories and macronutrient intake of the user that day. The calculation results are stored in a database.

[0044] Energy Expenditure Estimation

[0045] Users input their daily activity levels (e.g., commuting) into the system via their device. The device then sends the information to the server, which then estimates the user's total daily energy expenditure based on their activity level. The estimated results are stored in a database.

[0046] Generating meal suggestions

[0047] The server uses a generative AI model to create appropriate dinner suggestions based on the user's total calorie intake, nutrient content, and energy expenditure data. The generative AI model analyzes the user's calorie and nutritional needs and generates a meal menu based on them. This menu includes specific recipes, ingredients, quantities, and cooking methods. It also provides advice on convenience store substitutions.

[0048] Notification of proposal details

[0049] The server compiles the generated meal suggestions into images and text format and sends them to the device. The device displays the received meal suggestions on the user's screen. The user checks the suggested meals and prepares dinner accordingly.

[0050] Gathering feedback

[0051] After consuming the suggested meal, the user takes a photo of the meal they actually ate and uploads it to the system via their device. The device then sends the image to the server. The server receives feedback from the user and stores the analysis results in a database using an image analysis engine. This leads to improved accuracy of meal suggestions for future visits.

[0052] Specific examples

[0053] For example, let's say a 30-year-old female user A uses the system. After logging in, user A enters her age, gender, height, weight, and health condition, and eats toast and salad for breakfast, and bibimbap bento and pudding for lunch. This information is uploaded to the system as image data. The server uses an image analysis engine to calculate the calorie intake for that day, and the generative AI model makes a dinner suggestion based on user A's energy expenditure from her commute. Ultimately, a specific meal menu such as "grilled chicken salad, tofu vegetable soup, and brown rice" is suggested, and user A follows it. After eating, user A uploads an image of her actual meal to the system, which will help improve the accuracy of suggestions from next time onwards.

[0054] In this way, the present system effectively supports health management according to the individual needs of the user.

[0055] The processing flow will be explained below.

[0056] Step 1:

[0057] Registering user information

[0058] User: Logs into the system and enters personal data such as age, gender, height, weight, and health status.

[0059] Terminal: Receives input information and sends it to the server.

[0060] Server: Stores the received user information in a database.

[0061] Step 2:

[0062] Uploading meal information

[0063] User: Takes pictures of meals such as breakfast, lunch, and dinner on the device and uploads them to the system.

[0064] Terminal: Sends the captured image data to the server.

[0065] Server: Transfers the received image data to the image analysis engine.

[0066] Step 3:

[0067] Image analysis and calorie counting

[0068] Server: Uses an image analysis engine to identify the calories and key nutrients of a meal from an uploaded image.

[0069] Server: Aggregates the identified calorie and nutrient information and calculates the total calories and macronutrients consumed by the user for the day.

[0070] Server: Saves the calculation results in a database.

[0071] Step 4:

[0072] Energy Expenditure Estimation

[0073] User: Enters daily activity data (commuting, exercise, etc.) on the device and sends it to the system.

[0074] Terminal: Sends the entered activity information to the server.

[0075] Server: Estimates total daily energy expenditure based on the user's activity level.

[0076] Server: Saves the estimation results in a database.

[0077] Step 5:

[0078] Generating meal suggestions

[0079] Server: Uses a generative AI model to create appropriate meal suggestions based on the user's total calorie intake, nutrient content, and energy expenditure data.

[0080] Server: Runs the generative AI model, analyzes the user's calorie and nutritional needs, and generates a meal menu based on that.

[0081] Server: Generated meal suggestions include specific recipes, ingredients, serving sizes, and cooking methods, and also provide advice on convenience store substitutions.

[0082] Step 6:

[0083] Notification of proposal details

[0084] Server: The generated meal suggestions are compiled into images and text format and sent to the device.

[0085] Device: Displays the received meal suggestions on the user's screen.

[0086] User: Review the suggestions provided and prepare your meal accordingly.

[0087] Step 7:

[0088] Gathering feedback

[0089] User: After consuming the suggested meal, the user takes a photo of the menu item they actually ate with their device and uploads it to the system.

[0090] Terminal: Sends the captured image to the server.

[0091] Server: Receives user feedback and forwards it to the image analysis engine.

[0092] Server: The analysis results are saved in a database and used to improve the accuracy of future meal suggestions.

[0093] In this way, the system performs a series of processes based on the data input by the user and makes meal suggestions tailored to individual needs.

[0094] Example 1

[0095] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0096] Conventional health management systems have had problems with the accuracy of their balanced dietary recommendations and the inability to fully reflect the user's individual health and activity data. Furthermore, the dietary recommendations lacked specificity, making it difficult for users to actually consume the suggested meals. Furthermore, there was also the issue of not effectively utilizing feedback after consumption, which led to poor accuracy in future recommendations.

[0097] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0098] In this invention, the server includes: a means for inputting personal information such as the user's age, gender, height, weight, and health status; a means for uploading image data of the user's meals; a means for identifying the calories and nutrients of the meals using an image analysis engine; a means for estimating the user's energy expenditure; a means for generating meal suggestions based on the user's calorie and nutritional needs using a generative AI model; a means for notifying the user of the generated meal suggestions; a means for authenticating the user upon login; a means for transmitting and storing photo data on the server; a means for inputting the user's activity data and estimating energy expenditure; and a means for displaying the generated suggestions to the user in text and image format. This enables highly accurate meal suggestions that accurately reflect the user's individual health information and actual intake and activity data, providing more specific and easy-to-follow menu suggestions. Furthermore, by utilizing feedback, the accuracy of future suggestions can be improved.

[0099] "Means for inputting personal information such as the user's age, sex, height, weight, and health condition" refers to the means by which a user inputs personal information such as their age, sex, height, weight, and health condition into the system and transmits that information to the server.

[0100] The "means for uploading image data of meals taken by the user" refers to a means for the user to upload image data of meals taken using the camera function to the system and send the images to the server.

[0101] "Means for identifying calories and nutrients in meals using an image analysis engine" refers to means for analyzing uploaded images of meals and using image analysis software or algorithms to identify calories and nutrients in the meals.

[0102] "Means for estimating a user's energy expenditure" refers to means for calculating and estimating a user's daily energy expenditure based on the user's daily activity data and exercise data.

[0103] "Means for using a generative AI model to generate meal suggestions based on a user's calorie and nutritional needs" means means for using an artificial intelligence model to analyze a user's calorie intake and nutritional needs and generate appropriate meal menus and recipes.

[0104] The "means for notifying the user of the generated meal suggestions" refers to means for notifying and displaying the generated meal suggestions in text and image format on the user's terminal.

[0105] "Means for authenticating users when they log in" refers to the means for authenticating users using an ID and password, etc., when they access the system.

[0106] The "means for transmitting photo data and storing it on the server" refers to the means for transmitting photo data taken by the user to the server and storing it within the server.

[0107] "Means for inputting user activity data and estimating energy expenditure" refers to means for a user to input data about their own activities and exercise and estimate energy expenditure based on that data.

[0108] "Means for displaying the generated suggestions to the user in text and image format" refers to means for displaying the contents of the generated meal suggestions to the user in text and image format on the user's terminal and visually presenting them to the user.

[0109] The present invention is a system that makes appropriate meal recommendations based on a user's individual health information and dietary data. The system of the present invention consists of steps to process the user's input information, make meal recommendations using a generative AI model based on the analysis results, and notify the user and collect feedback. The specific system configuration is described below.

[0110] Registering user information

[0111] The user logs in to the system and enters personal data such as age, gender, height, weight, and health condition. The device collects this information and sends it to the server. The server stores the received data in a database. Through this process, the user's basic health information is accumulated in the system.

[0112] Uploading meal information

[0113] A user takes a photo of their meal using a mobile device or camera and uploads the image data. The device sends the captured image to a server. The server then forwards the received image data to an image analysis engine. This analysis engine can be, for example, image analysis software.

[0114] Image analysis and calorie counting

[0115] The server uses an image analysis engine to analyze the contents of the meal from the uploaded image and identify calories and key nutrients. The analysis can be performed using cloud-based image analysis software. The identified nutrient and calorie data is aggregated and stored in a database on the server.

[0116] Energy Expenditure Estimation

[0117] The user enters their daily activity (e.g., commuting) into the device. The device then sends the information to the server. The server analyzes the user's activity data and estimates their energy expenditure. This analysis can be performed using activity tracking software. The estimated results are stored in a database.

[0118] Generating meal suggestions

[0119] The server uses a generative AI model to generate appropriate meal suggestions based on the user's total calorie intake, nutrient content, and energy expenditure data. This process can be performed using generative AI model software, for example. The generated menu includes specific recipes, ingredients, serving sizes, and cooking methods.

[0120] For example, here's a prompt to input to a generative AI model:

[0121] "Please suggest a balanced dinner menu for a 30-year-old woman based on the following criteria: Breakfast: toast and salad, Lunch: boxed lunch and dessert, Daily exercise: light exercise during commute."

[0122] Notification of proposal details

[0123] The server then compiles the generated meal suggestions into images and text format and sends them to the device, which then displays them on the user's screen. The user can then review the suggested menu and follow the specific steps to execute it.

[0124] Gathering feedback

[0125] After consuming the suggested meal, the user takes a photo of the meal and uploads it back to the system. The device then sends the image to the server, which then receives it, analyzes it using an image analysis engine, and stores the results in a database. This feedback improves the accuracy of future meal suggestions.

[0126] As described above, this system can respond to the individual needs of users and support more effective health management.

[0127] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0128] Step 1: Register your user information

[0129] Input: The user logs into the system and enters personal information such as age, gender, height, weight, and health status.

[0130] Specific behavior: A user accesses the system using a web browser or mobile app. They enter their credentials on the login screen to log in to the system. They then enter the required personal data on the profile screen.

[0131] Output: The device sends the entered personal data to the server, which stores the received data in a database.

[0132] Step 2: Upload your meal information

[0133] Input: The user takes a photo of the food they have eaten and uploads the image data to the system.

[0134] Specific actions: The user takes a photo of their meal using the smartphone camera, selects the photo in the form to upload it to the system, and clicks the submit button.

[0135] Output: The terminal sends the image data to the server, which then transfers the received image data to the image analysis engine.

[0136] Step 3: Image analysis and calorie calculation

[0137] Input: The server analyzes the image data using an image analysis engine.

[0138] Specific operation: The server passes the received image data to the image analysis engine and sends an API request. The image analysis engine analyzes the contents of the meal and returns information about calories and nutrients.

[0139] Output: The server aggregates the calorie and nutrient information obtained from the image analysis engine and stores it in a database.

[0140] Step 4: Estimate energy consumption

[0141] Input: The user inputs their daily activity level into the device.

[0142] Specific actions: The user enters information about their daily activities (commuting, exercise, etc.) into a form on the device and clicks the submit button.

[0143] Output: The device sends the input activity data to the server, which uses activity tracking software or an API to estimate energy expenditure based on the input data and stores the results in a database.

[0144] Step 5: Generate meal suggestions

[0145] Input: The server retrieves data on total calorie intake, nutrient amounts, and energy expenditure from a database.

[0146] Specific operation: The server reads the user's latest health and dietary information from the database and sends prompt sentences to the generative AI model.

[0147] Example prompt: "For a 30-year-old woman, please suggest a balanced dinner menu based on the following criteria: Breakfast: toast and salad; Lunch: boxed lunch and dessert; Daily exercise: light exercise during commute."

[0148] Output: Take the generated meal suggestions and format them as a menu with specific recipes, ingredients, serving sizes, and cooking instructions.

[0149] Step 6: Notification of proposal

[0150] Input: Generated meal suggestions

[0151] Specific operation: The server sends the generated meal suggestions in text and image format to the device, which then displays them on the user's screen.

[0152] Output: The user confirms and implements the received meal suggestions.

[0153] Step 7: Gather feedback

[0154] Input: The user consumes the suggested meal and then takes and uploads a photo of the meal after consumption.

[0155] Specific operation: After eating a meal, the user takes a photo and uploads it to the system. The device then sends the photo data to the server.

[0156] Output: The server passes the received image back to the image analysis engine and stores the analysis results in a database, thereby improving the accuracy of future meal suggestions.

[0157] (Application example 1)

[0158] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0159] Many users today struggle to manage their health and daily diet. Even when they receive healthy dietary recommendations, they lack concrete advice on how to translate them into real-world purchasing behavior. This creates a growing need for a system that can efficiently select the right ingredients and meal plans.

[0160] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0161] In this invention, the server includes means for inputting information such as the user's age, sex, height, weight, and health condition, means for uploading image data of meals consumed by the user, means for identifying the calories and nutrients of the meals using an image analysis engine, means for estimating the user's energy expenditure, means for generating meal suggestions based on the user's calorie and nutritional needs using a generative AI model, means for notifying the user of the generated meal suggestions, and means for displaying the suggested ingredients and menus when the user visits a physical store. This makes it easier for users to reflect healthy meal suggestions in their actual purchasing behavior, enabling more effective health management.

[0162] "User information" refers to basic physical data about each individual user, such as age, gender, height, weight, and health status.

[0163] "Meal image data" refers to digitally saved photographs of meals eaten by a user.

[0164] An "image analysis engine" is software or algorithm that analyzes image data to identify the calories and nutrients of a meal.

[0165] "Energy expenditure" means the total amount of calories burned through a user's daily activities and exercise.

[0166] A "generative AI model" is a model that uses artificial intelligence to generate personalized meal suggestions based on user data.

[0167] "Meal Suggestions" means suggestions that provide specific meals or menus to be consumed based on the user's calorie and nutritional needs.

[0168] "Notification means" refers to a method or device for notifying the user of the generated meal suggestions.

[0169] "Physical stores" refer to physical stores that users visit and use, such as grocery stores and restaurants.

[0170] "Means for displaying ingredients and menu items" refers to methods or devices that visually display suggested ingredients and menu items when a user visits a physical store.

[0171] A specific embodiment of a system for implementing the present invention will be described below. This system makes appropriate meal suggestions based on a user's individual health information and dietary data, and supports purchasing behavior, particularly in physical stores. This system involves inputting user information, analyzing dietary image data, estimating energy consumption, generating and notifying meal suggestions, and displaying ingredients and menus in physical stores.

[0172] System Overview

[0173] (1) User information registration:

[0174] Users log in to the system and enter personal data such as age, sex, height, weight, health condition, etc. through a terminal. The terminal sends this information to the server, which stores it in a database.

[0175] (2) Uploading meal information:

[0176] Users take photos of the food they eat and upload the image data to the system via their device, which then sends the images to the server, which then forwards them to the image analysis engine.

[0177] (3) Image analysis and calorie calculation:

[0178] The server uses an image analysis engine to identify the calories and key nutrients of the food from the uploaded image, and the calculations are stored in a database.

[0179] (4) Energy consumption estimation:

[0180] Users input their daily activity levels (e.g., commuting) into the system via their devices. The server estimates the total daily energy expenditure based on the user's activity level and stores the estimated results in a database.

[0181] (5) Meal suggestion generation:

[0182] Based on the user's total calorie intake, nutrient content, and energy expenditure data, the server uses a generative AI model to create appropriate meal suggestions, including specific recipes, ingredients, serving sizes, and cooking methods, and also provides advice on substitute ingredients from convenience stores and supermarkets.

[0183] (6) Notification of proposal:

[0184] The server sends the generated meal suggestions to the terminal, which displays them on the user's screen. The user can then confirm the suggested meals and act accordingly.

[0185] (7) Display in physical stores:

[0186] When a user visits a physical store, they use their device to scan the store's QR code, which allows the server to display ingredients and menu items to purchase in real time based on the user's meal suggestions.

[0187] (8) Feedback Collection:

[0188] After consuming the suggested meal, the user takes a photo of the meal and uploads it to the system via their device. The server receives the feedback and stores the analysis results in a database using an image analysis engine. This improves the accuracy of meal suggestions for future visits.

[0189] Examples:

[0190] For example, let's say User A, a 30-year-old woman, uses this system. After logging in, User A enters her age, gender, height, weight, and health status, and eats toast and salad for breakfast, and bibimbap bento and pudding for lunch. This information is uploaded to the system as image data. The server uses an image analysis engine to calculate the calorie intake for that day, and a generative AI model uses the energy expenditure from User A's commute to suggest dinner. Ultimately, a specific meal menu such as "grilled chicken salad, tofu vegetable soup, and brown rice" is suggested. User A goes to a nearby physical store and scans the store's QR code to check the ingredients and menu items to purchase on her smartphone. She then purchases the ingredients based on the suggestions and cooks. After eating, User A uploads an image of the menu she actually ate back to the system, contributing to improving the accuracy of suggestions from next time onwards.

[0191] Example prompt sentence:

[0192] User ID: 12345

[0193] Remaining calorie intake: 600

[0194] Suggest a suitable dinner menu based on the following criteria:

[0195] 1. Calories: 600 kcal or less

[0196] 2. Health status: Healthy adult female

[0197] 3. Nutritional balance: high protein, low carbohydrate

[0198] Suggested menu items should include specific ingredients, portions, and cooking methods.

[0199] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0200] Step 1:

[0201] Users log in to the device and enter information such as their age, gender, height, weight, and health condition. The device receives this information and sends it to the server, which then stores the received user information in a database.

[0202] Input: User information (age, gender, height, weight, health condition)

[0203] Output: User information stored in the database

[0204] Specific operation: When a user fills in information in the application's input form and presses the "Save" button, the terminal sends it to the server, which then stores it in the database.

[0205] Step 2:

[0206] Users take photos of the food they eat and upload the image data to the system via their device. The device sends these images to the server, which then forwards them to the image analysis engine.

[0207] Input: Food image data

[0208] Output: Image data sent to the server

[0209] How it works: When a user takes a photo of their meal and presses the "upload" button in the application, the device sends the image data to the server, which then passes it to the image analysis engine.

[0210] Step 3:

[0211] The server uses an image analysis engine to identify the calories and key nutrients of the food from the uploaded image, and the analysis results are stored in a database.

[0212] Input: Food image data

[0213] Output: Calorie and macronutrient data for the meal

[0214] Specific operation: The image analysis engine inputs image data into an algorithm and performs analysis. The analysis results are stored in a database via a server.

[0215] Step 4:

[0216] Users input their daily activity levels (e.g., commuting) into the system via their devices. The server receives the information and estimates the user's total daily energy expenditure based on their activity level. This estimation is then stored in a database.

[0217] Input: Activity data (commuting, exercise, etc.)

[0218] Output: Estimated energy consumption

[0219] How it works: After a user fills out the application's activity recording form and presses the "Submit" button, the device sends the data to the server, which then uses it to calculate activity levels.

[0220] Step 5:

[0221] The server uses a generative AI model to generate appropriate meal suggestions based on the user's total calorie intake, nutrient content, and energy expenditure data, including specific recipes, ingredients, serving sizes, and cooking methods.

[0222] Input: Total calories, nutrient content, energy expenditure

[0223] Output: Specific meal suggestions (recipe, ingredients, portions, cooking method)

[0224] How it works: The server inputs this data into a generative AI model to generate optimal meal suggestions for the user.

[0225] Step 6:

[0226] The server sends the generated meal suggestions to the terminal, which displays them on the user's screen. The user can then confirm the suggested meals and act accordingly.

[0227] Input: Meal suggestion data

[0228] Output: Meal suggestions displayed on the user's device

[0229] Specific operation: The server generates meal suggestions and sends them to the device, which receives them and displays them on the application.

[0230] Step 7:

[0231] When a user visits a physical store, they use their device to scan the store's QR code, and the server displays ingredients and menu items to purchase in real time based on the user's meal suggestions.

[0232] Input: QR code from physical store, suggested meal data

[0233] Output: List of ingredients and menu items to purchase

[0234] How it works: The user scans a QR code at a store and sends the information to the server, which then generates a shopping list based on the user's meal suggestions and displays it on the device.

[0235] Step 8:

[0236] After consuming the suggested meal, the user takes a photo of the meal and uploads it to the system via their device. The server receives the feedback and stores the analysis results in a database using an image analysis engine.

[0237] Input: Image data of the menu you ate

[0238] Output: Parsed feedback data

[0239] How it works: The user takes a photo of the menu item they ate and uploads it to the server through the application. The server passes the data to the image analysis engine and stores the analysis results in a database.

[0240] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0241] A specific embodiment of a system for implementing the present invention will be described below. This system recognizes the user's emotional state in addition to their individual health information and dietary data to make meal suggestions. The overall configuration of the system consists of the steps of processing the user's input information, making meal suggestions based on the analysis results using a generative AI model and emotion engine, and notifying the user and collecting feedback.

[0242] System Overview

[0243] Registering user information

[0244] Users log in to the system and enter personal data such as age, gender, height, weight, and health status. The terminal receives this information and sends it to the server, which then stores the received user information in a database.

[0245] Collecting Emotional Data

[0246] Users can input data about their physical condition and mood, or communicate their emotions to the system using facial and voice recognition technology. The device receives this emotional data and sends it to the server, which then transfers it to the emotion engine.

[0247] Uploading meal information

[0248] Users take photos of the food they eat before and after eating and upload the image data to the system via their device. The device sends these images to the server, which receives the image data and transfers them to the image analysis engine.

[0249] Image analysis and calorie counting

[0250] The server uses an image analysis engine to identify the calories and macronutrients of the food from the uploaded image. The server then aggregates the calorie and nutrient information to calculate the total calories and macronutrient intake of the user that day. The calculation results are stored in a database.

[0251] Energy Expenditure Estimation

[0252] Users input their daily activity levels (e.g., commuting) into the system via their device. The device then sends the information to the server, which then estimates the user's total daily energy expenditure based on their activity level. The estimated results are stored in a database.

[0253] Emotion analysis and reflection

[0254] The server uses an emotion engine to analyze the user's current emotional state from the emotion data. The analysis results are used to adjust subsequent meal suggestions. For example, if the user is feeling stressed, the suggestions will include ingredients with a relaxation effect.

[0255] Generating meal suggestions

[0256] The server uses a generative AI model to create appropriate meal suggestions based on the user's total calorie intake, nutrient content, energy expenditure, and emotional state. The generative AI model generates meal menus that take into account the user's calorie and nutritional needs as well as their emotional state. The menus include specific recipes, ingredients, serving sizes, and cooking methods. It also provides advice on convenience store substitutions.

[0257] Notification of proposal details

[0258] The server compiles the generated meal suggestions into images and text format and sends them to the device. The device displays the received meal suggestions on the user's screen. The user checks the displayed suggestions and prepares dinner accordingly.

[0259] Gathering feedback

[0260] After consuming the suggested meal, the user takes a photo of the meal they actually ate and uploads it to the system via their device. The device then sends the image to the server. The server receives feedback from the user and stores the analysis results in a database using an image analysis engine. This leads to improved accuracy of meal suggestions for future visits.

[0261] Specific examples

[0262] For example, let's say User B, a 30-year-old woman, uses the system. After logging in, User B enters her age, gender, height, weight, and health condition, and also enters emotional data about the stress she is currently feeling. She uploads image data to the system, showing that she had oatmeal and fruit for breakfast and a sandwich and salad for lunch. The server uses an image analysis engine to calculate the calories consumed that day, and uses a generative AI model to suggest dinner based on User B's energy expenditure during her commute. Additionally, based on her emotional state analyzed by the emotion engine, it suggests relaxing herbal tea and meals rich in B vitamins.

[0263] In this way, the system provides appropriate and effective health management and dietary recommendations based on the user's individual needs and emotional state.

[0264] The processing flow will be explained below.

[0265] Step 1:

[0266] Registering user information

[0267] User: Logs into the system and enters personal data such as age, gender, height, weight, and health status.

[0268] Terminal: Receives input information and sends it to the server.

[0269] Server: Stores the received user information in a database.

[0270] Step 2:

[0271] Collecting Emotional Data

[0272] User: Enters data about physical condition and mood into the device, and communicates emotions to the system using facial and voice recognition technology.

[0273] Terminal: Receives these emotion data and sends them to the server.

[0274] Server: Transfers the received emotion data to the emotion engine.

[0275] Step 3:

[0276] Uploading meal information

[0277] User: Takes pictures of meals such as breakfast, lunch, and dinner on the device and uploads them to the system.

[0278] Terminal: Sends the captured image data to the server.

[0279] Server: Transfers the received image data to the image analysis engine.

[0280] Step 4:

[0281] Image analysis and calorie counting

[0282] Server: Uses an image analysis engine to identify the calories and key nutrients of a meal from an uploaded image.

[0283] Server: Aggregates the identified calorie and nutrient information and calculates the total calories and macronutrients consumed by the user for the day.

[0284] Server: Saves the calculation results in a database.

[0285] Step 5:

[0286] Energy Expenditure Estimation

[0287] User: Enters daily activity data (commuting, exercise, etc.) on the device and sends it to the system.

[0288] Terminal: Sends the entered activity information to the server.

[0289] Server: Estimates total daily energy expenditure based on the user's activity level.

[0290] Server: Saves the estimation results in a database.

[0291] Step 6:

[0292] Emotion analysis and reflection

[0293] Server: Uses the emotion engine to analyze the user's current emotional state from the emotion data.

[0294] Server: The analysis results are stored and used to adjust future meal suggestions. For example, if the user is feeling stressed, the suggestions may include foods with a relaxing effect.

[0295] Step 7:

[0296] Generating meal suggestions

[0297] Server: Uses a generative AI model to create appropriate meal suggestions based on the user's total calorie intake, nutrient amounts, energy expenditure, and emotional state data.

[0298] Server: Runs the generative AI model, analyzes the user's calorie and nutritional needs, and generates a meal menu based on that.

[0299] Server: Generated meal suggestions include specific recipes, ingredients, serving sizes, and cooking methods, and also provide advice on convenience store substitutions.

[0300] Step 8:

[0301] Notification of proposal details

[0302] Server: The generated meal suggestions are compiled into images and text format and sent to the device.

[0303] Device: Displays the received meal suggestions on the user's screen.

[0304] User: Review the suggestions provided and prepare your meal accordingly.

[0305] Step 9:

[0306] Gathering feedback

[0307] User: After consuming the suggested meal, the user takes a photo of the menu item they actually ate with their device and uploads it to the system.

[0308] Terminal: Sends the captured image to the server.

[0309] Server: Receives user feedback and forwards it to the image analysis engine.

[0310] Server: The analysis results are saved in a database and used to improve the accuracy of future meal suggestions.

[0311] In this way, the system processes user input and emotional data to provide personalized meal recommendations, and can continuously improve the accuracy of these recommendations through feedback.

[0312] Example 2

[0313] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0314] In today's busy lifestyles, it is difficult for users to receive appropriate dietary recommendations based on their health status and emotions. Furthermore, there is no system that can provide dietary recommendations that take into account the user's emotional state, rather than just calories and nutrients. This leads to a delay in effective health management and dietary improvement.

[0315] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0316] In this invention, the server includes means for inputting information such as the user's age, sex, height, weight, and health condition, means for uploading image data of meals consumed by the user, means for identifying the calories and nutrients of the meals using an image analysis engine, means for estimating the user's energy expenditure, means for collecting the user's emotion data and analyzing it using an emotion engine, means for generating meal suggestions based on the user's calorie and nutritional needs and emotional state using a generative AI model, and means for notifying the user of the generated meal suggestions. This enables appropriate meal suggestions based on the user's health condition and emotions.

[0317] "User" refers to an individual who uses this system.

[0318] "Age" refers to the number of years since the user was born.

[0319] "Gender" refers to a user's biological or social sex.

[0320] "Height" refers to the vertical length of the user's body.

[0321] "Weight" refers to the mass of a user's body.

[0322] "Health Status" refers to the state of a User's physical and mental health.

[0323] "Information" refers to data such as the user's age, gender, height, weight, and health status.

[0324] "Means" refers to a method or device for achieving a particular purpose.

[0325] "Meal image data" refers to image files of meals consumed by the user.

[0326] "Upload" refers to the act of sending data from a user's device to a server.

[0327] "Image analysis engine" refers to software or hardware for analyzing image data and extracting specific information.

[0328] "Calories" refers to the amount of energy a food contains.

[0329] "Nutrients" are substances contained in food that are necessary for the growth and maintenance of health of the body.

[0330] "Energy Expenditure" refers to the total amount of energy consumed by a user's daily activities.

[0331] "Emotional data" refers to data related to the user's physical condition and mood.

[0332] "Emotion engine" refers to software or hardware for analyzing emotional data to identify a user's emotional state.

[0333] A "generative AI model" refers to an artificial intelligence system that generates appropriate meal suggestions based on user information.

[0334] "Notification" refers to the act of informing the user of the generated meal suggestions.

[0335] A "system" refers to a set of multiple means or devices that function in conjunction with one another.

[0336] A specific embodiment of a system for implementing the present invention will be described below. This system recognizes the user's emotional state as well as their individual health information and dietary data to make meal suggestions. This system processes the user's input information, makes meal suggestions based on the analysis results using a generative AI model and an emotion engine, and notifies the user and collects feedback.

[0337] Registering user information

[0338] Users log in to the system and enter personal data such as age, gender, height, weight, and health status. The terminal receives this information and sends it to the server, which then stores the received user information in a database.

[0339] Collecting Emotional Data

[0340] Users can input data about their physical condition and mood, or communicate their emotions to the system using facial and voice recognition technology. The device receives this emotional data and sends it to the server, which then transfers it to the emotion engine.

[0341] Uploading meal information

[0342] Users take photos of the food they eat before and after eating and upload the image data to the system via their device. The device sends these images to the server, which receives the image data and transfers them to the image analysis engine.

[0343] Image analysis and calorie counting

[0344] The server uses an image analysis engine to identify the calories and macronutrients of the food from the uploaded image. The server then aggregates the calorie and nutrient information to calculate the total calories and macronutrient intake of the user that day. The calculation results are stored in a database.

[0345] Energy Expenditure Estimation

[0346] Users input their daily activity levels (e.g., commuting) into the system via their device. The device then sends the information to the server, which then estimates the user's total daily energy expenditure based on their activity level. The estimated results are stored in a database.

[0347] Emotion analysis and reflection

[0348] The server uses an emotion engine to analyze the user's current emotional state from the emotion data. The analysis results are used to adjust subsequent meal suggestions. For example, if the user is feeling stressed, the suggestions will include ingredients with a relaxation effect.

[0349] Generating meal suggestions

[0350] The server uses a generative AI model to create appropriate meal suggestions based on the user's total calorie intake, nutrient content, energy expenditure, and emotional state. The generative AI model generates meal menus that take into account the user's calorie and nutritional needs as well as their emotional state. The menus include specific recipes, ingredients, serving sizes, and cooking methods. It also provides advice on convenience store substitutions.

[0351] Notification of proposal details

[0352] The server compiles the generated meal suggestions into images and text format and sends them to the device. The device displays the received meal suggestions on the user's screen. The user checks the displayed suggestions and prepares dinner accordingly.

[0353] Gathering feedback

[0354] After consuming the suggested meal, the user takes a photo of the meal they actually ate and uploads it to the system via their device. The device then sends the image to the server. The server receives feedback from the user and stores the analysis results in a database using an image analysis engine. This leads to improved accuracy of meal suggestions for future visits.

[0355] Specific examples

[0356] For example, suppose a 30-year-old female user uses the system. After logging in, the user enters her age, gender, height, weight, and health condition, and also enters emotional data about the stress she is currently experiencing. She uploads image data of her breakfast of oatmeal and fruit, and lunch of a sandwich and salad, to the system. The server uses an image analysis engine to calculate the calorie intake for that day, and uses a generative AI model to suggest dinner based on the user's energy expenditure from commuting. Based on the emotional state analyzed by the emotion engine, the system also suggests relaxing herbal teas and meals rich in B vitamins. In this way, the system provides appropriate and effective health management and dietary suggestions based on the user's individual needs and emotional state.

[0357] Prompt Sentence Examples

[0358] "A 30-year-old female user uses the system and enters her age, gender, height, weight, and health status. The user is currently stressed and ate oatmeal and fruit for breakfast, and a sandwich and salad for lunch. Please generate dinner suggestions taking into account her energy expenditure from her commute."

[0359] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0360] Step 1: Register your user information

[0361] Users log in to the system and enter personal data such as age, gender, height, weight, and health status.

[0362] Input: Age, gender, height, weight, health condition

[0363] Output: User information data (e.g., JSON format)

[0364] The terminal receives the entered information and transmits the data to the server.

[0365] Specific operation: The terminal encodes the input data into JSON format and sends it to the server using the HTTPS protocol.

[0366] The server stores the received data in a database.

[0367] Input: User information data (e.g., JSON format)

[0368] Output: User information record in database

[0369] Specific operation: The server analyzes the received data and performs an insert operation on a specific table in the database.

[0370] Step 2: Collecting emotion data

[0371] Users input data about their physical condition and mood, or communicate their emotions to the system using facial and voice recognition technology.

[0372] Input: Physical condition, mood data, or emotional data

[0373] Output: Emotion data (numerical and text format)

[0374] The terminal receives this data and transmits it to the server.

[0375] Specific operation: The device generates an API request to send emotion data to the server in real time.

[0376] The server transfers the received emotion data to the emotion engine for analysis.

[0377] Input: Received emotion data

[0378] Output: Parsed emotional state data

[0379] Specific operation: The server passes the emotion data to the emotion engine's API and receives the analysis results.

[0380] Step 3: Upload your meal information

[0381] Users take photos of the food they eat before and after eating and upload the image data to the system.

[0382] Input: Food image

[0383] Output: Image data file

[0384] The terminal transmits the image data to the server.

[0385] Specific operation: The device compresses the captured image and generates an API request to upload it to the server.

[0386] The server receives the image data and transfers it to the image analysis engine.

[0387] Input: Image data file

[0388] Output: Image analysis results (calories, nutrient data)

[0389] Specific operation: The server calls the API of the image analysis engine and sends a request to analyze the image data.

[0390] Step 4: Image analysis and calorie calculation

[0391] The server uses an image analysis engine to identify the calories and key nutrients of a meal from the uploaded image.

[0392] Input: Food image data

[0393] Output: Calorie information, nutrient information

[0394] How it works: The image analysis engine applies algorithms to recognize ingredients in an image and identify the calorie and nutritional information for each ingredient.

[0395] The server compiles the results and calculates the user's total calorie and nutrient intake.

[0396] Input: Calorie information and nutrient information for each meal

[0397] Output: Total calorie intake and macronutrient data

[0398] What it does: The server uses an aggregation algorithm to add up the calories and nutrients for each meal.

[0399] The calculation results are saved in a database.

[0400] Input: Total calorie intake and amount of major nutrients

[0401] Output: Accumulated data in the database

[0402] Specific operation: The server performs an insert operation on the results into a specific table in the database.

[0403] Step 5: Estimate energy consumption

[0404] Users enter their daily activity levels into the system.

[0405] Input: Activity data (commuting, exercise, etc.)

[0406] Output: Activity log

[0407] The terminal transmits the information to the server.

[0408] Specific operation: The device encodes the activity data and sends an API request to the server.

[0409] The server estimates total energy expenditure based on the user's activity level.

[0410] Input: Activity data

[0411] Output: Total energy consumption data

[0412] Specific operation: The server applies a calculation algorithm based on the activity data to estimate energy expenditure.

[0413] The estimation results are saved in a database.

[0414] Input: Total energy consumption data

[0415] Output: Energy consumption records in a database

[0416] Specific operation: The server inserts the estimation results into the database.

[0417] Step 6: Analyze and reflect on emotions

[0418] The server uses an emotion engine to parse the user's current emotional state from the emotion data.

[0419] Input: Emotion data

[0420] Output: Parsed emotional state

[0421] Specific operation: The emotion engine analyzes the emotion data and generates a quantified emotional state.

[0422] Configure settings to reflect the analysis results in meal suggestions.

[0423] Input: Parsed emotional state

[0424] Output: Adjusted meal suggestion data

[0425] What it does: The server applies an algorithm that adjusts the suggestions depending on the emotional state.

[0426] Step 7: Generate meal suggestions

[0427] The server uses a generative AI model to create appropriate meal suggestions based on the user's total calorie intake, nutrient amounts, energy expenditure, and emotional state data.

[0428] Input: User's total calorie intake, nutritional information, energy expenditure, emotional state data

[0429] Output: Generated meal suggestions

[0430] Specific operation: The generative AI model generates an appropriate meal menu based on the prompt text.

[0431] The generated suggestions include specific recipes, ingredients, quantities, and cooking methods.

[0432] Input: Generated meal suggestions

[0433] Output: Detailed meal menu (including recipe, ingredients, portions, and cooking method)

[0434] What it does: The server formats the results of the generative AI model and compiles them into a detailed menu.

[0435] Step 8: Notification of proposal

[0436] The server compiles the generated meal suggestions into image and text format and sends them to the terminal.

[0437] Input: Detailed meal menu

[0438] Output: Notification format

[0439] Specific operation: The server converts the menu information into image and text format and sends it to the terminal.

[0440] The device displays the received meal suggestions on the user's screen.

[0441] Input: Notification Format

[0442] Output: The suggestions that are displayed on the user's screen

[0443] Specific operation: The terminal obtains the notification data and displays it on the user interface.

[0444] The user reviews the displayed suggestions and prepares dinner accordingly.

[0445] Step 9: Gather feedback

[0446] After consuming the suggested meal, the user takes a photo of the meal they actually ate and uploads it to the system.

[0447] Input: Food image

[0448] Output: Uploaded image data

[0449] The terminal transmits the image data to the server again.

[0450] Specific operation: The device generates an API request to compress the image and send it to the server.

[0451] The server analyzes the feedback using an image analysis engine and stores the results in a database.

[0452] Input: Uploaded image data

[0453] Output: Analysis results

[0454] Specific operation: The server calls the API of the image analysis engine again and inserts the feedback results into the database.

[0455] (Application example 2)

[0456] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0457] Conventional health management systems often do not provide meal suggestions that take into account the user's emotional state, and comprehensive health management, including the user's mental health, has not been fully realized. Furthermore, virtual stores are unable to provide meal suggestions based on real-time health information, making it difficult for users to efficiently select ingredients and recipes that are suitable for them. This makes it difficult to maintain and improve users' health through appropriate diets.

[0458] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0459] In this invention, the server includes: means for inputting information such as the user's age, sex, height, weight, and health condition; means for uploading image data of the user's meals; means for identifying the calories and nutrients of the meals using an image analysis engine; means for estimating the user's energy expenditure; means for collecting and analyzing the user's emotional data; means for generating meal suggestions based on the user's calorie and nutritional needs and emotional state using a generative AI model; means for notifying the user of the generated meal suggestions; and means for providing meal suggestions in real time in a virtual store based on the user's health information, dietary data, and emotional state. This enables comprehensive health management that takes the user's emotional state into consideration and enables the selection of appropriate ingredients and recipes in real time in the virtual store.

[0460] "User Information" refers to personal information about a user, such as their age, gender, height, weight, and health status.

[0461] "Image data" refers to photographic data of the food consumed by the user.

[0462] The "image analysis engine" is software that identifies the calories and nutrients of meals from uploaded image data.

[0463] "Energy expenditure" is the amount of energy a user expends through daily activities.

[0464] "Emotion data" is data that indicates information about the user's physical condition and mood, and is obtained using facial recognition technology or voice recognition technology.

[0465] A "generative AI model" is an artificial intelligence model that generates meal suggestions based on a user's calorie and nutritional needs and emotional state.

[0466] "Meal suggestions" are suggestions such as specific meal menus, recipes, ingredients, portions, cooking methods, etc., that are proposed to users.

[0467] A "virtual store" is a digital platform that allows users to shop in a virtual space.

[0468] "Real-time" refers to data processing and information provision occurring immediately or nearly simultaneously.

[0469] "Notification" is the act or mechanism of communicating generated meal suggestions to a user.

[0470] "Feedback" means providing the system with information about the food the user has eaten, which helps improve the accuracy of the next suggestion.

[0471] MODE FOR CARRYING OUT THE INVENTION

[0472] A detailed description of a system for realizing the present invention is given below. This system makes meal suggestions in real time in a virtual store based on the user's individual health information, dietary data, and emotional state.

[0473] System Overview

[0474] Registering user information

[0475] After accessing the virtual store, users use their smartphones or head-mounted displays (HMDs) to input personal information such as age, gender, height, weight, and health status. The devices receive this information and send it to a cloud server, which then stores the received user information in a database.

[0476] Collecting Emotional Data

[0477] Users can input data about their physical condition and mood using a camera or microphone installed on their smartphone or HMD, or communicate their emotions to the system using facial or voice recognition technology. The device receives this emotional data and sends it to a cloud server. The server then analyzes the received emotional data using an emotion engine to determine the user's current emotional state.

[0478] Uploading meal information

[0479] Users take photos of the meals they have eaten over the past 48 hours with their smartphones and upload them to the system. The device sends these images to a cloud server. The server receives the image data and transfers it to an image analysis engine. The image analysis engine identifies the calories and nutrients of the meals from the uploaded images.

[0480] Generating meal suggestions

[0481] The server uses a generative AI model based on the user's total calorie intake, nutrient content, energy expenditure, and emotional state data to generate appropriate meal suggestions. The generative AI model considers the user's calorie and nutritional needs as well as their emotional state to generate a meal menu that includes specific recipes, ingredients, serving sizes, and cooking methods.

[0482] Notification of proposal details

[0483] The server compiles the generated meal suggestions into images and text format and sends them to the device. The device then displays the received meal suggestions on the user's smartphone or HMD at appropriate locations within the virtual store. As the user moves around the virtual store, notifications pop up at appropriate locations.

[0484] Gathering feedback

[0485] After consuming the suggested meal, the user takes a photo of the meal and uploads it to the system via their device. The device then sends the image to a cloud server. The server receives feedback from the user and stores the analysis results in a database using an image analysis engine. This improves the accuracy of meal suggestions for future visits.

[0486] Hardware and Software Used

[0487] Smartphone: Use an Android or iOS device.

[0488] Head-mounted display (HMD): Use Oculus Rift, HTC Vive, etc.

[0489] Cloud server: Use cloud platforms such as Google Cloud, AWS, and Azure.

[0490] Emotion recognition technology: Uses technologies such as OpenCV and TensorFlow.

[0491] Generative AI models: Utilize generative AI such as OpenAI GPT-3.

[0492] Specific examples

[0493] For example, a user visits a virtual store and enters their personal information using a smartphone. Emotional data is collected using facial recognition with a camera installed in the HMD, and data on physical condition and mood is collected. After the user uploads photos of the meals they have eaten in the past 48 hours, the cloud server uses an image analysis engine to identify calories and nutrients, and then uses a generative AI model to make meal recommendations.

[0494] Example prompt sentence:

[0495] Based on their health, users need 2000 kcal of calories, 50g protein, 70g fat, and 250g carbohydrates.

[0496] Please suggest a dish using the following ingredients and method:

[0497] 1) Breakfast: Oatmeal and fruit

[0498] 2) Lunch: Sandwich and salad

[0499] 3) Dinner: Chicken steak and vegetables

[0500] The user's emotional state is stress.

[0501] This allows users to select appropriate ingredients and recipes in real time in a virtual store, enabling comprehensive health management that takes into account their emotional state.

[0502] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0503] Step 1:

[0504] Users access the virtual store and use their smartphone or HMD to enter personal information such as age, gender, height, weight, and health status. This information is then sent from the device to a cloud server, which then stores the received information in a database.

[0505] Input: User's age, gender, height, weight, health status

[0506] Output: User information stored in the database

[0507] Step 2:

[0508] Users input data about their physical condition and mood into the system using a camera and microphone installed on their smartphone or HMD. The device receives this emotional data and sends it to a cloud server. The server then uses an emotion engine to analyze the received emotional data and identify the user's current emotional state.

[0509] Input: Data about the user's physical condition and mood

[0510] Output: Parsed user emotional state

[0511] Step 3:

[0512] Users take photos of the meals they have eaten over the past 48 hours with their smartphones and upload them to the system. The device sends these image data to a cloud server. The server receives the image data and transfers it to an image analysis engine. The image analysis engine identifies the calories and nutrients of the meals from the uploaded images.

[0513] Input: User's food photo

[0514] Output: Parsed calorie and nutrient information

[0515] Step 4:

[0516] The server uses a generative AI model to generate appropriate meal suggestions based on the user's total calorie intake, nutrient intake, energy expenditure, and emotional state. The generative AI model generates meal menus that take into account the user's calorie and nutritional needs as well as their emotional state.

[0517] Input: User's total calorie intake, nutrient amounts, energy expenditure, emotional state

[0518] Output: Generated meal suggestions

[0519] Step 5:

[0520] The server compiles the generated meal suggestions into images and text format and sends them to the device. The device then displays the received meal suggestions on the user's smartphone or HMD at appropriate locations within the virtual store. As the user moves around the virtual store, notifications pop up at appropriate locations.

[0521] Input: Generated meal suggestions

[0522] Output: Notification and display in the user's virtual space

[0523] Step 6:

[0524] After consuming the suggested meal, the user takes a photo of the meal and uploads it to the system via their device. The device then sends the image to a cloud server. The server receives feedback from the user and stores the analysis results in a database using an image analysis engine. This improves the accuracy of meal suggestions for future visits.

[0525] Input: Food photos provided as feedback

[0526] Output: Database update to improve future meal suggestions

[0527] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0528] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0529] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0530] [Second embodiment]

[0531] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0532] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0533] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0534] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0535] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0536] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0537] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0538] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0539] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0540] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0541] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0542] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0543] A specific embodiment of a system for implementing the present invention will be described below. This system makes appropriate meal suggestions based on the user's individual health information and dietary data. The overall configuration of the system consists of steps to process the user's input information, make meal suggestions using a generative AI model based on the analysis results, and notify the user and collect feedback.

[0544] System Overview

[0545] Registering user information

[0546] Users log in to the system and enter personal data such as age, gender, height, weight, and health status. The terminal receives this information and sends it to the server, which then stores the received user information in a database.

[0547] Uploading meal information

[0548] Users take photos of the food they eat before and after eating and upload the image data to the system via their device. The device sends these images to the server, which receives the image data and transfers them to the image analysis engine.

[0549] Image analysis and calorie counting

[0550] The server uses an image analysis engine to identify the calories and macronutrients of the food from the uploaded image. The server then aggregates the calorie and nutrient information to calculate the total calories and macronutrient intake of the user that day. The calculation results are stored in a database.

[0551] Energy Expenditure Estimation

[0552] Users input their daily activity levels (e.g., commuting) into the system via their device. The device then sends the information to the server, which then estimates the user's total daily energy expenditure based on their activity level. The estimated results are stored in a database.

[0553] Generating meal suggestions

[0554] The server uses a generative AI model to create appropriate dinner suggestions based on the user's total calorie intake, nutrient content, and energy expenditure data. The generative AI model analyzes the user's calorie and nutritional needs and generates a meal menu based on them. This menu includes specific recipes, ingredients, quantities, and cooking methods. It also provides advice on convenience store substitutions.

[0555] Notification of proposal details

[0556] The server compiles the generated meal suggestions into images and text format and sends them to the device. The device displays the received meal suggestions on the user's screen. The user checks the suggested meals and prepares dinner accordingly.

[0557] Gathering feedback

[0558] After consuming the suggested meal, the user takes a photo of the meal they actually ate and uploads it to the system via their device. The device then sends the image to the server. The server receives feedback from the user and stores the analysis results in a database using an image analysis engine. This leads to improved accuracy of meal suggestions for future visits.

[0559] Specific examples

[0560] For example, let's say a 30-year-old female user A uses the system. After logging in, user A enters her age, gender, height, weight, and health condition, and eats toast and salad for breakfast, and bibimbap bento and pudding for lunch. This information is uploaded to the system as image data. The server uses an image analysis engine to calculate the calorie intake for that day, and the generative AI model makes a dinner suggestion based on user A's energy expenditure from her commute. Ultimately, a specific meal menu such as "grilled chicken salad, tofu vegetable soup, and brown rice" is suggested, and user A follows it. After eating, user A uploads an image of her actual meal to the system, which will help improve the accuracy of suggestions from next time onwards.

[0561] In this way, the present system effectively supports health management according to the individual needs of the user.

[0562] The processing flow will be explained below.

[0563] Step 1:

[0564] Registering user information

[0565] User: Logs into the system and enters personal data such as age, gender, height, weight, and health status.

[0566] Terminal: Receives input information and sends it to the server.

[0567] Server: Stores the received user information in a database.

[0568] Step 2:

[0569] Uploading meal information

[0570] User: Takes pictures of meals such as breakfast, lunch, and dinner on the device and uploads them to the system.

[0571] Terminal: Sends the captured image data to the server.

[0572] Server: Transfers the received image data to the image analysis engine.

[0573] Step 3:

[0574] Image analysis and calorie counting

[0575] Server: Uses an image analysis engine to identify the calories and key nutrients of a meal from an uploaded image.

[0576] Server: Aggregates the identified calorie and nutrient information and calculates the total calories and macronutrients consumed by the user for the day.

[0577] Server: Saves the calculation results in a database.

[0578] Step 4:

[0579] Energy Expenditure Estimation

[0580] User: Enters daily activity data (commuting, exercise, etc.) on the device and sends it to the system.

[0581] Terminal: Sends the entered activity information to the server.

[0582] Server: Estimates total daily energy expenditure based on the user's activity level.

[0583] Server: Saves the estimation results in a database.

[0584] Step 5:

[0585] Generating meal suggestions

[0586] Server: Uses a generative AI model to create appropriate meal suggestions based on the user's total calorie intake, nutrient content, and energy expenditure data.

[0587] Server: Runs the generative AI model, analyzes the user's calorie and nutritional needs, and generates a meal menu based on that.

[0588] Server: Generated meal suggestions include specific recipes, ingredients, serving sizes, and cooking methods, and also provide advice on convenience store substitutions.

[0589] Step 6:

[0590] Notification of proposal details

[0591] Server: The generated meal suggestions are compiled into images and text format and sent to the device.

[0592] Device: Displays the received meal suggestions on the user's screen.

[0593] User: Review the suggestions provided and prepare your meal accordingly.

[0594] Step 7:

[0595] Gathering feedback

[0596] User: After consuming the suggested meal, the user takes a photo of the menu item they actually ate with their device and uploads it to the system.

[0597] Terminal: Sends the captured image to the server.

[0598] Server: Receives user feedback and forwards it to the image analysis engine.

[0599] Server: The analysis results are saved in a database and used to improve the accuracy of future meal suggestions.

[0600] In this way, the system performs a series of processes based on the data input by the user and makes meal suggestions tailored to individual needs.

[0601] Example 1

[0602] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0603] Conventional health management systems have had problems with the accuracy of their balanced dietary recommendations and the inability to fully reflect the user's individual health and activity data. Furthermore, the dietary recommendations lacked specificity, making it difficult for users to actually consume the suggested meals. Furthermore, there was also the issue of not effectively utilizing feedback after consumption, which led to poor accuracy in future recommendations.

[0604] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0605] In this invention, the server includes: a means for inputting personal information such as the user's age, gender, height, weight, and health status; a means for uploading image data of the user's meals; a means for identifying the calories and nutrients of the meals using an image analysis engine; a means for estimating the user's energy expenditure; a means for generating meal suggestions based on the user's calorie and nutritional needs using a generative AI model; a means for notifying the user of the generated meal suggestions; a means for authenticating the user upon login; a means for transmitting and storing photo data on the server; a means for inputting the user's activity data and estimating energy expenditure; and a means for displaying the generated suggestions to the user in text and image format. This enables highly accurate meal suggestions that accurately reflect the user's individual health information and actual intake and activity data, providing more specific and easy-to-follow menu suggestions. Furthermore, by utilizing feedback, the accuracy of future suggestions can be improved.

[0606] "Means for inputting personal information such as the user's age, sex, height, weight, and health condition" refers to the means by which a user inputs personal information such as their age, sex, height, weight, and health condition into the system and transmits that information to the server.

[0607] The "means for uploading image data of meals taken by the user" refers to a means for the user to upload image data of meals taken using the camera function to the system and send the images to the server.

[0608] "Means for identifying calories and nutrients in meals using an image analysis engine" refers to means for analyzing uploaded images of meals and using image analysis software or algorithms to identify calories and nutrients in the meals.

[0609] "Means for estimating a user's energy expenditure" refers to means for calculating and estimating a user's daily energy expenditure based on the user's daily activity data and exercise data.

[0610] "Means for using a generative AI model to generate meal suggestions based on a user's calorie and nutritional needs" means means for using an artificial intelligence model to analyze a user's calorie intake and nutritional needs and generate appropriate meal menus and recipes.

[0611] The "means for notifying the user of the generated meal suggestions" refers to means for notifying and displaying the generated meal suggestions in text and image format on the user's terminal.

[0612] "Means for authenticating users when they log in" refers to the means for authenticating users using an ID and password, etc., when they access the system.

[0613] The "means for transmitting photo data and storing it on the server" refers to the means for transmitting photo data taken by the user to the server and storing it within the server.

[0614] "Means for inputting user activity data and estimating energy expenditure" refers to means for a user to input data about their own activities and exercise and estimate energy expenditure based on that data.

[0615] "Means for displaying the generated suggestions to the user in text and image format" refers to means for displaying the contents of the generated meal suggestions to the user in text and image format on the user's terminal and visually presenting them to the user.

[0616] The present invention is a system that makes appropriate meal recommendations based on a user's individual health information and dietary data. The system of the present invention consists of steps to process the user's input information, make meal recommendations using a generative AI model based on the analysis results, and notify the user and collect feedback. The specific system configuration is described below.

[0617] Registering user information

[0618] The user logs in to the system and enters personal data such as age, gender, height, weight, and health condition. The device collects this information and sends it to the server. The server stores the received data in a database. Through this process, the user's basic health information is accumulated in the system.

[0619] Uploading meal information

[0620] A user takes a photo of their meal using a mobile device or camera and uploads the image data. The device sends the captured image to a server. The server then forwards the received image data to an image analysis engine. This analysis engine can be, for example, image analysis software.

[0621] Image analysis and calorie counting

[0622] The server uses an image analysis engine to analyze the contents of the meal from the uploaded image and identify calories and key nutrients. The analysis can be performed using cloud-based image analysis software. The identified nutrient and calorie data is aggregated and stored in a database on the server.

[0623] Energy Expenditure Estimation

[0624] The user enters their daily activity (e.g., commuting) into the device. The device then sends the information to the server. The server analyzes the user's activity data and estimates their energy expenditure. This analysis can be performed using activity tracking software. The estimated results are stored in a database.

[0625] Generating meal suggestions

[0626] The server uses a generative AI model to generate appropriate meal suggestions based on the user's total calorie intake, nutrient content, and energy expenditure data. This process can be performed using generative AI model software, for example. The generated menu includes specific recipes, ingredients, serving sizes, and cooking methods.

[0627] For example, here's a prompt to input to a generative AI model:

[0628] "Please suggest a balanced dinner menu for a 30-year-old woman based on the following criteria: Breakfast: toast and salad, Lunch: boxed lunch and dessert, Daily exercise: light exercise during commute."

[0629] Notification of proposal details

[0630] The server then compiles the generated meal suggestions into images and text format and sends them to the device, which then displays them on the user's screen. The user can then review the suggested menu and follow the specific steps to execute it.

[0631] Gathering feedback

[0632] After consuming the suggested meal, the user takes a photo of the meal and uploads it back to the system. The device then sends the image to the server, which then receives it, analyzes it using an image analysis engine, and stores the results in a database. This feedback improves the accuracy of future meal suggestions.

[0633] As described above, this system can respond to the individual needs of users and support more effective health management.

[0634] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0635] Step 1: Register your user information

[0636] Input: The user logs into the system and enters personal information such as age, gender, height, weight, and health status.

[0637] Specific behavior: A user accesses the system using a web browser or mobile app. They enter their credentials on the login screen to log in to the system. They then enter the required personal data on the profile screen.

[0638] Output: The device sends the entered personal data to the server, which stores the received data in a database.

[0639] Step 2: Upload your meal information

[0640] Input: The user takes a photo of the food they have eaten and uploads the image data to the system.

[0641] Specific actions: The user takes a photo of their meal using the smartphone camera, selects the photo in the form to upload it to the system, and clicks the submit button.

[0642] Output: The terminal sends the image data to the server, which then transfers the received image data to the image analysis engine.

[0643] Step 3: Image analysis and calorie calculation

[0644] Input: The server analyzes the image data using an image analysis engine.

[0645] Specific operation: The server passes the received image data to the image analysis engine and sends an API request. The image analysis engine analyzes the contents of the meal and returns information about calories and nutrients.

[0646] Output: The server aggregates the calorie and nutrient information obtained from the image analysis engine and stores it in a database.

[0647] Step 4: Estimate energy consumption

[0648] Input: The user inputs their daily activity level into the device.

[0649] Specific actions: The user enters information about their daily activities (commuting, exercise, etc.) into a form on the device and clicks the submit button.

[0650] Output: The device sends the input activity data to the server, which uses activity tracking software or an API to estimate energy expenditure based on the input data and stores the results in a database.

[0651] Step 5: Generate meal suggestions

[0652] Input: The server retrieves data on total calorie intake, nutrient amounts, and energy expenditure from a database.

[0653] Specific operation: The server reads the user's latest health and dietary information from the database and sends prompt sentences to the generative AI model.

[0654] Example prompt: "For a 30-year-old woman, please suggest a balanced dinner menu based on the following criteria: Breakfast: toast and salad; Lunch: boxed lunch and dessert; Daily exercise: light exercise during commute."

[0655] Output: Take the generated meal suggestions and format them as a menu with specific recipes, ingredients, serving sizes, and cooking instructions.

[0656] Step 6: Notification of proposal

[0657] Input: Generated meal suggestions

[0658] Specific operation: The server sends the generated meal suggestions in text and image format to the device, which then displays them on the user's screen.

[0659] Output: The user confirms and implements the received meal suggestions.

[0660] Step 7: Gather feedback

[0661] Input: The user consumes the suggested meal and then takes and uploads a photo of the meal after consumption.

[0662] Specific operation: After eating a meal, the user takes a photo and uploads it to the system. The device then sends the photo data to the server.

[0663] Output: The server passes the received image back to the image analysis engine and stores the analysis results in a database, thereby improving the accuracy of future meal suggestions.

[0664] (Application example 1)

[0665] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0666] Many users today struggle to manage their health and daily diet. Even when they receive healthy dietary recommendations, they lack concrete advice on how to translate them into real-world purchasing behavior. This creates a growing need for a system that can efficiently select the right ingredients and meal plans.

[0667] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0668] In this invention, the server includes means for inputting information such as the user's age, sex, height, weight, and health condition, means for uploading image data of meals consumed by the user, means for identifying the calories and nutrients of the meals using an image analysis engine, means for estimating the user's energy expenditure, means for generating meal suggestions based on the user's calorie and nutritional needs using a generative AI model, means for notifying the user of the generated meal suggestions, and means for displaying the suggested ingredients and menus when the user visits a physical store. This makes it easier for users to reflect healthy meal suggestions in their actual purchasing behavior, enabling more effective health management.

[0669] "User information" refers to basic physical data about each individual user, such as age, gender, height, weight, and health status.

[0670] "Meal image data" refers to digitally saved photographs of meals eaten by a user.

[0671] An "image analysis engine" is software or algorithm that analyzes image data to identify the calories and nutrients of a meal.

[0672] "Energy expenditure" means the total amount of calories burned through a user's daily activities and exercise.

[0673] A "generative AI model" is a model that uses artificial intelligence to generate personalized meal suggestions based on user data.

[0674] "Meal Suggestions" means suggestions that provide specific meals or menus to be consumed based on the user's calorie and nutritional needs.

[0675] "Notification means" refers to a method or device for notifying the user of the generated meal suggestions.

[0676] "Physical stores" refer to physical stores that users visit and use, such as grocery stores and restaurants.

[0677] "Means for displaying ingredients and menu items" refers to methods or devices that visually display suggested ingredients and menu items when a user visits a physical store.

[0678] A specific embodiment of a system for implementing the present invention will be described below. This system makes appropriate meal suggestions based on a user's individual health information and dietary data, and supports purchasing behavior, particularly in physical stores. This system involves inputting user information, analyzing dietary image data, estimating energy consumption, generating and notifying meal suggestions, and displaying ingredients and menus in physical stores.

[0679] System Overview

[0680] (1) User information registration:

[0681] Users log in to the system and enter personal data such as age, sex, height, weight, health condition, etc. through a terminal. The terminal sends this information to the server, which stores it in a database.

[0682] (2) Uploading meal information:

[0683] Users take photos of the food they eat and upload the image data to the system via their device, which then sends the images to the server, which then forwards them to the image analysis engine.

[0684] (3) Image analysis and calorie calculation:

[0685] The server uses an image analysis engine to identify the calories and key nutrients of the food from the uploaded image, and the calculations are stored in a database.

[0686] (4) Energy consumption estimation:

[0687] Users input their daily activity levels (e.g., commuting) into the system via their devices. The server estimates the total daily energy expenditure based on the user's activity level and stores the estimated results in a database.

[0688] (5) Meal suggestion generation:

[0689] Based on the user's total calorie intake, nutrient content, and energy expenditure data, the server uses a generative AI model to create appropriate meal suggestions, including specific recipes, ingredients, serving sizes, and cooking methods, and also provides advice on substitute ingredients from convenience stores and supermarkets.

[0690] (6) Notification of proposal:

[0691] The server sends the generated meal suggestions to the terminal, which displays them on the user's screen. The user can then confirm the suggested meals and act accordingly.

[0692] (7) Display in physical stores:

[0693] When a user visits a physical store, they use their device to scan the store's QR code, which allows the server to display ingredients and menu items to purchase in real time based on the user's meal suggestions.

[0694] (8) Feedback Collection:

[0695] After consuming the suggested meal, the user takes a photo of the meal and uploads it to the system via their device. The server receives the feedback and stores the analysis results in a database using an image analysis engine. This improves the accuracy of meal suggestions for future visits.

[0696] Examples:

[0697] For example, let's say User A, a 30-year-old woman, uses this system. After logging in, User A enters her age, gender, height, weight, and health status, and eats toast and salad for breakfast, and bibimbap bento and pudding for lunch. This information is uploaded to the system as image data. The server uses an image analysis engine to calculate the calorie intake for that day, and a generative AI model uses the energy expenditure from User A's commute to suggest dinner. Ultimately, a specific meal menu such as "grilled chicken salad, tofu vegetable soup, and brown rice" is suggested. User A goes to a nearby physical store and scans the store's QR code to check the ingredients and menu items to purchase on her smartphone. She then purchases the ingredients based on the suggestions and cooks. After eating, User A uploads an image of the menu she actually ate back to the system, contributing to improving the accuracy of suggestions from next time onwards.

[0698] Example prompt sentence:

[0699] User ID: 12345

[0700] Remaining calorie intake: 600

[0701] Suggest a suitable dinner menu based on the following criteria:

[0702] 1. Calories: 600 kcal or less

[0703] 2. Health status: Healthy adult female

[0704] 3. Nutritional balance: high protein, low carbohydrate

[0705] Suggested menu items should include specific ingredients, portions, and cooking methods.

[0706] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0707] Step 1:

[0708] Users log in to the device and enter information such as their age, gender, height, weight, and health condition. The device receives this information and sends it to the server, which then stores the received user information in a database.

[0709] Input: User information (age, gender, height, weight, health condition)

[0710] Output: User information stored in the database

[0711] Specific operation: When a user fills in information in the application's input form and presses the "Save" button, the terminal sends it to the server, which then stores it in the database.

[0712] Step 2:

[0713] Users take photos of the food they eat and upload the image data to the system via their device. The device sends these images to the server, which then forwards them to the image analysis engine.

[0714] Input: Food image data

[0715] Output: Image data sent to the server

[0716] How it works: When a user takes a photo of their meal and presses the "upload" button in the application, the device sends the image data to the server, which then passes it to the image analysis engine.

[0717] Step 3:

[0718] The server uses an image analysis engine to identify the calories and key nutrients of the food from the uploaded image, and the analysis results are stored in a database.

[0719] Input: Food image data

[0720] Output: Calorie and macronutrient data for the meal

[0721] Specific operation: The image analysis engine inputs image data into an algorithm and performs analysis. The analysis results are stored in a database via a server.

[0722] Step 4:

[0723] Users input their daily activity levels (e.g., commuting) into the system via their devices. The server receives the information and estimates the user's total daily energy expenditure based on their activity level. This estimation is then stored in a database.

[0724] Input: Activity data (commuting, exercise, etc.)

[0725] Output: Estimated energy consumption

[0726] How it works: After a user fills out the application's activity recording form and presses the "Submit" button, the device sends the data to the server, which then uses it to calculate activity levels.

[0727] Step 5:

[0728] The server uses a generative AI model to generate appropriate meal suggestions based on the user's total calorie intake, nutrient content, and energy expenditure data, including specific recipes, ingredients, serving sizes, and cooking methods.

[0729] Input: Total calories, nutrient content, energy expenditure

[0730] Output: Specific meal suggestions (recipe, ingredients, portions, cooking method)

[0731] How it works: The server inputs this data into a generative AI model to generate optimal meal suggestions for the user.

[0732] Step 6:

[0733] The server sends the generated meal suggestions to the terminal, which displays them on the user's screen. The user can then confirm the suggested meals and act accordingly.

[0734] Input: Meal suggestion data

[0735] Output: Meal suggestions displayed on the user's device

[0736] Specific operation: The server generates meal suggestions and sends them to the device, which receives them and displays them on the application.

[0737] Step 7:

[0738] When a user visits a physical store, they use their device to scan the store's QR code, and the server displays ingredients and menu items to purchase in real time based on the user's meal suggestions.

[0739] Input: QR code from physical store, suggested meal data

[0740] Output: List of ingredients and menu items to purchase

[0741] How it works: The user scans a QR code at a store and sends the information to the server, which then generates a shopping list based on the user's meal suggestions and displays it on the device.

[0742] Step 8:

[0743] After consuming the suggested meal, the user takes a photo of the meal and uploads it to the system via their device. The server receives the feedback and stores the analysis results in a database using an image analysis engine.

[0744] Input: Image data of the menu you ate

[0745] Output: Parsed feedback data

[0746] How it works: The user takes a photo of the menu item they ate and uploads it to the server through the application. The server passes the data to the image analysis engine and stores the analysis results in a database.

[0747] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0748] A specific embodiment of a system for implementing the present invention will be described below. This system recognizes the user's emotional state in addition to their individual health information and dietary data to make meal suggestions. The overall configuration of the system consists of the steps of processing the user's input information, making meal suggestions based on the analysis results using a generative AI model and emotion engine, and notifying the user and collecting feedback.

[0749] System Overview

[0750] Registering user information

[0751] Users log in to the system and enter personal data such as age, gender, height, weight, and health status. The terminal receives this information and sends it to the server, which then stores the received user information in a database.

[0752] Collecting Emotional Data

[0753] Users can input data about their physical condition and mood, or communicate their emotions to the system using facial and voice recognition technology. The device receives this emotional data and sends it to the server, which then transfers it to the emotion engine.

[0754] Uploading meal information

[0755] Users take photos of the food they eat before and after eating and upload the image data to the system via their device. The device sends these images to the server, which receives the image data and transfers them to the image analysis engine.

[0756] Image analysis and calorie counting

[0757] The server uses an image analysis engine to identify the calories and macronutrients of the food from the uploaded image. The server then aggregates the calorie and nutrient information to calculate the total calories and macronutrient intake of the user that day. The calculation results are stored in a database.

[0758] Energy Expenditure Estimation

[0759] Users input their daily activity levels (e.g., commuting) into the system via their device. The device then sends the information to the server, which then estimates the user's total daily energy expenditure based on their activity level. The estimated results are stored in a database.

[0760] Emotion analysis and reflection

[0761] The server uses an emotion engine to analyze the user's current emotional state from the emotion data. The analysis results are used to adjust subsequent meal suggestions. For example, if the user is feeling stressed, the suggestions will include ingredients with a relaxation effect.

[0762] Generating meal suggestions

[0763] The server uses a generative AI model to create appropriate meal suggestions based on the user's total calorie intake, nutrient content, energy expenditure, and emotional state. The generative AI model generates meal menus that take into account the user's calorie and nutritional needs as well as their emotional state. The menus include specific recipes, ingredients, serving sizes, and cooking methods. It also provides advice on convenience store substitutions.

[0764] Notification of proposal details

[0765] The server compiles the generated meal suggestions into images and text format and sends them to the device. The device displays the received meal suggestions on the user's screen. The user checks the displayed suggestions and prepares dinner accordingly.

[0766] Gathering feedback

[0767] After consuming the suggested meal, the user takes a photo of the meal they actually ate and uploads it to the system via their device. The device then sends the image to the server. The server receives feedback from the user and stores the analysis results in a database using an image analysis engine. This leads to improved accuracy of meal suggestions for future visits.

[0768] Specific examples

[0769] For example, let's say User B, a 30-year-old woman, uses the system. After logging in, User B enters her age, gender, height, weight, and health condition, and also enters emotional data about the stress she is currently feeling. She uploads image data to the system, showing that she had oatmeal and fruit for breakfast and a sandwich and salad for lunch. The server uses an image analysis engine to calculate the calories consumed that day, and uses a generative AI model to suggest dinner based on User B's energy expenditure during her commute. Additionally, based on her emotional state analyzed by the emotion engine, it suggests relaxing herbal tea and meals rich in B vitamins.

[0770] In this way, the system provides appropriate and effective health management and dietary recommendations based on the user's individual needs and emotional state.

[0771] The processing flow will be explained below.

[0772] Step 1:

[0773] Registering user information

[0774] User: Logs into the system and enters personal data such as age, gender, height, weight, and health status.

[0775] Terminal: Receives input information and sends it to the server.

[0776] Server: Stores the received user information in a database.

[0777] Step 2:

[0778] Collecting Emotional Data

[0779] User: Enters data about physical condition and mood into the device, and communicates emotions to the system using facial and voice recognition technology.

[0780] Terminal: Receives these emotion data and sends them to the server.

[0781] Server: Transfers the received emotion data to the emotion engine.

[0782] Step 3:

[0783] Uploading meal information

[0784] User: Takes pictures of meals such as breakfast, lunch, and dinner on the device and uploads them to the system.

[0785] Terminal: Sends the captured image data to the server.

[0786] Server: Transfers the received image data to the image analysis engine.

[0787] Step 4:

[0788] Image analysis and calorie counting

[0789] Server: Uses an image analysis engine to identify the calories and key nutrients of a meal from an uploaded image.

[0790] Server: Aggregates the identified calorie and nutrient information and calculates the total calories and macronutrients consumed by the user for the day.

[0791] Server: Saves the calculation results in a database.

[0792] Step 5:

[0793] Energy Expenditure Estimation

[0794] User: Enters daily activity data (commuting, exercise, etc.) on the device and sends it to the system.

[0795] Terminal: Sends the entered activity information to the server.

[0796] Server: Estimates total daily energy expenditure based on the user's activity level.

[0797] Server: Saves the estimation results in a database.

[0798] Step 6:

[0799] Emotion analysis and reflection

[0800] Server: Uses the emotion engine to analyze the user's current emotional state from the emotion data.

[0801] Server: The analysis results are stored and used to adjust future meal suggestions. For example, if the user is feeling stressed, the suggestions may include foods with a relaxing effect.

[0802] Step 7:

[0803] Generating meal suggestions

[0804] Server: Uses a generative AI model to create appropriate meal suggestions based on the user's total calorie intake, nutrient amounts, energy expenditure, and emotional state data.

[0805] Server: Runs the generative AI model, analyzes the user's calorie and nutritional needs, and generates a meal menu based on that.

[0806] Server: Generated meal suggestions include specific recipes, ingredients, serving sizes, and cooking methods, and also provide advice on convenience store substitutions.

[0807] Step 8:

[0808] Notification of proposal details

[0809] Server: The generated meal suggestions are compiled into images and text format and sent to the device.

[0810] Device: Displays the received meal suggestions on the user's screen.

[0811] User: Review the suggestions provided and prepare your meal accordingly.

[0812] Step 9:

[0813] Collecting feedback

[0814] User: After consuming the suggested meal, the user takes a photo of the menu item they actually ate with their device and uploads it to the system.

[0815] Terminal: Sends the captured image to the server.

[0816] Server: Receives user feedback and forwards it to the image analysis engine.

[0817] Server: The analysis results are saved in a database and used to improve the accuracy of future meal suggestions.

[0818] In this way, the system processes user input and emotional data to provide personalized meal recommendations, and can continuously improve the accuracy of these recommendations through feedback.

[0819] Example 2

[0820] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0821] In today's busy lifestyles, it is difficult for users to receive appropriate dietary recommendations based on their health status and emotions. Furthermore, there is no system that can provide dietary recommendations that take into account the user's emotional state, rather than just calories and nutrients. This leads to a delay in effective health management and dietary improvement.

[0822] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0823] In this invention, the server includes means for inputting information such as the user's age, sex, height, weight, and health condition, means for uploading image data of meals consumed by the user, means for identifying the calories and nutrients of the meals using an image analysis engine, means for estimating the user's energy expenditure, means for collecting the user's emotion data and analyzing it using an emotion engine, means for generating meal suggestions based on the user's calorie and nutritional needs and emotional state using a generative AI model, and means for notifying the user of the generated meal suggestions. This enables appropriate meal suggestions based on the user's health condition and emotions.

[0824] "User" refers to an individual who uses this system.

[0825] "Age" refers to the number of years since the user was born.

[0826] "Gender" refers to a user's biological or social sex.

[0827] "Height" refers to the vertical length of the user's body.

[0828] "Weight" refers to the mass of a user's body.

[0829] "Health Status" refers to the state of a User's physical and mental health.

[0830] "Information" refers to data such as the user's age, gender, height, weight, and health status.

[0831] "Means" refers to a method or device for achieving a particular purpose.

[0832] "Meal image data" refers to image files of meals consumed by the user.

[0833] "Upload" refers to the act of sending data from a user's device to a server.

[0834] "Image analysis engine" refers to software or hardware for analyzing image data and extracting specific information.

[0835] "Calories" refers to the amount of energy a food contains.

[0836] "Nutrients" are substances contained in food that are necessary for the growth and maintenance of health of the body.

[0837] "Energy Expenditure" refers to the total amount of energy consumed by a user's daily activities.

[0838] "Emotional data" refers to data related to the user's physical condition and mood.

[0839] "Emotion engine" refers to software or hardware for analyzing emotional data to identify a user's emotional state.

[0840] A "generative AI model" refers to an artificial intelligence system that generates appropriate meal suggestions based on user information.

[0841] "Notification" refers to the act of informing the user of the generated meal suggestions.

[0842] A "system" refers to a set of multiple means or devices that function in conjunction with one another.

[0843] A specific embodiment of a system for implementing the present invention will be described below. This system recognizes the user's emotional state as well as their individual health information and dietary data to make meal suggestions. This system processes the user's input information, makes meal suggestions based on the analysis results using a generative AI model and an emotion engine, and notifies the user and collects feedback.

[0844] Registering user information

[0845] Users log in to the system and enter personal data such as age, gender, height, weight, and health status. The terminal receives this information and sends it to the server, which then stores the received user information in a database.

[0846] Collecting Emotional Data

[0847] Users can input data about their physical condition and mood, or communicate their emotions to the system using facial and voice recognition technology. The device receives this emotional data and sends it to the server, which then transfers it to the emotion engine.

[0848] Uploading meal information

[0849] Users take photos of the food they eat before and after eating and upload the image data to the system via their device. The device sends these images to the server, which receives the image data and transfers them to the image analysis engine.

[0850] Image analysis and calorie counting

[0851] The server uses an image analysis engine to identify the calories and macronutrients of the food from the uploaded image. The server then aggregates the calorie and nutrient information to calculate the total calories and macronutrient intake of the user that day. The calculation results are stored in a database.

[0852] Energy Expenditure Estimation

[0853] Users input their daily activity levels (e.g., commuting) into the system via their device. The device then sends the information to the server, which then estimates the user's total daily energy expenditure based on their activity level. The estimated results are stored in a database.

[0854] Emotion analysis and reflection

[0855] The server uses an emotion engine to analyze the user's current emotional state from the emotion data. The analysis results are used to adjust subsequent meal suggestions. For example, if the user is feeling stressed, the suggestions will include ingredients with a relaxation effect.

[0856] Generating meal suggestions

[0857] The server uses a generative AI model to create appropriate meal suggestions based on the user's total calorie intake, nutrient content, energy expenditure, and emotional state. The generative AI model generates meal menus that take into account the user's calorie and nutritional needs as well as their emotional state. The menus include specific recipes, ingredients, serving sizes, and cooking methods. It also provides advice on convenience store substitutions.

[0858] Notification of proposal details

[0859] The server compiles the generated meal suggestions into images and text format and sends them to the device. The device displays the received meal suggestions on the user's screen. The user checks the displayed suggestions and prepares dinner accordingly.

[0860] Gathering feedback

[0861] After consuming the suggested meal, the user takes a photo of the meal they actually ate and uploads it to the system via their device. The device then sends the image to the server. The server receives feedback from the user and stores the analysis results in a database using an image analysis engine. This leads to improved accuracy of meal suggestions for future visits.

[0862] Specific examples

[0863] For example, suppose a 30-year-old female user uses the system. After logging in, the user enters her age, gender, height, weight, and health condition, and also enters emotional data about the stress she is currently experiencing. She uploads image data of her breakfast of oatmeal and fruit, and lunch of a sandwich and salad, to the system. The server uses an image analysis engine to calculate the calorie intake for that day, and uses a generative AI model to suggest dinner based on the user's energy expenditure from commuting. Based on the emotional state analyzed by the emotion engine, the system also suggests relaxing herbal teas and meals rich in B vitamins. In this way, the system provides appropriate and effective health management and dietary suggestions based on the user's individual needs and emotional state.

[0864] Prompt Sentence Examples

[0865] "A 30-year-old female user uses the system and enters her age, gender, height, weight, and health status. The user is currently stressed and ate oatmeal and fruit for breakfast, and a sandwich and salad for lunch. Please generate dinner suggestions taking into account her energy expenditure from her commute."

[0866] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0867] Step 1: Register your user information

[0868] Users log in to the system and enter personal data such as age, gender, height, weight, and health status.

[0869] Input: Age, gender, height, weight, health condition

[0870] Output: User information data (e.g., JSON format)

[0871] The terminal receives the entered information and transmits the data to the server.

[0872] Specific operation: The terminal encodes the input data into JSON format and sends it to the server using the HTTPS protocol.

[0873] The server stores the received data in a database.

[0874] Input: User information data (e.g., JSON format)

[0875] Output: User information record in database

[0876] Specific operation: The server analyzes the received data and performs an insert operation on a specific table in the database.

[0877] Step 2: Collecting emotion data

[0878] Users input data about their physical condition and mood, or communicate their emotions to the system using facial and voice recognition technology.

[0879] Input: Physical condition, mood data, or emotional data

[0880] Output: Emotion data (numerical and text format)

[0881] The terminal receives this data and transmits it to the server.

[0882] Specific operation: The device generates an API request to send emotion data to the server in real time.

[0883] The server transfers the received emotion data to the emotion engine for analysis.

[0884] Input: Received emotion data

[0885] Output: Parsed emotional state data

[0886] Specific operation: The server passes the emotion data to the emotion engine's API and receives the analysis results.

[0887] Step 3: Upload your meal information

[0888] Users take photos of the food they eat before and after eating and upload the image data to the system.

[0889] Input: Food image

[0890] Output: Image data file

[0891] The terminal transmits the image data to the server.

[0892] Specific operation: The device compresses the captured image and generates an API request to upload it to the server.

[0893] The server receives the image data and transfers it to the image analysis engine.

[0894] Input: Image data file

[0895] Output: Image analysis results (calories, nutrient data)

[0896] Specific operation: The server calls the API of the image analysis engine and sends a request to analyze the image data.

[0897] Step 4: Image analysis and calorie calculation

[0898] The server uses an image analysis engine to identify the calories and key nutrients of a meal from the uploaded image.

[0899] Input: Food image data

[0900] Output: Calorie information, nutrient information

[0901] How it works: The image analysis engine applies algorithms to recognize ingredients in an image and identify the calorie and nutritional information for each ingredient.

[0902] The server compiles the results and calculates the user's total calorie and nutrient intake.

[0903] Input: Calorie information and nutrient information for each meal

[0904] Output: Total calorie intake and macronutrient data

[0905] What it does: The server uses an aggregation algorithm to add up the calories and nutrients for each meal.

[0906] The calculation results are saved in a database.

[0907] Input: Total calorie intake and amount of major nutrients

[0908] Output: Accumulated data in the database

[0909] Specific operation: The server performs an insert operation on the results into a specific table in the database.

[0910] Step 5: Estimate energy consumption

[0911] Users enter their daily activity levels into the system.

[0912] Input: Activity data (commuting, exercise, etc.)

[0913] Output: Activity log

[0914] The terminal transmits the information to the server.

[0915] Specific operation: The device encodes the activity data and sends an API request to the server.

[0916] The server estimates total energy expenditure based on the user's activity level.

[0917] Input: Activity data

[0918] Output: Total energy consumption data

[0919] Specific operation: The server applies a calculation algorithm based on the activity data to estimate energy expenditure.

[0920] The estimation results are saved in a database.

[0921] Input: Total energy consumption data

[0922] Output: Energy consumption records in a database

[0923] Specific operation: The server inserts the estimation results into the database.

[0924] Step 6: Analyze and reflect on emotions

[0925] The server uses an emotion engine to parse the user's current emotional state from the emotion data.

[0926] Input: Emotion data

[0927] Output: Parsed emotional state

[0928] Specific operation: The emotion engine analyzes the emotion data and generates a quantified emotional state.

[0929] Configure settings to reflect the analysis results in meal suggestions.

[0930] Input: Parsed emotional state

[0931] Output: Adjusted meal suggestion data

[0932] What it does: The server applies an algorithm that adjusts the suggestions depending on the emotional state.

[0933] Step 7: Generate meal suggestions

[0934] The server uses a generative AI model to create appropriate meal suggestions based on the user's total calorie intake, nutrient amounts, energy expenditure, and emotional state data.

[0935] Input: User's total calorie intake, nutritional information, energy expenditure, emotional state data

[0936] Output: Generated meal suggestions

[0937] Specific operation: The generative AI model generates an appropriate meal menu based on the prompt text.

[0938] The generated suggestions include specific recipes, ingredients, quantities, and cooking methods.

[0939] Input: Generated meal suggestions

[0940] Output: Detailed meal menu (including recipe, ingredients, portions, and cooking method)

[0941] What it does: The server formats the results of the generative AI model and compiles them into a detailed menu.

[0942] Step 8: Notification of proposal

[0943] The server compiles the generated meal suggestions into image and text format and sends them to the terminal.

[0944] Input: Detailed meal menu

[0945] Output: Notification format

[0946] Specific operation: The server converts the menu information into image and text format and sends it to the terminal.

[0947] The device displays the received meal suggestions on the user's screen.

[0948] Input: Notification Format

[0949] Output: The suggestions that are displayed on the user's screen

[0950] Specific operation: The terminal obtains the notification data and displays it on the user interface.

[0951] The user reviews the displayed suggestions and prepares dinner accordingly.

[0952] Step 9: Gather feedback

[0953] After consuming the suggested meal, the user takes a photo of the meal they actually ate and uploads it to the system.

[0954] Input: Food image

[0955] Output: Uploaded image data

[0956] The terminal transmits the image data to the server again.

[0957] Specific operation: The device generates an API request to compress the image and send it to the server.

[0958] The server analyzes the feedback using an image analysis engine and stores the results in a database.

[0959] Input: Uploaded image data

[0960] Output: Analysis results

[0961] Specific operation: The server calls the API of the image analysis engine again and inserts the feedback results into the database.

[0962] (Application example 2)

[0963] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0964] Conventional health management systems often do not provide meal suggestions that take into account the user's emotional state, and comprehensive health management, including the user's mental health, has not been fully realized. Furthermore, virtual stores are unable to provide meal suggestions based on real-time health information, making it difficult for users to efficiently select ingredients and recipes that are suitable for them. This makes it difficult to maintain and improve users' health through appropriate diets.

[0965] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0966] In this invention, the server includes: means for inputting information such as the user's age, sex, height, weight, and health condition; means for uploading image data of the user's meals; means for identifying the calories and nutrients of the meals using an image analysis engine; means for estimating the user's energy expenditure; means for collecting and analyzing the user's emotional data; means for generating meal suggestions based on the user's calorie and nutritional needs and emotional state using a generative AI model; means for notifying the user of the generated meal suggestions; and means for providing meal suggestions in real time in a virtual store based on the user's health information, dietary data, and emotional state. This enables comprehensive health management that takes the user's emotional state into consideration and enables the selection of appropriate ingredients and recipes in real time in the virtual store.

[0967] "User Information" refers to personal information about a user, such as their age, gender, height, weight, and health status.

[0968] "Image data" refers to photographic data of the food consumed by the user.

[0969] The "image analysis engine" is software that identifies the calories and nutrients of meals from uploaded image data.

[0970] "Energy expenditure" is the amount of energy a user expends through daily activities.

[0971] "Emotion data" is data that indicates information about the user's physical condition and mood, and is obtained using facial recognition technology or voice recognition technology.

[0972] A "generative AI model" is an artificial intelligence model that generates meal suggestions based on a user's calorie and nutritional needs and emotional state.

[0973] "Meal suggestions" are suggestions such as specific meal menus, recipes, ingredients, portions, cooking methods, etc., that are proposed to users.

[0974] A "virtual store" is a digital platform that allows users to shop in a virtual space.

[0975] "Real-time" refers to data processing and information provision occurring immediately or nearly simultaneously.

[0976] "Notification" is the act or mechanism of communicating generated meal suggestions to a user.

[0977] "Feedback" means providing the system with information about the food the user has eaten, which helps improve the accuracy of the next suggestion.

[0978] MODE FOR CARRYING OUT THE INVENTION

[0979] A detailed description of a system for realizing the present invention is given below. This system makes meal suggestions in real time in a virtual store based on the user's individual health information, dietary data, and emotional state.

[0980] System Overview

[0981] Registering user information

[0982] After accessing the virtual store, users use their smartphones or head-mounted displays (HMDs) to input personal information such as age, gender, height, weight, and health status. The devices receive this information and send it to a cloud server, which then stores the received user information in a database.

[0983] Collecting Emotional Data

[0984] Users can input data about their physical condition and mood using a camera or microphone installed on their smartphone or HMD, or communicate their emotions to the system using facial or voice recognition technology. The device receives this emotional data and sends it to a cloud server. The server then analyzes the received emotional data using an emotion engine to determine the user's current emotional state.

[0985] Uploading meal information

[0986] Users take photos of the meals they have eaten over the past 48 hours with their smartphones and upload them to the system. The device sends these images to a cloud server. The server receives the image data and transfers it to an image analysis engine. The image analysis engine identifies the calories and nutrients of the meals from the uploaded images.

[0987] Generating meal suggestions

[0988] The server uses a generative AI model based on the user's total calorie intake, nutrient content, energy expenditure, and emotional state data to generate appropriate meal suggestions. The generative AI model considers the user's calorie and nutritional needs as well as their emotional state to generate a meal menu that includes specific recipes, ingredients, serving sizes, and cooking methods.

[0989] Notification of proposal details

[0990] The server compiles the generated meal suggestions into images and text format and sends them to the device. The device then displays the received meal suggestions on the user's smartphone or HMD at appropriate locations within the virtual store. As the user moves around the virtual store, notifications pop up at appropriate locations.

[0991] Gathering feedback

[0992] After consuming the suggested meal, the user takes a photo of the meal and uploads it to the system via their device. The device then sends the image to a cloud server. The server receives feedback from the user and stores the analysis results in a database using an image analysis engine. This improves the accuracy of meal suggestions for future visits.

[0993] Hardware and Software Used

[0994] Smartphone: Use an Android or iOS device.

[0995] Head-mounted display (HMD): Use Oculus Rift, HTC Vive, etc.

[0996] Cloud server: Use cloud platforms such as Google Cloud, AWS, and Azure.

[0997] Emotion recognition technology: Uses technologies such as OpenCV and TensorFlow.

[0998] Generative AI models: Utilize generative AI such as OpenAI GPT-3.

[0999] Specific examples

[1000] For example, a user visits a virtual store and enters their personal information using a smartphone. Emotional data is collected using facial recognition with a camera installed in the HMD, and data on physical condition and mood is collected. After the user uploads photos of the meals they have eaten in the past 48 hours, the cloud server uses an image analysis engine to identify calories and nutrients, and then uses a generative AI model to make meal recommendations.

[1001] Example prompt sentence:

[1002] Based on their health, users need 2000 kcal of calories, 50g protein, 70g fat, and 250g carbohydrates.

[1003] Please suggest a dish using the following ingredients and method:

[1004] 1) Breakfast: Oatmeal and fruit

[1005] 2) Lunch: Sandwich and salad

[1006] 3) Dinner: Chicken steak and vegetables

[1007] The user's emotional state is stress.

[1008] This allows users to select appropriate ingredients and recipes in real time in a virtual store, enabling comprehensive health management that takes into account their emotional state.

[1009] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1010] Step 1:

[1011] Users access the virtual store and use their smartphone or HMD to enter personal information such as age, gender, height, weight, and health status. This information is then sent from the device to a cloud server, which then stores the received information in a database.

[1012] Input: User's age, gender, height, weight, health status

[1013] Output: User information stored in the database

[1014] Step 2:

[1015] Users input data about their physical condition and mood into the system using a camera and microphone installed on their smartphone or HMD. The device receives this emotional data and sends it to a cloud server. The server then uses an emotion engine to analyze the received emotional data and identify the user's current emotional state.

[1016] Input: Data about the user's physical condition and mood

[1017] Output: Parsed user emotional state

[1018] Step 3:

[1019] Users take photos of the meals they have eaten over the past 48 hours with their smartphones and upload them to the system. The device sends these image data to a cloud server. The server receives the image data and transfers it to an image analysis engine. The image analysis engine identifies the calories and nutrients of the meals from the uploaded images.

[1020] Input: User's food photo

[1021] Output: Parsed calorie and nutrient information

[1022] Step 4:

[1023] The server uses a generative AI model to generate appropriate meal suggestions based on the user's total calorie intake, nutrient intake, energy expenditure, and emotional state. The generative AI model generates meal menus that take into account the user's calorie and nutritional needs as well as their emotional state.

[1024] Input: User's total calorie intake, nutrient amounts, energy expenditure, emotional state

[1025] Output: Generated meal suggestions

[1026] Step 5:

[1027] The server compiles the generated meal suggestions into images and text format and sends them to the device. The device then displays the received meal suggestions on the user's smartphone or HMD at appropriate locations within the virtual store. As the user moves around the virtual store, notifications pop up at appropriate locations.

[1028] Input: Generated meal suggestions

[1029] Output: Notification and display in the user's virtual space

[1030] Step 6:

[1031] After consuming the suggested meal, the user takes a photo of the meal and uploads it to the system via their device. The device then sends the image to a cloud server. The server receives feedback from the user and stores the analysis results in a database using an image analysis engine. This improves the accuracy of meal suggestions for future visits.

[1032] Input: Food photos provided as feedback

[1033] Output: Database update to improve future meal suggestions

[1034] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1035] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1036] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1037] [Third embodiment]

[1038] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1039] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1040] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1041] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1042] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1043] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1044] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1045] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1046] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1047] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1048] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1049] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1050] A specific embodiment of a system for implementing the present invention will be described below. This system makes appropriate meal suggestions based on the user's individual health information and dietary data. The overall configuration of the system consists of steps to process the user's input information, make meal suggestions using a generative AI model based on the analysis results, and notify the user and collect feedback.

[1051] System Overview

[1052] Registering user information

[1053] Users log in to the system and enter personal data such as age, gender, height, weight, and health status. The terminal receives this information and sends it to the server, which then stores the received user information in a database.

[1054] Uploading meal information

[1055] Users take photos of the food they eat before and after eating and upload the image data to the system via their device. The device sends these images to the server, which receives the image data and transfers them to the image analysis engine.

[1056] Image analysis and calorie counting

[1057] The server uses an image analysis engine to identify the calories and macronutrients of the food from the uploaded image. The server then aggregates the calorie and nutrient information to calculate the total calories and macronutrient intake of the user that day. The calculation results are stored in a database.

[1058] Energy Expenditure Estimation

[1059] Users input their daily activity levels (e.g., commuting) into the system via their device. The device then sends the information to the server, which then estimates the user's total daily energy expenditure based on their activity level. The estimated results are stored in a database.

[1060] Generating meal suggestions

[1061] The server uses a generative AI model to create appropriate dinner suggestions based on the user's total calorie intake, nutrient content, and energy expenditure data. The generative AI model analyzes the user's calorie and nutritional needs and generates a meal menu based on them. This menu includes specific recipes, ingredients, quantities, and cooking methods. It also provides advice on convenience store substitutions.

[1062] Notification of proposal details

[1063] The server compiles the generated meal suggestions into images and text format and sends them to the device. The device displays the received meal suggestions on the user's screen. The user checks the suggested meals and prepares dinner accordingly.

[1064] Gathering feedback

[1065] After consuming the suggested meal, the user takes a photo of the meal they actually ate and uploads it to the system via their device. The device then sends the image to the server. The server receives feedback from the user and stores the analysis results in a database using an image analysis engine. This leads to improved accuracy of meal suggestions for future visits.

[1066] Specific examples

[1067] For example, let's say a 30-year-old female user A uses the system. After logging in, user A enters her age, gender, height, weight, and health condition, and eats toast and salad for breakfast, and bibimbap bento and pudding for lunch. This information is uploaded to the system as image data. The server uses an image analysis engine to calculate the calorie intake for that day, and the generative AI model makes a dinner suggestion based on user A's energy expenditure from her commute. Ultimately, a specific meal menu such as "grilled chicken salad, tofu vegetable soup, and brown rice" is suggested, and user A follows it. After eating, user A uploads an image of her actual meal to the system, which will help improve the accuracy of suggestions from next time onwards.

[1068] In this way, the present system effectively supports health management according to the individual needs of the user.

[1069] The processing flow will be explained below.

[1070] Step 1:

[1071] Registering user information

[1072] User: Logs into the system and enters personal data such as age, gender, height, weight, and health status.

[1073] Terminal: Receives input information and sends it to the server.

[1074] Server: Stores the received user information in a database.

[1075] Step 2:

[1076] Uploading meal information

[1077] User: Takes pictures of meals such as breakfast, lunch, and dinner on the device and uploads them to the system.

[1078] Terminal: Sends the captured image data to the server.

[1079] Server: Transfers the received image data to the image analysis engine.

[1080] Step 3:

[1081] Image analysis and calorie counting

[1082] Server: Uses an image analysis engine to identify the calories and key nutrients of a meal from an uploaded image.

[1083] Server: Aggregates the identified calorie and nutrient information and calculates the total calories and macronutrients consumed by the user for the day.

[1084] Server: Saves the calculation results in a database.

[1085] Step 4:

[1086] Energy Expenditure Estimation

[1087] User: Enters daily activity data (commuting, exercise, etc.) on the device and sends it to the system.

[1088] Terminal: Sends the entered activity information to the server.

[1089] Server: Estimates total daily energy expenditure based on the user's activity level.

[1090] Server: Saves the estimation results in a database.

[1091] Step 5:

[1092] Generating meal suggestions

[1093] Server: Uses a generative AI model to create appropriate meal suggestions based on the user's total calorie intake, nutrient content, and energy expenditure data.

[1094] Server: Runs the generative AI model, analyzes the user's calorie and nutritional needs, and generates a meal menu based on that.

[1095] Server: Generated meal suggestions include specific recipes, ingredients, serving sizes, and cooking methods, and also provide advice on convenience store substitutions.

[1096] Step 6:

[1097] Notification of proposal details

[1098] Server: The generated meal suggestions are compiled into images and text format and sent to the device.

[1099] Device: Displays the received meal suggestions on the user's screen.

[1100] User: Review the suggestions provided and prepare your meal accordingly.

[1101] Step 7:

[1102] Gathering feedback

[1103] User: After consuming the suggested meal, the user takes a photo of the menu item they actually ate with their device and uploads it to the system.

[1104] Terminal: Sends the captured image to the server.

[1105] Server: Receives user feedback and forwards it to the image analysis engine.

[1106] Server: The analysis results are saved in a database and used to improve the accuracy of future meal suggestions.

[1107] In this way, the system performs a series of processes based on the data input by the user and makes meal suggestions tailored to individual needs.

[1108] Example 1

[1109] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1110] Conventional health management systems have had problems with the accuracy of their balanced dietary recommendations and the inability to fully reflect the user's individual health and activity data. Furthermore, the dietary recommendations lacked specificity, making it difficult for users to actually consume the suggested meals. Furthermore, there was also the issue of not effectively utilizing feedback after consumption, which led to poor accuracy in future recommendations.

[1111] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1112] In this invention, the server includes: a means for inputting personal information such as the user's age, gender, height, weight, and health status; a means for uploading image data of the user's meals; a means for identifying the calories and nutrients of the meals using an image analysis engine; a means for estimating the user's energy expenditure; a means for generating meal suggestions based on the user's calorie and nutritional needs using a generative AI model; a means for notifying the user of the generated meal suggestions; a means for authenticating the user upon login; a means for transmitting and storing photo data on the server; a means for inputting the user's activity data and estimating energy expenditure; and a means for displaying the generated suggestions to the user in text and image format. This enables highly accurate meal suggestions that accurately reflect the user's individual health information and actual intake and activity data, providing more specific and easy-to-follow menu suggestions. Furthermore, by utilizing feedback, the accuracy of future suggestions can be improved.

[1113] "Means for inputting personal information such as the user's age, sex, height, weight, and health condition" refers to the means by which a user inputs personal information such as their age, sex, height, weight, and health condition into the system and transmits that information to the server.

[1114] The "means for uploading image data of meals taken by the user" refers to a means for the user to upload image data of meals taken using the camera function to the system and send the images to the server.

[1115] "Means for identifying calories and nutrients in meals using an image analysis engine" refers to means for analyzing uploaded images of meals and using image analysis software or algorithms to identify calories and nutrients in the meals.

[1116] "Means for estimating a user's energy expenditure" refers to means for calculating and estimating a user's daily energy expenditure based on the user's daily activity data and exercise data.

[1117] "Means for using a generative AI model to generate meal suggestions based on a user's calorie and nutritional needs" means means for using an artificial intelligence model to analyze a user's calorie intake and nutritional needs and generate appropriate meal menus and recipes.

[1118] The "means for notifying the user of the generated meal suggestions" refers to means for notifying and displaying the generated meal suggestions in text and image format on the user's terminal.

[1119] "Means for authenticating users when they log in" refers to the means for authenticating users using an ID and password, etc., when they access the system.

[1120] The "means for transmitting photo data and storing it on the server" refers to the means for transmitting photo data taken by the user to the server and storing it within the server.

[1121] "Means for inputting user activity data and estimating energy expenditure" refers to means for a user to input data about their own activities and exercise and estimate energy expenditure based on that data.

[1122] "Means for displaying the generated suggestions to the user in text and image format" refers to means for displaying the contents of the generated meal suggestions to the user in text and image format on the user's terminal and visually presenting them to the user.

[1123] The present invention is a system that makes appropriate meal recommendations based on a user's individual health information and dietary data. The system of the present invention consists of steps to process the user's input information, make meal recommendations using a generative AI model based on the analysis results, and notify the user and collect feedback. The specific system configuration is described below.

[1124] Registering user information

[1125] The user logs in to the system and enters personal data such as age, gender, height, weight, and health condition. The device collects this information and sends it to the server. The server stores the received data in a database. Through this process, the user's basic health information is accumulated in the system.

[1126] Uploading meal information

[1127] A user takes a photo of their meal using a mobile device or camera and uploads the image data. The device sends the captured image to a server. The server then forwards the received image data to an image analysis engine. This analysis engine can be, for example, image analysis software.

[1128] Image analysis and calorie counting

[1129] The server uses an image analysis engine to analyze the contents of the meal from the uploaded image and identify calories and key nutrients. The analysis can be performed using cloud-based image analysis software. The identified nutrient and calorie data is aggregated and stored in a database on the server.

[1130] Energy Expenditure Estimation

[1131] The user enters their daily activity (e.g., commuting) into the device. The device then sends the information to the server. The server analyzes the user's activity data and estimates their energy expenditure. This analysis can be performed using activity tracking software. The estimated results are stored in a database.

[1132] Generating meal suggestions

[1133] The server uses a generative AI model to generate appropriate meal suggestions based on the user's total calorie intake, nutrient content, and energy expenditure data. This process can be performed using generative AI model software, for example. The generated menu includes specific recipes, ingredients, serving sizes, and cooking methods.

[1134] For example, here's a prompt to input to a generative AI model:

[1135] "Please suggest a balanced dinner menu for a 30-year-old woman based on the following criteria: Breakfast: toast and salad, Lunch: boxed lunch and dessert, Daily exercise: light exercise during commute."

[1136] Notification of proposal details

[1137] The server then compiles the generated meal suggestions into images and text format and sends them to the device, which then displays them on the user's screen. The user can then review the suggested menu and follow the specific steps to execute it.

[1138] Gathering feedback

[1139] After consuming the suggested meal, the user takes a photo of the meal and uploads it back to the system. The device then sends the image to the server, which then receives it, analyzes it using an image analysis engine, and stores the results in a database. This feedback improves the accuracy of future meal suggestions.

[1140] As described above, this system can respond to the individual needs of users and support more effective health management.

[1141] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1142] Step 1: Register your user information

[1143] Input: The user logs into the system and enters personal information such as age, gender, height, weight, and health status.

[1144] Specific behavior: A user accesses the system using a web browser or mobile app. They enter their credentials on the login screen to log in to the system. They then enter the required personal data on the profile screen.

[1145] Output: The device sends the entered personal data to the server, which stores the received data in a database.

[1146] Step 2: Upload your meal information

[1147] Input: The user takes a photo of the food they have eaten and uploads the image data to the system.

[1148] Specific actions: The user takes a photo of their meal using the smartphone camera, selects the photo in the form to upload it to the system, and clicks the submit button.

[1149] Output: The terminal sends the image data to the server, which then transfers the received image data to the image analysis engine.

[1150] Step 3: Image analysis and calorie calculation

[1151] Input: The server analyzes the image data using an image analysis engine.

[1152] Specific operation: The server passes the received image data to the image analysis engine and sends an API request. The image analysis engine analyzes the contents of the meal and returns information about calories and nutrients.

[1153] Output: The server aggregates the calorie and nutrient information obtained from the image analysis engine and stores it in a database.

[1154] Step 4: Estimate energy consumption

[1155] Input: The user inputs their daily activity level into the device.

[1156] Specific actions: The user enters information about their daily activities (commuting, exercise, etc.) into a form on the device and clicks the submit button.

[1157] Output: The device sends the input activity data to the server, which uses activity tracking software or an API to estimate energy expenditure based on the input data and stores the results in a database.

[1158] Step 5: Generate meal suggestions

[1159] Input: The server retrieves data on total calorie intake, nutrient amounts, and energy expenditure from a database.

[1160] Specific operation: The server reads the user's latest health and dietary information from the database and sends prompt sentences to the generative AI model.

[1161] Example prompt: "For a 30-year-old woman, please suggest a balanced dinner menu based on the following criteria: Breakfast: toast and salad; Lunch: boxed lunch and dessert; Daily exercise: light exercise during commute."

[1162] Output: Take the generated meal suggestions and format them as a menu with specific recipes, ingredients, serving sizes, and cooking instructions.

[1163] Step 6: Notification of proposal

[1164] Input: Generated meal suggestions

[1165] Specific operation: The server sends the generated meal suggestions in text and image format to the device, which then displays them on the user's screen.

[1166] Output: The user confirms and implements the received meal suggestions.

[1167] Step 7: Gather feedback

[1168] Input: The user consumes the suggested meal and then takes and uploads a photo of the meal after consumption.

[1169] Specific operation: After eating a meal, the user takes a photo and uploads it to the system. The device then sends the photo data to the server.

[1170] Output: The server passes the received image back to the image analysis engine and stores the analysis results in a database, thereby improving the accuracy of future meal suggestions.

[1171] (Application example 1)

[1172] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1173] Many users today struggle to manage their health and daily diet. Even when they receive healthy dietary recommendations, they lack concrete advice on how to translate them into real-world purchasing behavior. This creates a growing need for a system that can efficiently select the right ingredients and meal plans.

[1174] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1175] In this invention, the server includes means for inputting information such as the user's age, sex, height, weight, and health condition, means for uploading image data of meals consumed by the user, means for identifying the calories and nutrients of the meals using an image analysis engine, means for estimating the user's energy expenditure, means for generating meal suggestions based on the user's calorie and nutritional needs using a generative AI model, means for notifying the user of the generated meal suggestions, and means for displaying the suggested ingredients and menus when the user visits a physical store. This makes it easier for users to reflect healthy meal suggestions in their actual purchasing behavior, enabling more effective health management.

[1176] "User information" refers to basic physical data about each individual user, such as age, gender, height, weight, and health status.

[1177] "Meal image data" refers to digitally saved photographs of meals eaten by a user.

[1178] An "image analysis engine" is software or algorithm that analyzes image data to identify the calories and nutrients of a meal.

[1179] "Energy expenditure" means the total amount of calories burned through a user's daily activities and exercise.

[1180] A "generative AI model" is a model that uses artificial intelligence to generate personalized meal suggestions based on user data.

[1181] "Meal Suggestions" means suggestions that provide specific meals or menus to be consumed based on the user's calorie and nutritional needs.

[1182] "Notification means" refers to a method or device for notifying the user of the generated meal suggestions.

[1183] "Physical stores" refer to physical stores that users visit and use, such as grocery stores and restaurants.

[1184] "Means for displaying ingredients and menu items" refers to methods or devices that visually display suggested ingredients and menu items when a user visits a physical store.

[1185] A specific embodiment of a system for implementing the present invention will be described below. This system makes appropriate meal suggestions based on a user's individual health information and dietary data, and supports purchasing behavior, particularly in physical stores. This system involves inputting user information, analyzing dietary image data, estimating energy consumption, generating and notifying meal suggestions, and displaying ingredients and menus in physical stores.

[1186] System Overview

[1187] (1) User information registration:

[1188] Users log in to the system and enter personal data such as age, sex, height, weight, health condition, etc. through a terminal. The terminal sends this information to the server, which stores it in a database.

[1189] (2) Uploading meal information:

[1190] Users take photos of the food they eat and upload the image data to the system via their device, which then sends the images to the server, which then forwards them to the image analysis engine.

[1191] (3) Image analysis and calorie calculation:

[1192] The server uses an image analysis engine to identify the calories and key nutrients of the food from the uploaded image, and the calculations are stored in a database.

[1193] (4) Energy consumption estimation:

[1194] Users input their daily activity levels (e.g., commuting) into the system via their devices. The server estimates the total daily energy expenditure based on the user's activity level and stores the estimated results in a database.

[1195] (5) Meal suggestion generation:

[1196] Based on the user's total calorie intake, nutrient content, and energy expenditure data, the server uses a generative AI model to create appropriate meal suggestions, including specific recipes, ingredients, serving sizes, and cooking methods, and also provides advice on substitute ingredients from convenience stores and supermarkets.

[1197] (6) Notification of proposal:

[1198] The server sends the generated meal suggestions to the terminal, which displays them on the user's screen. The user can then confirm the suggested meals and act accordingly.

[1199] (7) Display in physical stores:

[1200] When a user visits a physical store, they use their device to scan the store's QR code, which allows the server to display ingredients and menu items to purchase in real time based on the user's meal suggestions.

[1201] (8) Feedback Collection:

[1202] After consuming the suggested meal, the user takes a photo of the meal and uploads it to the system via their device. The server receives the feedback and stores the analysis results in a database using an image analysis engine. This improves the accuracy of meal suggestions for future visits.

[1203] Examples:

[1204] For example, let's say User A, a 30-year-old woman, uses this system. After logging in, User A enters her age, gender, height, weight, and health status, and eats toast and salad for breakfast, and bibimbap bento and pudding for lunch. This information is uploaded to the system as image data. The server uses an image analysis engine to calculate the calorie intake for that day, and a generative AI model uses the energy expenditure from User A's commute to suggest dinner. Ultimately, a specific meal menu such as "grilled chicken salad, tofu vegetable soup, and brown rice" is suggested. User A goes to a nearby physical store and scans the store's QR code to check the ingredients and menu items to purchase on her smartphone. She then purchases the ingredients based on the suggestions and cooks. After eating, User A uploads an image of the menu she actually ate back to the system, contributing to improving the accuracy of suggestions from next time onwards.

[1205] Example prompt sentence:

[1206] User ID: 12345

[1207] Remaining calorie intake: 600

[1208] Suggest a suitable dinner menu based on the following criteria:

[1209] 1. Calories: 600 kcal or less

[1210] 2. Health status: Healthy adult female

[1211] 3. Nutritional balance: high protein, low carbohydrate

[1212] Suggested menu items should include specific ingredients, portions, and cooking methods.

[1213] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1214] Step 1:

[1215] Users log in to the device and enter information such as their age, gender, height, weight, and health condition. The device receives this information and sends it to the server, which then stores the received user information in a database.

[1216] Input: User information (age, gender, height, weight, health condition)

[1217] Output: User information stored in the database

[1218] Specific operation: When a user fills in information in the application's input form and presses the "Save" button, the terminal sends it to the server, which then stores it in the database.

[1219] Step 2:

[1220] Users take photos of the food they eat and upload the image data to the system via their device. The device sends these images to the server, which then forwards them to the image analysis engine.

[1221] Input: Food image data

[1222] Output: Image data sent to the server

[1223] How it works: When a user takes a photo of their meal and presses the "upload" button in the application, the device sends the image data to the server, which then passes it to the image analysis engine.

[1224] Step 3:

[1225] The server uses an image analysis engine to identify the calories and key nutrients of the food from the uploaded image, and the analysis results are stored in a database.

[1226] Input: Food image data

[1227] Output: Calorie and macronutrient data for the meal

[1228] Specific operation: The image analysis engine inputs image data into an algorithm and performs analysis. The analysis results are stored in a database via a server.

[1229] Step 4:

[1230] Users input their daily activity levels (e.g., commuting) into the system via their devices. The server receives the information and estimates the user's total daily energy expenditure based on their activity level. This estimation is then stored in a database.

[1231] Input: Activity data (commuting, exercise, etc.)

[1232] Output: Estimated energy consumption

[1233] How it works: After a user fills out the application's activity recording form and presses the "Submit" button, the device sends the data to the server, which then uses it to calculate activity levels.

[1234] Step 5:

[1235] The server uses a generative AI model to generate appropriate meal suggestions based on the user's total calorie intake, nutrient content, and energy expenditure data, including specific recipes, ingredients, serving sizes, and cooking methods.

[1236] Input: Total calories, nutrient content, energy expenditure

[1237] Output: Specific meal suggestions (recipe, ingredients, portions, cooking method)

[1238] How it works: The server inputs this data into a generative AI model to generate optimal meal suggestions for the user.

[1239] Step 6:

[1240] The server sends the generated meal suggestions to the terminal, which displays them on the user's screen. The user can then confirm the suggested meals and act accordingly.

[1241] Input: Meal suggestion data

[1242] Output: Meal suggestions displayed on the user's device

[1243] Specific operation: The server generates meal suggestions and sends them to the device, which receives them and displays them on the application.

[1244] Step 7:

[1245] When a user visits a physical store, they use their device to scan the store's QR code, and the server displays ingredients and menu items to purchase in real time based on the user's meal suggestions.

[1246] Input: QR code from physical store, suggested meal data

[1247] Output: List of ingredients and menu items to purchase

[1248] How it works: The user scans a QR code at a store and sends the information to the server, which then generates a shopping list based on the user's meal suggestions and displays it on the device.

[1249] Step 8:

[1250] After consuming the suggested meal, the user takes a photo of the meal and uploads it to the system via their device. The server receives the feedback and stores the analysis results in a database using an image analysis engine.

[1251] Input: Image data of the menu you ate

[1252] Output: Parsed feedback data

[1253] How it works: The user takes a photo of the menu item they ate and uploads it to the server through the application. The server passes the data to the image analysis engine and stores the analysis results in a database.

[1254] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1255] A specific embodiment of a system for implementing the present invention will be described below. This system recognizes the user's emotional state in addition to their individual health information and dietary data to make meal suggestions. The overall configuration of the system consists of the steps of processing the user's input information, making meal suggestions based on the analysis results using a generative AI model and emotion engine, and notifying the user and collecting feedback.

[1256] System Overview

[1257] Registering user information

[1258] Users log in to the system and enter personal data such as age, gender, height, weight, and health status. The terminal receives this information and sends it to the server, which then stores the received user information in a database.

[1259] Collecting Emotional Data

[1260] Users can input data about their physical condition and mood, or communicate their emotions to the system using facial and voice recognition technology. The device receives this emotional data and sends it to the server, which then transfers it to the emotion engine.

[1261] Uploading meal information

[1262] Users take photos of the food they eat before and after eating and upload the image data to the system via their device. The device sends these images to the server, which receives the image data and transfers them to the image analysis engine.

[1263] Image analysis and calorie counting

[1264] The server uses an image analysis engine to identify the calories and macronutrients of the food from the uploaded image. The server then aggregates the calorie and nutrient information to calculate the total calories and macronutrient intake of the user that day. The calculation results are stored in a database.

[1265] Energy Expenditure Estimation

[1266] Users input their daily activity levels (e.g., commuting) into the system via their device. The device then sends the information to the server, which then estimates the user's total daily energy expenditure based on their activity level. The estimated results are stored in a database.

[1267] Emotion analysis and reflection

[1268] The server uses an emotion engine to analyze the user's current emotional state from the emotion data. The analysis results are used to adjust subsequent meal suggestions. For example, if the user is feeling stressed, the suggestions will include ingredients with a relaxation effect.

[1269] Generating meal suggestions

[1270] The server uses a generative AI model to create appropriate meal suggestions based on the user's total calorie intake, nutrient content, energy expenditure, and emotional state. The generative AI model generates meal menus that take into account the user's calorie and nutritional needs as well as their emotional state. The menus include specific recipes, ingredients, serving sizes, and cooking methods. It also provides advice on convenience store substitutions.

[1271] Notification of proposal details

[1272] The server compiles the generated meal suggestions into images and text format and sends them to the device. The device displays the received meal suggestions on the user's screen. The user checks the displayed suggestions and prepares dinner accordingly.

[1273] Gathering feedback

[1274] After consuming the suggested meal, the user takes a photo of the meal they actually ate and uploads it to the system via their device. The device then sends the image to the server. The server receives feedback from the user and stores the analysis results in a database using an image analysis engine. This leads to improved accuracy of meal suggestions for future visits.

[1275] Specific examples

[1276] For example, let's say User B, a 30-year-old woman, uses the system. After logging in, User B enters her age, gender, height, weight, and health condition, and also enters emotional data about the stress she is currently feeling. She uploads image data to the system, showing that she had oatmeal and fruit for breakfast and a sandwich and salad for lunch. The server uses an image analysis engine to calculate the calories consumed that day, and uses a generative AI model to suggest dinner based on User B's energy expenditure during her commute. Additionally, based on her emotional state analyzed by the emotion engine, it suggests relaxing herbal tea and meals rich in B vitamins.

[1277] In this way, the system provides appropriate and effective health management and dietary recommendations based on the user's individual needs and emotional state.

[1278] The processing flow will be explained below.

[1279] Step 1:

[1280] Registering user information

[1281] User: Logs into the system and enters personal data such as age, gender, height, weight, and health status.

[1282] Terminal: Receives input information and sends it to the server.

[1283] Server: Stores the received user information in a database.

[1284] Step 2:

[1285] Collecting Emotional Data

[1286] User: Enters data about physical condition and mood into the device, and communicates emotions to the system using facial and voice recognition technology.

[1287] Terminal: Receives these emotion data and sends them to the server.

[1288] Server: Transfers the received emotion data to the emotion engine.

[1289] Step 3:

[1290] Uploading meal information

[1291] User: Takes pictures of meals such as breakfast, lunch, and dinner on the device and uploads them to the system.

[1292] Terminal: Sends the captured image data to the server.

[1293] Server: Transfers the received image data to the image analysis engine.

[1294] Step 4:

[1295] Image analysis and calorie counting

[1296] Server: Uses an image analysis engine to identify the calories and key nutrients of a meal from an uploaded image.

[1297] Server: Aggregates the identified calorie and nutrient information and calculates the total calories and macronutrients consumed by the user for the day.

[1298] Server: Saves the calculation results in a database.

[1299] Step 5:

[1300] Energy Expenditure Estimation

[1301] User: Enters daily activity data (commuting, exercise, etc.) on the device and sends it to the system.

[1302] Terminal: Sends the entered activity information to the server.

[1303] Server: Estimates total daily energy expenditure based on the user's activity level.

[1304] Server: Saves the estimation results in a database.

[1305] Step 6:

[1306] Emotion analysis and reflection

[1307] Server: Uses the emotion engine to analyze the user's current emotional state from the emotion data.

[1308] Server: The analysis results are stored and used to adjust future meal suggestions. For example, if the user is feeling stressed, the suggestions may include foods with a relaxing effect.

[1309] Step 7:

[1310] Generating meal suggestions

[1311] Server: Uses a generative AI model to create appropriate meal suggestions based on the user's total calorie intake, nutrient amounts, energy expenditure, and emotional state data.

[1312] Server: Runs the generative AI model, analyzes the user's calorie and nutritional needs, and generates a meal menu based on that.

[1313] Server: Generated meal suggestions include specific recipes, ingredients, serving sizes, and cooking methods, and also provide advice on convenience store substitutions.

[1314] Step 8:

[1315] Notification of proposal details

[1316] Server: The generated meal suggestions are compiled into images and text format and sent to the device.

[1317] Device: Displays the received meal suggestions on the user's screen.

[1318] User: Review the suggestions provided and prepare your meal accordingly.

[1319] Step 9:

[1320] Gathering feedback

[1321] User: After consuming the suggested meal, the user takes a photo of the menu item they actually ate with their device and uploads it to the system.

[1322] Terminal: Sends the captured image to the server.

[1323] Server: Receives user feedback and forwards it to the image analysis engine.

[1324] Server: The analysis results are saved in a database and used to improve the accuracy of future meal suggestions.

[1325] In this way, the system processes user input and emotional data to provide personalized meal recommendations, and can continuously improve the accuracy of these recommendations through feedback.

[1326] Example 2

[1327] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1328] In today's busy lifestyles, it is difficult for users to receive appropriate dietary recommendations based on their health status and emotions. Furthermore, there is no system that can provide dietary recommendations that take into account the user's emotional state, rather than just calories and nutrients. This leads to a delay in effective health management and dietary improvement.

[1329] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1330] In this invention, the server includes means for inputting information such as the user's age, sex, height, weight, and health condition, means for uploading image data of meals consumed by the user, means for identifying the calories and nutrients of the meals using an image analysis engine, means for estimating the user's energy expenditure, means for collecting the user's emotion data and analyzing it using an emotion engine, means for generating meal suggestions based on the user's calorie and nutritional needs and emotional state using a generative AI model, and means for notifying the user of the generated meal suggestions. This enables appropriate meal suggestions based on the user's health condition and emotions.

[1331] "User" refers to an individual who uses this system.

[1332] "Age" refers to the number of years since the user was born.

[1333] "Gender" refers to a user's biological or social sex.

[1334] "Height" refers to the vertical length of the user's body.

[1335] "Weight" refers to the mass of a user's body.

[1336] "Health Status" refers to the state of a User's physical and mental health.

[1337] "Information" refers to data such as the user's age, gender, height, weight, and health status.

[1338] "Means" refers to a method or device for achieving a particular purpose.

[1339] "Meal image data" refers to image files of meals consumed by the user.

[1340] "Upload" refers to the act of sending data from a user's device to a server.

[1341] "Image analysis engine" refers to software or hardware for analyzing image data and extracting specific information.

[1342] "Calories" refers to the amount of energy a food contains.

[1343] "Nutrients" are substances contained in food that are necessary for the growth and maintenance of health of the body.

[1344] "Energy Expenditure" refers to the total amount of energy consumed by a user's daily activities.

[1345] "Emotional data" refers to data related to the user's physical condition and mood.

[1346] "Emotion engine" refers to software or hardware for analyzing emotional data to identify a user's emotional state.

[1347] A "generative AI model" refers to an artificial intelligence system that generates appropriate meal suggestions based on user information.

[1348] "Notification" refers to the act of informing the user of the generated meal suggestions.

[1349] A "system" refers to a set of multiple means or devices that function in conjunction with one another.

[1350] A specific embodiment of a system for implementing the present invention will be described below. This system recognizes the user's emotional state as well as their individual health information and dietary data to make meal suggestions. This system processes the user's input information, makes meal suggestions based on the analysis results using a generative AI model and an emotion engine, and notifies the user and collects feedback.

[1351] Registering user information

[1352] Users log in to the system and enter personal data such as age, gender, height, weight, and health status. The terminal receives this information and sends it to the server, which then stores the received user information in a database.

[1353] Collecting Emotional Data

[1354] Users can input data about their physical condition and mood, or communicate their emotions to the system using facial and voice recognition technology. The device receives this emotional data and sends it to the server, which then transfers it to the emotion engine.

[1355] Uploading meal information

[1356] Users take photos of the food they eat before and after eating and upload the image data to the system via their device. The device sends these images to the server, which receives the image data and transfers them to the image analysis engine.

[1357] Image analysis and calorie counting

[1358] The server uses an image analysis engine to identify the calories and macronutrients of the food from the uploaded image. The server then aggregates the calorie and nutrient information to calculate the total calories and macronutrient intake of the user that day. The calculation results are stored in a database.

[1359] Energy Expenditure Estimation

[1360] Users input their daily activity levels (e.g., commuting) into the system via their device. The device then sends the information to the server, which then estimates the user's total daily energy expenditure based on their activity level. The estimated results are stored in a database.

[1361] Emotion analysis and reflection

[1362] The server uses an emotion engine to analyze the user's current emotional state from the emotion data. The analysis results are used to adjust subsequent meal suggestions. For example, if the user is feeling stressed, the suggestions will include ingredients with a relaxation effect.

[1363] Generating meal suggestions

[1364] The server uses a generative AI model to create appropriate meal suggestions based on the user's total calorie intake, nutrient content, energy expenditure, and emotional state. The generative AI model generates meal menus that take into account the user's calorie and nutritional needs as well as their emotional state. The menus include specific recipes, ingredients, serving sizes, and cooking methods. It also provides advice on convenience store substitutions.

[1365] Notification of proposal details

[1366] The server compiles the generated meal suggestions into images and text format and sends them to the device. The device displays the received meal suggestions on the user's screen. The user checks the displayed suggestions and prepares dinner accordingly.

[1367] Gathering feedback

[1368] After consuming the suggested meal, the user takes a photo of the meal they actually ate and uploads it to the system via their device. The device then sends the image to the server. The server receives feedback from the user and stores the analysis results in a database using an image analysis engine. This leads to improved accuracy of meal suggestions for future visits.

[1369] Specific examples

[1370] For example, suppose a 30-year-old female user uses the system. After logging in, the user enters her age, gender, height, weight, and health condition, and also enters emotional data about the stress she is currently experiencing. She uploads image data of her breakfast of oatmeal and fruit, and lunch of a sandwich and salad, to the system. The server uses an image analysis engine to calculate the calorie intake for that day, and uses a generative AI model to suggest dinner based on the user's energy expenditure from commuting. Based on the emotional state analyzed by the emotion engine, the system also suggests relaxing herbal teas and meals rich in B vitamins. In this way, the system provides appropriate and effective health management and dietary suggestions based on the user's individual needs and emotional state.

[1371] Prompt Sentence Examples

[1372] "A 30-year-old female user uses the system and enters her age, gender, height, weight, and health status. The user is currently stressed and ate oatmeal and fruit for breakfast, and a sandwich and salad for lunch. Please generate dinner suggestions taking into account her energy expenditure from her commute."

[1373] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1374] Step 1: Register your user information

[1375] Users log in to the system and enter personal data such as age, gender, height, weight, and health status.

[1376] Input: Age, gender, height, weight, health condition

[1377] Output: User information data (e.g., JSON format)

[1378] The terminal receives the entered information and transmits the data to the server.

[1379] Specific operation: The terminal encodes the input data into JSON format and sends it to the server using the HTTPS protocol.

[1380] The server stores the received data in a database.

[1381] Input: User information data (e.g., JSON format)

[1382] Output: User information record in database

[1383] Specific operation: The server analyzes the received data and performs an insert operation on a specific table in the database.

[1384] Step 2: Collecting emotion data

[1385] Users input data about their physical condition and mood, or communicate their emotions to the system using facial and voice recognition technology.

[1386] Input: Physical condition, mood data, or emotional data

[1387] Output: Emotion data (numerical and text format)

[1388] The terminal receives this data and transmits it to the server.

[1389] Specific operation: The device generates an API request to send emotion data to the server in real time.

[1390] The server transfers the received emotion data to the emotion engine for analysis.

[1391] Input: Received emotion data

[1392] Output: Parsed emotional state data

[1393] Specific operation: The server passes the emotion data to the emotion engine's API and receives the analysis results.

[1394] Step 3: Upload your meal information

[1395] Users take photos of the food they eat before and after eating and upload the image data to the system.

[1396] Input: Food image

[1397] Output: Image data file

[1398] The terminal transmits the image data to the server.

[1399] Specific operation: The device compresses the captured image and generates an API request to upload it to the server.

[1400] The server receives the image data and transfers it to the image analysis engine.

[1401] Input: Image data file

[1402] Output: Image analysis results (calories, nutrient data)

[1403] Specific operation: The server calls the API of the image analysis engine and sends a request to analyze the image data.

[1404] Step 4: Image analysis and calorie calculation

[1405] The server uses an image analysis engine to identify the calories and key nutrients of a meal from the uploaded image.

[1406] Input: Food image data

[1407] Output: Calorie information, nutrient information

[1408] How it works: The image analysis engine applies algorithms to recognize ingredients in an image and identify the calorie and nutritional information for each ingredient.

[1409] The server compiles the results and calculates the user's total calorie and nutrient intake.

[1410] Input: Calorie information and nutrient information for each meal

[1411] Output: Total calorie intake and macronutrient data

[1412] What it does: The server uses an aggregation algorithm to add up the calories and nutrients for each meal.

[1413] The calculation results are saved in a database.

[1414] Input: Total calorie intake and amount of major nutrients

[1415] Output: Accumulated data in the database

[1416] Specific operation: The server performs an insert operation on the results into a specific table in the database.

[1417] Step 5: Estimate energy consumption

[1418] Users enter their daily activity levels into the system.

[1419] Input: Activity data (commuting, exercise, etc.)

[1420] Output: Activity log

[1421] The terminal transmits the information to the server.

[1422] Specific operation: The device encodes the activity data and sends an API request to the server.

[1423] The server estimates total energy expenditure based on the user's activity level.

[1424] Input: Activity data

[1425] Output: Total energy consumption data

[1426] Specific operation: The server applies a calculation algorithm based on the activity data to estimate energy expenditure.

[1427] The estimation results are saved in a database.

[1428] Input: Total energy consumption data

[1429] Output: Energy consumption records in a database

[1430] Specific operation: The server inserts the estimation results into the database.

[1431] Step 6: Analyze and reflect on emotions

[1432] The server uses an emotion engine to parse the user's current emotional state from the emotion data.

[1433] Input: Emotion data

[1434] Output: Parsed emotional state

[1435] Specific operation: The emotion engine analyzes the emotion data and generates a quantified emotional state.

[1436] Configure settings to reflect the analysis results in meal suggestions.

[1437] Input: Parsed emotional state

[1438] Output: Adjusted meal suggestion data

[1439] What it does: The server applies an algorithm that adjusts the suggestions depending on the emotional state.

[1440] Step 7: Generate meal suggestions

[1441] The server uses a generative AI model to create appropriate meal suggestions based on the user's total calorie intake, nutrient amounts, energy expenditure, and emotional state data.

[1442] Input: User's total calorie intake, nutritional information, energy expenditure, emotional state data

[1443] Output: Generated meal suggestions

[1444] Specific operation: The generative AI model generates an appropriate meal menu based on the prompt text.

[1445] The generated suggestions include specific recipes, ingredients, quantities, and cooking methods.

[1446] Input: Generated meal suggestions

[1447] Output: Detailed meal menu (including recipe, ingredients, portions, and cooking method)

[1448] What it does: The server formats the results of the generative AI model and compiles them into a detailed menu.

[1449] Step 8: Notification of proposal

[1450] The server compiles the generated meal suggestions into image and text format and sends them to the terminal.

[1451] Input: Detailed meal menu

[1452] Output: Notification format

[1453] Specific operation: The server converts the menu information into image and text format and sends it to the terminal.

[1454] The device displays the received meal suggestions on the user's screen.

[1455] Input: Notification Format

[1456] Output: The suggestions that are displayed on the user's screen

[1457] Specific operation: The terminal obtains the notification data and displays it on the user interface.

[1458] The user reviews the displayed suggestions and prepares dinner accordingly.

[1459] Step 9: Gather feedback

[1460] After consuming the suggested meal, the user takes a photo of the meal they actually ate and uploads it to the system.

[1461] Input: Food image

[1462] Output: Uploaded image data

[1463] The terminal transmits the image data to the server again.

[1464] Specific operation: The device generates an API request to compress the image and send it to the server.

[1465] The server analyzes the feedback using an image analysis engine and stores the results in a database.

[1466] Input: Uploaded image data

[1467] Output: Analysis results

[1468] Specific operation: The server calls the API of the image analysis engine again and inserts the feedback results into the database.

[1469] (Application example 2)

[1470] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1471] Conventional health management systems often do not provide meal suggestions that take into account the user's emotional state, and comprehensive health management, including the user's mental health, has not been fully realized. Furthermore, virtual stores are unable to provide meal suggestions based on real-time health information, making it difficult for users to efficiently select ingredients and recipes that are suitable for them. This makes it difficult to maintain and improve users' health through appropriate diets.

[1472] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1473] In this invention, the server includes: means for inputting information such as the user's age, sex, height, weight, and health condition; means for uploading image data of the user's meals; means for identifying the calories and nutrients of the meals using an image analysis engine; means for estimating the user's energy expenditure; means for collecting and analyzing the user's emotional data; means for generating meal suggestions based on the user's calorie and nutritional needs and emotional state using a generative AI model; means for notifying the user of the generated meal suggestions; and means for providing meal suggestions in real time in a virtual store based on the user's health information, dietary data, and emotional state. This enables comprehensive health management that takes the user's emotional state into consideration and enables the selection of appropriate ingredients and recipes in real time in the virtual store.

[1474] "User Information" refers to personal information about a user, such as their age, gender, height, weight, and health status.

[1475] "Image data" refers to photographic data of the food consumed by the user.

[1476] The "image analysis engine" is software that identifies the calories and nutrients of meals from uploaded image data.

[1477] "Energy expenditure" is the amount of energy a user expends through daily activities.

[1478] "Emotion data" is data that indicates information about the user's physical condition and mood, and is obtained using facial recognition technology or voice recognition technology.

[1479] A "generative AI model" is an artificial intelligence model that generates meal suggestions based on a user's calorie and nutritional needs and emotional state.

[1480] "Meal suggestions" are suggestions such as specific meal menus, recipes, ingredients, portions, cooking methods, etc., that are proposed to users.

[1481] A "virtual store" is a digital platform that allows users to shop in a virtual space.

[1482] "Real-time" refers to data processing and information provision occurring immediately or nearly simultaneously.

[1483] "Notification" is the act or mechanism of communicating generated meal suggestions to a user.

[1484] "Feedback" means providing the system with information about the food the user has eaten, which helps improve the accuracy of the next suggestion.

[1485] MODE FOR CARRYING OUT THE INVENTION

[1486] A detailed description of a system for realizing the present invention is given below. This system makes meal suggestions in real time in a virtual store based on the user's individual health information, dietary data, and emotional state.

[1487] System Overview

[1488] Registering user information

[1489] After accessing the virtual store, users use their smartphones or head-mounted displays (HMDs) to input personal information such as age, gender, height, weight, and health status. The devices receive this information and send it to a cloud server, which then stores the received user information in a database.

[1490] Collecting Emotional Data

[1491] Users can input data about their physical condition and mood using a camera or microphone installed on their smartphone or HMD, or communicate their emotions to the system using facial or voice recognition technology. The device receives this emotional data and sends it to a cloud server. The server then analyzes the received emotional data using an emotion engine to determine the user's current emotional state.

[1492] Uploading meal information

[1493] Users take photos of the meals they have eaten over the past 48 hours with their smartphones and upload them to the system. The device sends these images to a cloud server. The server receives the image data and transfers it to an image analysis engine. The image analysis engine identifies the calories and nutrients of the meals from the uploaded images.

[1494] Generating meal suggestions

[1495] The server uses a generative AI model based on the user's total calorie intake, nutrient content, energy expenditure, and emotional state data to generate appropriate meal suggestions. The generative AI model considers the user's calorie and nutritional needs as well as their emotional state to generate a meal menu that includes specific recipes, ingredients, serving sizes, and cooking methods.

[1496] Notification of proposal details

[1497] The server compiles the generated meal suggestions into images and text format and sends them to the device. The device then displays the received meal suggestions on the user's smartphone or HMD at appropriate locations within the virtual store. As the user moves around the virtual store, notifications pop up at appropriate locations.

[1498] Gathering feedback

[1499] After consuming the suggested meal, the user takes a photo of the meal and uploads it to the system via their device. The device then sends the image to a cloud server. The server receives feedback from the user and stores the analysis results in a database using an image analysis engine. This improves the accuracy of meal suggestions for future visits.

[1500] Hardware and Software Used

[1501] Smartphone: Use an Android or iOS device.

[1502] Head-mounted display (HMD): Use Oculus Rift, HTC Vive, etc.

[1503] Cloud server: Use cloud platforms such as Google Cloud, AWS, and Azure.

[1504] Emotion recognition technology: Uses technologies such as OpenCV and TensorFlow.

[1505] Generative AI models: Utilize generative AI such as OpenAI GPT-3.

[1506] Specific examples

[1507] For example, a user visits a virtual store and enters their personal information using a smartphone. Emotional data is collected using facial recognition with a camera installed in the HMD, and data on physical condition and mood is collected. After the user uploads photos of the meals they have eaten in the past 48 hours, the cloud server uses an image analysis engine to identify calories and nutrients, and then uses a generative AI model to make meal recommendations.

[1508] Example prompt sentence:

[1509] Based on their health, users need 2000 kcal of calories, 50g protein, 70g fat, and 250g carbohydrates.

[1510] Please suggest a dish using the following ingredients and method:

[1511] 1) Breakfast: Oatmeal and fruit

[1512] 2) Lunch: Sandwich and salad

[1513] 3) Dinner: Chicken steak and vegetables

[1514] The user's emotional state is stress.

[1515] This allows users to select appropriate ingredients and recipes in real time in a virtual store, enabling comprehensive health management that takes into account their emotional state.

[1516] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1517] Step 1:

[1518] Users access the virtual store and use their smartphone or HMD to enter personal information such as age, gender, height, weight, and health status. This information is then sent from the device to a cloud server, which then stores the received information in a database.

[1519] Input: User's age, gender, height, weight, health status

[1520] Output: User information stored in the database

[1521] Step 2:

[1522] Users input data about their physical condition and mood into the system using a camera and microphone installed on their smartphone or HMD. The device receives this emotional data and sends it to a cloud server. The server then uses an emotion engine to analyze the received emotional data and identify the user's current emotional state.

[1523] Input: Data about the user's physical condition and mood

[1524] Output: Parsed user emotional state

[1525] Step 3:

[1526] Users take photos of the meals they have eaten over the past 48 hours with their smartphones and upload them to the system. The device sends these image data to a cloud server. The server receives the image data and transfers it to an image analysis engine. The image analysis engine identifies the calories and nutrients of the meals from the uploaded images.

[1527] Input: User's food photo

[1528] Output: Parsed calorie and nutrient information

[1529] Step 4:

[1530] The server uses a generative AI model to generate appropriate meal suggestions based on the user's total calorie intake, nutrient intake, energy expenditure, and emotional state. The generative AI model generates meal menus that take into account the user's calorie and nutritional needs as well as their emotional state.

[1531] Input: User's total calorie intake, nutrient amounts, energy expenditure, emotional state

[1532] Output: Generated meal suggestions

[1533] Step 5:

[1534] The server compiles the generated meal suggestions into images and text format and sends them to the device. The device then displays the received meal suggestions on the user's smartphone or HMD at appropriate locations within the virtual store. As the user moves around the virtual store, notifications pop up at appropriate locations.

[1535] Input: Generated meal suggestions

[1536] Output: Notification and display in the user's virtual space

[1537] Step 6:

[1538] After consuming the suggested meal, the user takes a photo of the meal and uploads it to the system via their device. The device then sends the image to a cloud server. The server receives feedback from the user and stores the analysis results in a database using an image analysis engine. This improves the accuracy of meal suggestions for future visits.

[1539] Input: Food photos provided as feedback

[1540] Output: Database update to improve future meal suggestions

[1541] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1542] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1543] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1544] [Fourth embodiment]

[1545] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1546] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1547] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1548] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1549] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1550] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1551] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1552] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1553] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1554] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1555] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1556] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1557] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1558] A specific embodiment of a system for implementing the present invention will be described below. This system makes appropriate meal suggestions based on the user's individual health information and dietary data. The overall configuration of the system consists of steps to process the user's input information, make meal suggestions using a generative AI model based on the analysis results, and notify the user and collect feedback.

[1559] System Overview

[1560] Registering user information

[1561] Users log in to the system and enter personal data such as age, gender, height, weight, and health status. The terminal receives this information and sends it to the server, which then stores the received user information in a database.

[1562] Uploading meal information

[1563] Users take photos of the food they eat before and after eating and upload the image data to the system via their device. The device sends these images to the server, which receives the image data and transfers them to the image analysis engine.

[1564] Image analysis and calorie counting

[1565] The server uses an image analysis engine to identify the calories and macronutrients of the food from the uploaded image. The server then aggregates the calorie and nutrient information to calculate the total calories and macronutrient intake of the user that day. The calculation results are stored in a database.

[1566] Energy Expenditure Estimation

[1567] Users input their daily activity levels (e.g., commuting) into the system via their device. The device then sends the information to the server, which then estimates the user's total daily energy expenditure based on their activity level. The estimated results are stored in a database.

[1568] Generating meal suggestions

[1569] The server uses a generative AI model to create appropriate dinner suggestions based on the user's total calorie intake, nutrient content, and energy expenditure data. The generative AI model analyzes the user's calorie and nutritional needs and generates a meal menu based on them. This menu includes specific recipes, ingredients, quantities, and cooking methods. It also provides advice on convenience store substitutions.

[1570] Notification of proposal details

[1571] The server compiles the generated meal suggestions into images and text format and sends them to the device. The device displays the received meal suggestions on the user's screen. The user checks the suggested meals and prepares dinner accordingly.

[1572] Gathering feedback

[1573] After consuming the suggested meal, the user takes a photo of the meal they actually ate and uploads it to the system via their device. The device then sends the image to the server. The server receives feedback from the user and stores the analysis results in a database using an image analysis engine. This leads to improved accuracy of meal suggestions for future visits.

[1574] Specific examples

[1575] For example, let's say a 30-year-old female user A uses the system. After logging in, user A enters her age, gender, height, weight, and health condition, and eats toast and salad for breakfast, and bibimbap bento and pudding for lunch. This information is uploaded to the system as image data. The server uses an image analysis engine to calculate the calorie intake for that day, and the generative AI model makes a dinner suggestion based on user A's energy expenditure from her commute. Ultimately, a specific meal menu such as "grilled chicken salad, tofu vegetable soup, and brown rice" is suggested, and user A follows it. After eating, user A uploads an image of her actual meal to the system, which will help improve the accuracy of suggestions from next time onwards.

[1576] In this way, the present system effectively supports health management according to the individual needs of the user.

[1577] The processing flow will be explained below.

[1578] Step 1:

[1579] Registering user information

[1580] User: Logs into the system and enters personal data such as age, gender, height, weight, and health status.

[1581] Terminal: Receives input information and sends it to the server.

[1582] Server: Stores the received user information in a database.

[1583] Step 2:

[1584] Uploading meal information

[1585] User: Takes pictures of meals such as breakfast, lunch, and dinner on the device and uploads them to the system.

[1586] Terminal: Sends the captured image data to the server.

[1587] Server: Transfers the received image data to the image analysis engine.

[1588] Step 3:

[1589] Image analysis and calorie counting

[1590] Server: Uses an image analysis engine to identify the calories and key nutrients of a meal from an uploaded image.

[1591] Server: Aggregates the identified calorie and nutrient information and calculates the total calories and macronutrients consumed by the user for the day.

[1592] Server: Saves the calculation results in a database.

[1593] Step 4:

[1594] Energy Expenditure Estimation

[1595] User: Enters daily activity data (commuting, exercise, etc.) on the device and sends it to the system.

[1596] Terminal: Sends the entered activity information to the server.

[1597] Server: Estimates total daily energy expenditure based on the user's activity level.

[1598] Server: Saves the estimation results in a database.

[1599] Step 5:

[1600] Generating meal suggestions

[1601] Server: Uses a generative AI model to create appropriate meal suggestions based on the user's total calorie intake, nutrient content, and energy expenditure data.

[1602] Server: Runs the generative AI model, analyzes the user's calorie and nutritional needs, and generates a meal menu based on that.

[1603] Server: Generated meal suggestions include specific recipes, ingredients, serving sizes, and cooking methods, and also provide advice on convenience store substitutions.

[1604] Step 6:

[1605] Notification of proposal details

[1606] Server: The generated meal suggestions are compiled into images and text format and sent to the device.

[1607] Device: Displays the received meal suggestions on the user's screen.

[1608] User: Review the suggestions provided and prepare your meal accordingly.

[1609] Step 7:

[1610] Gathering feedback

[1611] User: After consuming the suggested meal, the user takes a photo of the menu item they actually ate with their device and uploads it to the system.

[1612] Terminal: Sends the captured image to the server.

[1613] Server: Receives user feedback and forwards it to the image analysis engine.

[1614] Server: The analysis results are saved in a database and used to improve the accuracy of future meal suggestions.

[1615] In this way, the system performs a series of processes based on the data input by the user and makes meal suggestions tailored to individual needs.

[1616] Example 1

[1617] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1618] Conventional health management systems have had problems with the accuracy of their balanced dietary recommendations and the inability to fully reflect the user's individual health and activity data. Furthermore, the dietary recommendations lacked specificity, making it difficult for users to actually consume the suggested meals. Furthermore, there was also the issue of not effectively utilizing feedback after consumption, which led to poor accuracy in future recommendations.

[1619] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1620] In this invention, the server includes: a means for inputting personal information such as the user's age, gender, height, weight, and health status; a means for uploading image data of the user's meals; a means for identifying the calories and nutrients of the meals using an image analysis engine; a means for estimating the user's energy expenditure; a means for generating meal suggestions based on the user's calorie and nutritional needs using a generative AI model; a means for notifying the user of the generated meal suggestions; a means for authenticating the user upon login; a means for transmitting and storing photo data on the server; a means for inputting the user's activity data and estimating energy expenditure; and a means for displaying the generated suggestions to the user in text and image format. This enables highly accurate meal suggestions that accurately reflect the user's individual health information and actual intake and activity data, providing more specific and easy-to-follow menu suggestions. Furthermore, by utilizing feedback, the accuracy of future suggestions can be improved.

[1621] "Means for inputting personal information such as the user's age, sex, height, weight, and health condition" refers to the means by which a user inputs personal information such as their age, sex, height, weight, and health condition into the system and transmits that information to the server.

[1622] The "means for uploading image data of meals taken by the user" refers to a means for the user to upload image data of meals taken using the camera function to the system and send the images to the server.

[1623] "Means for identifying calories and nutrients in meals using an image analysis engine" refers to means for analyzing uploaded images of meals and using image analysis software or algorithms to identify calories and nutrients in the meals.

[1624] "Means for estimating a user's energy expenditure" refers to means for calculating and estimating a user's daily energy expenditure based on the user's daily activity data and exercise data.

[1625] "Means for using a generative AI model to generate meal suggestions based on a user's calorie and nutritional needs" means means for using an artificial intelligence model to analyze a user's calorie intake and nutritional needs and generate appropriate meal menus and recipes.

[1626] The "means for notifying the user of the generated meal suggestions" refers to means for notifying and displaying the generated meal suggestions in text and image format on the user's terminal.

[1627] "Means for authenticating users when they log in" refers to the means for authenticating users using an ID and password, etc., when they access the system.

[1628] The "means for transmitting photo data and storing it on the server" refers to the means for transmitting photo data taken by the user to the server and storing it within the server.

[1629] "Means for inputting user activity data and estimating energy expenditure" refers to means for a user to input data about their own activities and exercise and estimate energy expenditure based on that data.

[1630] "Means for displaying the generated suggestions to the user in text and image format" refers to means for displaying the contents of the generated meal suggestions to the user in text and image format on the user's terminal and visually presenting them to the user.

[1631] The present invention is a system that makes appropriate meal recommendations based on a user's individual health information and dietary data. The system of the present invention consists of steps to process the user's input information, make meal recommendations using a generative AI model based on the analysis results, and notify the user and collect feedback. The specific system configuration is described below.

[1632] Registering user information

[1633] The user logs in to the system and enters personal data such as age, gender, height, weight, and health condition. The device collects this information and sends it to the server. The server stores the received data in a database. Through this process, the user's basic health information is accumulated in the system.

[1634] Uploading meal information

[1635] A user takes a photo of their meal using a mobile device or camera and uploads the image data. The device sends the captured image to a server. The server then forwards the received image data to an image analysis engine. This analysis engine can be, for example, image analysis software.

[1636] Image analysis and calorie counting

[1637] The server uses an image analysis engine to analyze the contents of the meal from the uploaded image and identify calories and key nutrients. The analysis can be performed using cloud-based image analysis software. The identified nutrient and calorie data is aggregated and stored in a database on the server.

[1638] Energy Expenditure Estimation

[1639] The user enters their daily activity (e.g., commuting) into the device. The device then sends the information to the server. The server analyzes the user's activity data and estimates their energy expenditure. This analysis can be performed using activity tracking software. The estimated results are stored in a database.

[1640] Generating meal suggestions

[1641] The server uses a generative AI model to generate appropriate meal suggestions based on the user's total calorie intake, nutrient content, and energy expenditure data. This process can be performed using generative AI model software, for example. The generated menu includes specific recipes, ingredients, serving sizes, and cooking methods.

[1642] For example, here's a prompt to input to a generative AI model:

[1643] "Please suggest a balanced dinner menu for a 30-year-old woman based on the following criteria: Breakfast: toast and salad, Lunch: boxed lunch and dessert, Daily exercise: light exercise during commute."

[1644] Notification of proposal details

[1645] The server then compiles the generated meal suggestions into images and text format and sends them to the device, which then displays them on the user's screen. The user can then review the suggested menu and follow the specific steps to execute it.

[1646] Gathering feedback

[1647] After consuming the suggested meal, the user takes a photo of the meal and uploads it back to the system. The device then sends the image to the server, which then receives it, analyzes it using an image analysis engine, and stores the results in a database. This feedback improves the accuracy of future meal suggestions.

[1648] As described above, this system can respond to the individual needs of users and support more effective health management.

[1649] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1650] Step 1: Register your user information

[1651] Input: The user logs into the system and enters personal information such as age, gender, height, weight, and health status.

[1652] Specific behavior: A user accesses the system using a web browser or mobile app. They enter their credentials on the login screen to log in to the system. They then enter the required personal data on the profile screen.

[1653] Output: The device sends the entered personal data to the server, which stores the received data in a database.

[1654] Step 2: Upload your meal information

[1655] Input: The user takes a photo of the food they have eaten and uploads the image data to the system.

[1656] Specific actions: The user takes a photo of their meal using the smartphone camera, selects the photo in the form to upload it to the system, and clicks the submit button.

[1657] Output: The terminal sends the image data to the server, which then transfers the received image data to the image analysis engine.

[1658] Step 3: Image analysis and calorie calculation

[1659] Input: The server analyzes the image data using an image analysis engine.

[1660] Specific operation: The server passes the received image data to the image analysis engine and sends an API request. The image analysis engine analyzes the contents of the meal and returns information about calories and nutrients.

[1661] Output: The server aggregates the calorie and nutrient information obtained from the image analysis engine and stores it in a database.

[1662] Step 4: Estimate energy consumption

[1663] Input: The user inputs their daily activity level into the device.

[1664] Specific actions: The user enters information about their daily activities (commuting, exercise, etc.) into a form on the device and clicks the submit button.

[1665] Output: The device sends the input activity data to the server, which uses activity tracking software or an API to estimate energy expenditure based on the input data and stores the results in a database.

[1666] Step 5: Generate meal suggestions

[1667] Input: The server retrieves data on total calorie intake, nutrient amounts, and energy expenditure from a database.

[1668] Specific operation: The server reads the user's latest health and dietary information from the database and sends prompt sentences to the generative AI model.

[1669] Example prompt: "For a 30-year-old woman, please suggest a balanced dinner menu based on the following criteria: Breakfast: toast and salad; Lunch: boxed lunch and dessert; Daily exercise: light exercise during commute."

[1670] Output: Take the generated meal suggestions and format them as a menu with specific recipes, ingredients, serving sizes, and cooking instructions.

[1671] Step 6: Notification of proposal

[1672] Input: Generated meal suggestions

[1673] Specific operation: The server sends the generated meal suggestions in text and image format to the device, which then displays them on the user's screen.

[1674] Output: The user confirms and implements the received meal suggestions.

[1675] Step 7: Gather feedback

[1676] Input: The user consumes the suggested meal and then takes and uploads a photo of the meal after consumption.

[1677] Specific operation: After eating a meal, the user takes a photo and uploads it to the system. The device then sends the photo data to the server.

[1678] Output: The server passes the received image back to the image analysis engine and stores the analysis results in a database, thereby improving the accuracy of future meal suggestions.

[1679] (Application example 1)

[1680] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1681] Many users today struggle to manage their health and daily diet. Even when they receive healthy dietary recommendations, they lack concrete advice on how to translate them into real-world purchasing behavior. This creates a growing need for a system that can efficiently select the right ingredients and meal plans.

[1682] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1683] In this invention, the server includes means for inputting information such as the user's age, sex, height, weight, and health condition, means for uploading image data of meals consumed by the user, means for identifying the calories and nutrients of the meals using an image analysis engine, means for estimating the user's energy expenditure, means for generating meal suggestions based on the user's calorie and nutritional needs using a generative AI model, means for notifying the user of the generated meal suggestions, and means for displaying the suggested ingredients and menus when the user visits a physical store. This makes it easier for users to reflect healthy meal suggestions in their actual purchasing behavior, enabling more effective health management.

[1684] "User information" refers to basic physical data about each individual user, such as age, gender, height, weight, and health status.

[1685] "Meal image data" refers to digitally saved photographs of meals eaten by a user.

[1686] An "image analysis engine" is software or algorithm that analyzes image data to identify the calories and nutrients of a meal.

[1687] "Energy expenditure" means the total amount of calories burned through a user's daily activities and exercise.

[1688] A "generative AI model" is a model that uses artificial intelligence to generate personalized meal suggestions based on user data.

[1689] "Meal Suggestions" means suggestions that provide specific meals or menus to be consumed based on the user's calorie and nutritional needs.

[1690] "Notification means" refers to a method or device for notifying the user of the generated meal suggestions.

[1691] "Physical stores" refer to physical stores that users visit and use, such as grocery stores and restaurants.

[1692] "Means for displaying ingredients and menu items" refers to methods or devices that visually display suggested ingredients and menu items when a user visits a physical store.

[1693] A specific embodiment of a system for implementing the present invention will be described below. This system makes appropriate meal suggestions based on a user's individual health information and dietary data, and supports purchasing behavior, particularly in physical stores. This system involves inputting user information, analyzing dietary image data, estimating energy consumption, generating and notifying meal suggestions, and displaying ingredients and menus in physical stores.

[1694] System Overview

[1695] (1) User information registration:

[1696] Users log in to the system and enter personal data such as age, sex, height, weight, health condition, etc. through a terminal. The terminal sends this information to the server, which stores it in a database.

[1697] (2) Uploading meal information:

[1698] Users take photos of the food they eat and upload the image data to the system via their device, which then sends the images to the server, which then forwards them to the image analysis engine.

[1699] (3) Image analysis and calorie calculation:

[1700] The server uses an image analysis engine to identify the calories and key nutrients of the food from the uploaded image, and the calculations are stored in a database.

[1701] (4) Energy consumption estimation:

[1702] Users input their daily activity levels (e.g., commuting) into the system via their devices. The server estimates the total daily energy expenditure based on the user's activity level and stores the estimated results in a database.

[1703] (5) Meal suggestion generation:

[1704] Based on the user's total calorie intake, nutrient content, and energy expenditure data, the server uses a generative AI model to create appropriate meal suggestions, including specific recipes, ingredients, serving sizes, and cooking methods, and also provides advice on substitute ingredients from convenience stores and supermarkets.

[1705] (6) Notification of proposal:

[1706] The server sends the generated meal suggestions to the terminal, which displays them on the user's screen. The user can then confirm the suggested meals and act accordingly.

[1707] (7) Display in physical stores:

[1708] When a user visits a physical store, they use their device to scan the store's QR code, which allows the server to display ingredients and menu items to purchase in real time based on the user's meal suggestions.

[1709] (8) Feedback Collection:

[1710] After consuming the suggested meal, the user takes a photo of the meal and uploads it to the system via their device. The server receives the feedback and stores the analysis results in a database using an image analysis engine. This improves the accuracy of meal suggestions for future visits.

[1711] Examples:

[1712] For example, let's say User A, a 30-year-old woman, uses this system. After logging in, User A enters her age, gender, height, weight, and health status, and eats toast and salad for breakfast, and bibimbap bento and pudding for lunch. This information is uploaded to the system as image data. The server uses an image analysis engine to calculate the calorie intake for that day, and a generative AI model uses the energy expenditure from User A's commute to suggest dinner. Ultimately, a specific meal menu such as "grilled chicken salad, tofu vegetable soup, and brown rice" is suggested. User A goes to a nearby physical store and scans the store's QR code to check the ingredients and menu items to purchase on her smartphone. She then purchases the ingredients based on the suggestions and cooks. After eating, User A uploads an image of the menu she actually ate back to the system, contributing to improving the accuracy of suggestions from next time onwards.

[1713] Example prompt sentence:

[1714] User ID: 12345

[1715] Remaining calorie intake: 600

[1716] Suggest a suitable dinner menu based on the following criteria:

[1717] 1. Calories: 600 kcal or less

[1718] 2. Health status: Healthy adult female

[1719] 3. Nutritional balance: high protein, low carbohydrate

[1720] Suggested menu items should include specific ingredients, portions, and cooking methods.

[1721] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1722] Step 1:

[1723] Users log in to the device and enter information such as their age, gender, height, weight, and health condition. The device receives this information and sends it to the server, which then stores the received user information in a database.

[1724] Input: User information (age, gender, height, weight, health condition)

[1725] Output: User information stored in the database

[1726] Specific operation: When a user fills in information in the application's input form and presses the "Save" button, the terminal sends it to the server, which then stores it in the database.

[1727] Step 2:

[1728] Users take photos of the food they eat and upload the image data to the system via their device. The device sends these images to the server, which then forwards them to the image analysis engine.

[1729] Input: Food image data

[1730] Output: Image data sent to the server

[1731] How it works: When a user takes a photo of their meal and presses the "upload" button in the application, the device sends the image data to the server, which then passes it to the image analysis engine.

[1732] Step 3:

[1733] The server uses an image analysis engine to identify the calories and key nutrients of the food from the uploaded image, and the analysis results are stored in a database.

[1734] Input: Food image data

[1735] Output: Calorie and macronutrient data for the meal

[1736] Specific operation: The image analysis engine inputs image data into an algorithm and performs analysis. The analysis results are stored in a database via a server.

[1737] Step 4:

[1738] Users input their daily activity levels (e.g., commuting) into the system via their devices. The server receives the information and estimates the user's total daily energy expenditure based on their activity level. This estimation is then stored in a database.

[1739] Input: Activity data (commuting, exercise, etc.)

[1740] Output: Estimated energy consumption

[1741] How it works: After a user fills out the application's activity recording form and presses the "Submit" button, the device sends the data to the server, which then uses it to calculate activity levels.

[1742] Step 5:

[1743] The server uses a generative AI model to generate appropriate meal suggestions based on the user's total calorie intake, nutrient content, and energy expenditure data, including specific recipes, ingredients, serving sizes, and cooking methods.

[1744] Input: Total calories, nutrient content, energy expenditure

[1745] Output: Specific meal suggestions (recipe, ingredients, portions, cooking method)

[1746] How it works: The server inputs this data into a generative AI model to generate optimal meal suggestions for the user.

[1747] Step 6:

[1748] The server sends the generated meal suggestions to the terminal, which displays them on the user's screen. The user can then confirm the suggested meals and act accordingly.

[1749] Input: Meal suggestion data

[1750] Output: Meal suggestions displayed on the user's device

[1751] Specific operation: The server generates meal suggestions and sends them to the device, which receives them and displays them on the application.

[1752] Step 7:

[1753] When a user visits a physical store, they use their device to scan the store's QR code, and the server displays ingredients and menu items to purchase in real time based on the user's meal suggestions.

[1754] Input: QR code from physical store, suggested meal data

[1755] Output: List of ingredients and menu items to purchase

[1756] How it works: The user scans a QR code at a store and sends the information to the server, which then generates a shopping list based on the user's meal suggestions and displays it on the device.

[1757] Step 8:

[1758] After consuming the suggested meal, the user takes a photo of the meal and uploads it to the system via their device. The server receives the feedback and stores the analysis results in a database using an image analysis engine.

[1759] Input: Image data of the menu you ate

[1760] Output: Parsed feedback data

[1761] How it works: The user takes a photo of the menu item they ate and uploads it to the server through the application. The server passes the data to the image analysis engine and stores the analysis results in a database.

[1762] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1763] A specific embodiment of a system for implementing the present invention will be described below. This system recognizes the user's emotional state in addition to their individual health information and dietary data to make meal suggestions. The overall configuration of the system consists of the steps of processing the user's input information, making meal suggestions based on the analysis results using a generative AI model and emotion engine, and notifying the user and collecting feedback.

[1764] System Overview

[1765] Registering user information

[1766] Users log in to the system and enter personal data such as age, gender, height, weight, and health status. The terminal receives this information and sends it to the server, which then stores the received user information in a database.

[1767] Collecting Emotional Data

[1768] Users can input data about their physical condition and mood, or communicate their emotions to the system using facial and voice recognition technology. The device receives this emotional data and sends it to the server, which then transfers it to the emotion engine.

[1769] Uploading meal information

[1770] Users take photos of the food they eat before and after eating and upload the image data to the system via their device. The device sends these images to the server, which receives the image data and transfers them to the image analysis engine.

[1771] Image analysis and calorie counting

[1772] The server uses an image analysis engine to identify the calories and macronutrients of the food from the uploaded image. The server then aggregates the calorie and nutrient information to calculate the total calories and macronutrient intake of the user that day. The calculation results are stored in a database.

[1773] Energy Expenditure Estimation

[1774] Users input their daily activity levels (e.g., commuting) into the system via their device. The device then sends the information to the server, which then estimates the user's total daily energy expenditure based on their activity level. The estimated results are stored in a database.

[1775] Emotion analysis and reflection

[1776] The server uses an emotion engine to analyze the user's current emotional state from the emotion data. The analysis results are used to adjust subsequent meal suggestions. For example, if the user is feeling stressed, the suggestions will include ingredients with a relaxation effect.

[1777] Generating meal suggestions

[1778] The server uses a generative AI model to create appropriate meal suggestions based on the user's total calorie intake, nutrient content, energy expenditure, and emotional state. The generative AI model generates meal menus that take into account the user's calorie and nutritional needs as well as their emotional state. The menus include specific recipes, ingredients, serving sizes, and cooking methods. It also provides advice on convenience store substitutions.

[1779] Notification of proposal details

[1780] The server compiles the generated meal suggestions into images and text format and sends them to the device. The device displays the received meal suggestions on the user's screen. The user checks the displayed suggestions and prepares dinner accordingly.

[1781] Gathering feedback

[1782] After consuming the suggested meal, the user takes a photo of the meal they actually ate and uploads it to the system via their device. The device then sends the image to the server. The server receives feedback from the user and stores the analysis results in a database using an image analysis engine. This leads to improved accuracy of meal suggestions for future visits.

[1783] Specific examples

[1784] For example, let's say User B, a 30-year-old woman, uses the system. After logging in, User B enters her age, gender, height, weight, and health condition, and also enters emotional data about the stress she is currently feeling. She uploads image data to the system, showing that she had oatmeal and fruit for breakfast and a sandwich and salad for lunch. The server uses an image analysis engine to calculate the calories consumed that day, and uses a generative AI model to suggest dinner based on User B's energy expenditure during her commute. Additionally, based on her emotional state analyzed by the emotion engine, it suggests relaxing herbal tea and meals rich in B vitamins.

[1785] In this way, the system provides appropriate and effective health management and dietary recommendations based on the user's individual needs and emotional state.

[1786] The processing flow will be explained below.

[1787] Step 1:

[1788] Registering user information

[1789] User: Logs into the system and enters personal data such as age, gender, height, weight, and health status.

[1790] Terminal: Receives input information and sends it to the server.

[1791] Server: Stores the received user information in a database.

[1792] Step 2:

[1793] Collecting Emotional Data

[1794] User: Enters data about physical condition and mood into the device, and communicates emotions to the system using facial and voice recognition technology.

[1795] Terminal: Receives these emotion data and sends them to the server.

[1796] Server: Transfers the received emotion data to the emotion engine.

[1797] Step 3:

[1798] Uploading meal information

[1799] User: Takes pictures of meals such as breakfast, lunch, and dinner on the device and uploads them to the system.

[1800] Terminal: Sends the captured image data to the server.

[1801] Server: Transfers the received image data to the image analysis engine.

[1802] Step 4:

[1803] Image analysis and calorie counting

[1804] Server: Uses an image analysis engine to identify the calories and key nutrients of a meal from an uploaded image.

[1805] Server: Aggregates the identified calorie and nutrient information and calculates the total calories and macronutrients consumed by the user for the day.

[1806] Server: Saves the calculation results in a database.

[1807] Step 5:

[1808] Energy Expenditure Estimation

[1809] User: Enters daily activity data (commuting, exercise, etc.) on the device and sends it to the system.

[1810] Terminal: Sends the entered activity information to the server.

[1811] Server: Estimates total daily energy expenditure based on the user's activity level.

[1812] Server: Saves the estimation results in a database.

[1813] Step 6:

[1814] Emotion analysis and reflection

[1815] Server: Uses the emotion engine to analyze the user's current emotional state from the emotion data.

[1816] Server: The analysis results are stored and used to adjust future meal suggestions. For example, if the user is feeling stressed, the suggestions may include foods with a relaxing effect.

[1817] Step 7:

[1818] Generating meal suggestions

[1819] Server: Uses a generative AI model to create appropriate meal suggestions based on the user's total calorie intake, nutrient amounts, energy expenditure, and emotional state data.

[1820] Server: Runs the generative AI model, analyzes the user's calorie and nutritional needs, and generates a meal menu based on that.

[1821] Server: Generated meal suggestions include specific recipes, ingredients, serving sizes, and cooking methods, and also provide advice on convenience store substitutions.

[1822] Step 8:

[1823] Notification of proposal details

[1824] Server: The generated meal suggestions are compiled into images and text format and sent to the device.

[1825] Device: Displays the received meal suggestions on the user's screen.

[1826] User: Review the suggestions provided and prepare your meal accordingly.

[1827] Step 9:

[1828] Gathering feedback

[1829] User: After consuming the suggested meal, the user takes a photo of the menu item they actually ate with their device and uploads it to the system.

[1830] Terminal: Sends the captured image to the server.

[1831] Server: Receives user feedback and forwards it to the image analysis engine.

[1832] Server: The analysis results are saved in a database and used to improve the accuracy of future meal suggestions.

[1833] In this way, the system processes user input and emotional data to provide personalized meal recommendations, and can continuously improve the accuracy of these recommendations through feedback.

[1834] Example 2

[1835] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1836] In today's busy lifestyles, it is difficult for users to receive appropriate dietary recommendations based on their health status and emotions. Furthermore, there is no system that can provide dietary recommendations that take into account the user's emotional state, rather than just calories and nutrients. This leads to a delay in effective health management and dietary improvement.

[1837] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1838] In this invention, the server includes means for inputting information such as the user's age, sex, height, weight, and health condition, means for uploading image data of meals consumed by the user, means for identifying the calories and nutrients of the meals using an image analysis engine, means for estimating the user's energy expenditure, means for collecting the user's emotion data and analyzing it using an emotion engine, means for generating meal suggestions based on the user's calorie and nutritional needs and emotional state using a generative AI model, and means for notifying the user of the generated meal suggestions. This enables appropriate meal suggestions based on the user's health condition and emotions.

[1839] "User" refers to an individual who uses this system.

[1840] "Age" refers to the number of years since the user was born.

[1841] "Gender" refers to a user's biological or social sex.

[1842] "Height" refers to the vertical length of the user's body.

[1843] "Weight" refers to the mass of a user's body.

[1844] "Health Status" refers to the state of a User's physical and mental health.

[1845] "Information" refers to data such as the user's age, gender, height, weight, and health status.

[1846] "Means" refers to a method or device for achieving a particular purpose.

[1847] "Meal image data" refers to image files of meals consumed by the user.

[1848] "Upload" refers to the act of sending data from a user's device to a server.

[1849] "Image analysis engine" refers to software or hardware for analyzing image data and extracting specific information.

[1850] "Calories" refers to the amount of energy a food contains.

[1851] "Nutrients" are substances contained in food that are necessary for the growth and maintenance of health of the body.

[1852] "Energy Expenditure" refers to the total amount of energy consumed by a user's daily activities.

[1853] "Emotional data" refers to data related to the user's physical condition and mood.

[1854] "Emotion engine" refers to software or hardware for analyzing emotional data to identify a user's emotional state.

[1855] A "generative AI model" refers to an artificial intelligence system that generates appropriate meal suggestions based on user information.

[1856] "Notification" refers to the act of informing the user of the generated meal suggestions.

[1857] A "system" refers to a set of multiple means or devices that function in conjunction with one another.

[1858] A specific embodiment of a system for implementing the present invention will be described below. This system recognizes the user's emotional state as well as their individual health information and dietary data to make meal suggestions. This system processes the user's input information, makes meal suggestions based on the analysis results using a generative AI model and an emotion engine, and notifies the user and collects feedback.

[1859] Registering user information

[1860] Users log in to the system and enter personal data such as age, gender, height, weight, and health status. The terminal receives this information and sends it to the server, which then stores the received user information in a database.

[1861] Collecting Emotional Data

[1862] Users can input data about their physical condition and mood, or communicate their emotions to the system using facial and voice recognition technology. The device receives this emotional data and sends it to the server, which then transfers it to the emotion engine.

[1863] Uploading meal information

[1864] Users take photos of the food they eat before and after eating and upload the image data to the system via their device. The device sends these images to the server, which receives the image data and transfers them to the image analysis engine.

[1865] Image analysis and calorie counting

[1866] The server uses an image analysis engine to identify the calories and macronutrients of the food from the uploaded image. The server then aggregates the calorie and nutrient information to calculate the total calories and macronutrient intake of the user that day. The calculation results are stored in a database.

[1867] Energy Expenditure Estimation

[1868] Users input their daily activity levels (e.g., commuting) into the system via their device. The device then sends the information to the server, which then estimates the user's total daily energy expenditure based on their activity level. The estimated results are stored in a database.

[1869] Emotion analysis and reflection

[1870] The server uses an emotion engine to analyze the user's current emotional state from the emotion data. The analysis results are used to adjust subsequent meal suggestions. For example, if the user is feeling stressed, the suggestions will include ingredients with a relaxation effect.

[1871] Generating meal suggestions

[1872] The server uses a generative AI model to create appropriate meal suggestions based on the user's total calorie intake, nutrient content, energy expenditure, and emotional state. The generative AI model generates meal menus that take into account the user's calorie and nutritional needs as well as their emotional state. The menus include specific recipes, ingredients, serving sizes, and cooking methods. It also provides advice on convenience store substitutions.

[1873] Notification of proposal details

[1874] The server compiles the generated meal suggestions into images and text format and sends them to the device. The device displays the received meal suggestions on the user's screen. The user checks the displayed suggestions and prepares dinner accordingly.

[1875] Gathering feedback

[1876] After consuming the suggested meal, the user takes a photo of the meal they actually ate and uploads it to the system via their device. The device then sends the image to the server. The server receives feedback from the user and stores the analysis results in a database using an image analysis engine. This leads to improved accuracy of meal suggestions for future visits.

[1877] Specific examples

[1878] For example, suppose a 30-year-old female user uses the system. After logging in, the user enters her age, gender, height, weight, and health condition, and also enters emotional data about the stress she is currently experiencing. She uploads image data of her breakfast of oatmeal and fruit, and lunch of a sandwich and salad, to the system. The server uses an image analysis engine to calculate the calorie intake for that day, and uses a generative AI model to suggest dinner based on the user's energy expenditure from commuting. Based on the emotional state analyzed by the emotion engine, the system also suggests relaxing herbal teas and meals rich in B vitamins. In this way, the system provides appropriate and effective health management and dietary suggestions based on the user's individual needs and emotional state.

[1879] Prompt Sentence Examples

[1880] "A 30-year-old female user uses the system and enters her age, gender, height, weight, and health status. The user is currently stressed and ate oatmeal and fruit for breakfast, and a sandwich and salad for lunch. Please generate dinner suggestions taking into account her energy expenditure from her commute."

[1881] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1882] Step 1: Register your user information

[1883] Users log in to the system and enter personal data such as age, gender, height, weight, and health status.

[1884] Input: Age, gender, height, weight, health condition

[1885] Output: User information data (e.g., JSON format)

[1886] The terminal receives the entered information and transmits the data to the server.

[1887] Specific operation: The terminal encodes the input data into JSON format and sends it to the server using the HTTPS protocol.

[1888] The server stores the received data in a database.

[1889] Input: User information data (e.g., JSON format)

[1890] Output: User information record in database

[1891] Specific operation: The server analyzes the received data and performs an insert operation on a specific table in the database.

[1892] Step 2: Collecting emotion data

[1893] Users input data about their physical condition and mood, or communicate their emotions to the system using facial and voice recognition technology.

[1894] Input: Physical condition, mood data, or emotional data

[1895] Output: Emotion data (numerical and text format)

[1896] The terminal receives this data and transmits it to the server.

[1897] Specific operation: The device generates an API request to send emotion data to the server in real time.

[1898] The server transfers the received emotion data to the emotion engine for analysis.

[1899] Input: Received emotion data

[1900] Output: Parsed emotional state data

[1901] Specific operation: The server passes the emotion data to the emotion engine's API and receives the analysis results.

[1902] Step 3: Upload your meal information

[1903] Users take photos of the food they eat before and after eating and upload the image data to the system.

[1904] Input: Food image

[1905] Output: Image data file

[1906] The terminal transmits the image data to the server.

[1907] Specific operation: The device compresses the captured image and generates an API request to upload it to the server.

[1908] The server receives the image data and transfers it to the image analysis engine.

[1909] Input: Image data file

[1910] Output: Image analysis results (calories, nutrient data)

[1911] Specific operation: The server calls the API of the image analysis engine and sends a request to analyze the image data.

[1912] Step 4: Image analysis and calorie calculation

[1913] The server uses an image analysis engine to identify the calories and key nutrients of a meal from the uploaded image.

[1914] Input: Food image data

[1915] Output: Calorie information, nutrient information

[1916] How it works: The image analysis engine applies algorithms to recognize ingredients in an image and identify the calorie and nutritional information for each ingredient.

[1917] The server compiles the results and calculates the user's total calorie and nutrient intake.

[1918] Input: Calorie information and nutrient information for each meal

[1919] Output: Total calorie intake and macronutrient data

[1920] What it does: The server uses an aggregation algorithm to add up the calories and nutrients for each meal.

[1921] The calculation results are saved in a database.

[1922] Input: Total calorie intake and amount of major nutrients

[1923] Output: Accumulated data in the database

[1924] Specific operation: The server performs an insert operation on the results into a specific table in the database.

[1925] Step 5: Estimate energy consumption

[1926] Users enter their daily activity levels into the system.

[1927] Input: Activity data (commuting, exercise, etc.)

[1928] Output: Activity log

[1929] The terminal transmits the information to the server.

[1930] Specific operation: The device encodes the activity data and sends an API request to the server.

[1931] The server estimates total energy expenditure based on the user's activity level.

[1932] Input: Activity data

[1933] Output: Total energy consumption data

[1934] Specific operation: The server applies a calculation algorithm based on the activity data to estimate energy expenditure.

[1935] The estimation results are saved in a database.

[1936] Input: Total energy consumption data

[1937] Output: Energy consumption records in a database

[1938] Specific operation: The server inserts the estimation results into the database.

[1939] Step 6: Analyze and reflect on emotions

[1940] The server uses an emotion engine to parse the user's current emotional state from the emotion data.

[1941] Input: Emotion data

[1942] Output: Parsed emotional state

[1943] Specific operation: The emotion engine analyzes the emotion data and generates a quantified emotional state.

[1944] Configure settings to reflect the analysis results in meal suggestions.

[1945] Input: Parsed emotional state

[1946] Output: Adjusted meal suggestion data

[1947] What it does: The server applies an algorithm that adjusts the suggestions depending on the emotional state.

[1948] Step 7: Generate meal suggestions

[1949] The server uses a generative AI model to create appropriate meal suggestions based on the user's total calorie intake, nutrient amounts, energy expenditure, and emotional state data.

[1950] Input: User's total calorie intake, nutritional information, energy expenditure, emotional state data

[1951] Output: Generated meal suggestions

[1952] Specific operation: The generative AI model generates an appropriate meal menu based on the prompt text.

[1953] The generated suggestions include specific recipes, ingredients, quantities, and cooking methods.

[1954] Input: Generated meal suggestions

[1955] Output: Detailed meal menu (including recipe, ingredients, portions, and cooking method)

[1956] What it does: The server formats the results of the generative AI model and compiles them into a detailed menu.

[1957] Step 8: Notification of proposal

[1958] The server compiles the generated meal suggestions into image and text format and sends them to the terminal.

[1959] Input: Detailed meal menu

[1960] Output: Notification format

[1961] Specific operation: The server converts the menu information into image and text format and sends it to the terminal.

[1962] The device displays the received meal suggestions on the user's screen.

[1963] Input: Notification Format

[1964] Output: The suggestions that are displayed on the user's screen

[1965] Specific operation: The terminal obtains the notification data and displays it on the user interface.

[1966] The user reviews the displayed suggestions and prepares dinner accordingly.

[1967] Step 9: Gather feedback

[1968] After consuming the suggested meal, the user takes a photo of the meal they actually ate and uploads it to the system.

[1969] Input: Food image

[1970] Output: Uploaded image data

[1971] The terminal transmits the image data to the server again.

[1972] Specific operation: The device generates an API request to compress the image and send it to the server.

[1973] The server analyzes the feedback using an image analysis engine and stores the results in a database.

[1974] Input: Uploaded image data

[1975] Output: Analysis results

[1976] Specific operation: The server calls the API of the image analysis engine again and inserts the feedback results into the database.

[1977] (Application example 2)

[1978] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1979] Conventional health management systems often do not provide meal suggestions that take into account the user's emotional state, and comprehensive health management, including the user's mental health, has not been fully realized. Furthermore, virtual stores are unable to provide meal suggestions based on real-time health information, making it difficult for users to efficiently select ingredients and recipes that are suitable for them. This makes it difficult to maintain and improve users' health through appropriate diets.

[1980] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1981] In this invention, the server includes: means for inputting information such as the user's age, sex, height, weight, and health condition; means for uploading image data of the user's meals; means for identifying the calories and nutrients of the meals using an image analysis engine; means for estimating the user's energy expenditure; means for collecting and analyzing the user's emotional data; means for generating meal suggestions based on the user's calorie and nutritional needs and emotional state using a generative AI model; means for notifying the user of the generated meal suggestions; and means for providing meal suggestions in real time in a virtual store based on the user's health information, dietary data, and emotional state. This enables comprehensive health management that takes the user's emotional state into consideration and enables the selection of appropriate ingredients and recipes in real time in the virtual store.

[1982] "User Information" refers to personal information about a user, such as their age, gender, height, weight, and health status.

[1983] "Image data" refers to photographic data of the food consumed by the user.

[1984] The "image analysis engine" is software that identifies the calories and nutrients of meals from uploaded image data.

[1985] "Energy expenditure" is the amount of energy a user expends through daily activities.

[1986] "Emotion data" is data that indicates information about the user's physical condition and mood, and is obtained using facial recognition technology or voice recognition technology.

[1987] A "generative AI model" is an artificial intelligence model that generates meal suggestions based on a user's calorie and nutritional needs and emotional state.

[1988] "Meal suggestions" are suggestions such as specific meal menus, recipes, ingredients, portions, cooking methods, etc., that are proposed to users.

[1989] A "virtual store" is a digital platform that allows users to shop in a virtual space.

[1990] "Real-time" refers to data processing and information provision occurring immediately or nearly simultaneously.

[1991] "Notification" is the act or mechanism of communicating generated meal suggestions to a user.

[1992] "Feedback" means providing the system with information about the food the user has eaten, which helps improve the accuracy of the next suggestion.

[1993] MODE FOR CARRYING OUT THE INVENTION

[1994] A detailed description of a system for realizing the present invention is given below. This system makes meal suggestions in real time in a virtual store based on the user's individual health information, dietary data, and emotional state.

[1995] System Overview

[1996] Registering user information

[1997] After accessing the virtual store, users use their smartphones or head-mounted displays (HMDs) to input personal information such as age, gender, height, weight, and health status. The devices receive this information and send it to a cloud server, which then stores the received user information in a database.

[1998] Collecting Emotional Data

[1999] Users can input data about their physical condition and mood using a camera or microphone installed on their smartphone or HMD, or communicate their emotions to the system using facial or voice recognition technology. The device receives this emotional data and sends it to a cloud server. The server then analyzes the received emotional data using an emotion engine to determine the user's current emotional state.

[2000] Uploading meal information

[2001] Users take photos of the meals they have eaten over the past 48 hours with their smartphones and upload them to the system. The device sends these images to a cloud server. The server receives the image data and transfers it to an image analysis engine. The image analysis engine identifies the calories and nutrients of the meals from the uploaded images.

[2002] Generating meal suggestions

[2003] The server uses a generative AI model based on the user's total calorie intake, nutrient content, energy expenditure, and emotional state data to generate appropriate meal suggestions. The generative AI model considers the user's calorie and nutritional needs as well as their emotional state to generate a meal menu that includes specific recipes, ingredients, serving sizes, and cooking methods.

[2004] Notification of proposal details

[2005] The server compiles the generated meal suggestions into images and text format and sends them to the device. The device then displays the received meal suggestions on the user's smartphone or HMD at appropriate locations within the virtual store. As the user moves around the virtual store, notifications pop up at appropriate locations.

[2006] Gathering feedback

[2007] After consuming the suggested meal, the user takes a photo of the meal and uploads it to the system via their device. The device then sends the image to a cloud server. The server receives feedback from the user and stores the analysis results in a database using an image analysis engine. This improves the accuracy of meal suggestions for future visits.

[2008] Hardware and Software Used

[2009] Smartphone: Use an Android or iOS device.

[2010] Head-mounted display (HMD): Use Oculus Rift, HTC Vive, etc.

[2011] Cloud server: Use cloud platforms such as Google Cloud, AWS, and Azure.

[2012] Emotion recognition technology: Uses technologies such as OpenCV and TensorFlow.

[2013] Generative AI models: Utilize generative AI such as OpenAI GPT-3.

[2014] Specific examples

[2015] For example, a user visits a virtual store and enters their personal information using a smartphone. Emotional data is collected using facial recognition with a camera installed in the HMD, and data on physical condition and mood is collected. After the user uploads photos of the meals they have eaten in the past 48 hours, the cloud server uses an image analysis engine to identify calories and nutrients, and then uses a generative AI model to make meal recommendations.

[2016] Example prompt sentence:

[2017] Based on their health, users need 2000 kcal of calories, 50g protein, 70g fat, and 250g carbohydrates.

[2018] Please suggest a dish using the following ingredients and method:

[2019] 1) Breakfast: Oatmeal and fruit

[2020] 2) Lunch: Sandwich and salad

[2021] 3) Dinner: Chicken steak and vegetables

[2022] The user's emotional state is stress.

[2023] This allows users to select appropriate ingredients and recipes in real time in a virtual store, enabling comprehensive health management that takes into account their emotional state.

[2024] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2025] Step 1:

[2026] Users access the virtual store and use their smartphone or HMD to enter personal information such as age, gender, height, weight, and health status. This information is then sent from the device to a cloud server, which then stores the received information in a database.

[2027] Input: User's age, gender, height, weight, health status

[2028] Output: User information stored in the database

[2029] Step 2:

[2030] Users input data about their physical condition and mood into the system using a camera and microphone installed on their smartphone or HMD. The device receives this emotional data and sends it to a cloud server. The server then uses an emotion engine to analyze the received emotional data and identify the user's current emotional state.

[2031] Input: Data about the user's physical condition and mood

[2032] Output: Parsed user emotional state

[2033] Step 3:

[2034] Users take photos of the meals they have eaten over the past 48 hours with their smartphones and upload them to the system. The device sends these image data to a cloud server. The server receives the image data and transfers it to an image analysis engine. The image analysis engine identifies the calories and nutrients of the meals from the uploaded images.

[2035] Input: User's food photo

[2036] Output: Parsed calorie and nutrient information

[2037] Step 4:

[2038] The server uses a generative AI model to generate appropriate meal suggestions based on the user's total calorie intake, nutrient intake, energy expenditure, and emotional state. The generative AI model generates meal menus that take into account the user's calorie and nutritional needs as well as their emotional state.

[2039] Input: User's total calorie intake, nutrient amounts, energy expenditure, emotional state

[2040] Output: Generated meal suggestions

[2041] Step 5:

[2042] The server compiles the generated meal suggestions into images and text format and sends them to the device. The device then displays the received meal suggestions on the user's smartphone or HMD at appropriate locations within the virtual store. As the user moves around the virtual store, notifications pop up at appropriate locations.

[2043] Input: Generated meal suggestions

[2044] Output: Notification and display in the user's virtual space

[2045] Step 6:

[2046] After consuming the suggested meal, the user takes a photo of the meal and uploads it to the system via their device. The device then sends the image to a cloud server. The server receives feedback from the user and stores the analysis results in a database using an image analysis engine. This improves the accuracy of meal suggestions for future visits.

[2047] Input: Food photos provided as feedback

[2048] Output: Database update to improve future meal suggestions

[2049] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2050] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicati...

Claims

1. A means of inputting information such as the user's age, gender, height, weight, and health status; A means for users to upload image data of the meals they have eaten; a means for identifying calories and nutrients in a meal using an image analysis engine; a means for estimating the energy expenditure of a user; a means for generating meal suggestions based on a user's calorie and nutritional needs using a generative AI model; means for notifying the user of the generated meal suggestions; A system including:

2. The system of claim 1 , further comprising means for providing feedback to the user after the user consumes the suggested meal.

3. The system of claim 1 , further comprising means for including specific recipes, ingredients, serving sizes, and cooking methods in the generated meal suggestions.

Images