system

Abstract

We provide the system. [Solution] A means for communicating with a device for acquiring physical information, A means of identifying ingredients from captured images, A means for suggesting meals based on identified food ingredient information and the user's physical information, A means of identifying the necessary ingredients based on the proposed meal and indicating where to purchase them, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, and includes steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In modern society, there is a problem that many people have difficulty maintaining a healthy diet. In particular, there is a lack of means to easily obtain customized nutritional advice based on individual body information and activity levels. In addition, there is also a lack of means to easily manage and efficiently utilize the food ingredient information in the current refrigerator. Furthermore, it is also difficult to efficiently identify the places to purchase the necessary food ingredients, and these problems are generally hindering a healthy diet.

Means for Solving the Problems

[0005] This invention solves these problems by providing a system that includes means for communicating with a device for acquiring physical information, means for identifying food ingredients from captured images, means for suggesting meals based on the identified food ingredient information and the user's physical information, and means for identifying necessary ingredients based on the suggested meal and indicating where to purchase them.

[0006] Specifically, users can acquire physical information using wearable devices and link it with the application to always have access to the latest health data. Furthermore, users can take photos of their refrigerator and upload them to the application, allowing image recognition technology to automatically identify and list the ingredients inside. This makes it easy for users to manage their current food inventory. In addition, personalized meal suggestions are provided based on the identified ingredients and the user's health data using generative AI, and information on nearby stores or online shops is provided for necessary ingredients. This allows users to efficiently procure ingredients and easily maintain a healthy diet.

[0007] "Physical information" refers to data that indicates the user's health status and physical condition, such as the user's heart rate, steps taken, calories burned, weight, and height.

[0008] "Device" refers to electronic devices used to acquire or utilize a user's physical information, such as wearable devices and smartphones.

[0009] "Image recognition" is a technology that recognizes specific objects or information from a photograph and captures them as digital data.

[0010] "Food information" refers to data about the types and quantities of food items present in the refrigerator.

[0011] "User" refers to an individual who uses this system.

[0012] "Meal suggestions" is a process that suggests appropriate meal menus based on the user's physical information and ingredient information.

[0013] "Place of purchase" refers to a physical store or online store where the ingredients needed for the suggested meal menu can be purchased.

[0014] A "system" is a collective term for multiple devices and programs that perform a set of functions necessary to acquire physical information, identify ingredients, provide personalized meal suggestions, and indicate where to purchase ingredients. [Brief explanation of the drawing]

[0015] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11]It is a sequence diagram showing the processing flow of the data processing system in Embodiment 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Embodiment 2 when the emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when the emotion engine is combined.

Mode for Carrying Out the Invention

[0016] Hereinafter, an example of an embodiment of the system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0019] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0020] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0021] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0023] [First Embodiment]

[0024] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0025] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0028] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0031] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0035] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0036] This invention relates to a system that acquires a user's physical information, identifies food ingredients from captured images, makes meal suggestions based on this information, and indicates where to purchase the necessary ingredients. The following describes a specific form for realizing this system.

[0037] User registration and profile settings

[0038] 1. User: Install the application and, upon first launch, enter your name, age, gender, height, weight, and information about allergies, preferred foods, and health goals. This information will be used to create your individual profile.

[0039] 2. Server: Saves the entered information to the database and creates a profile for each user.

[0040] Integration with wearable devices

[0041] 1. User: Link the wearable device and application in the settings screen.

[0042] 2. Terminal: Regularly receives data such as heart rate, steps taken, and calories burned from wearable devices.

[0043] 3. Server: Saves received data to the database in real time and monitors the user's health status.

[0044] Making a list of ingredients in the refrigerator

[0045] 1. User: Take a photo of the inside of the refrigerator with your smartphone and upload it to the application.

[0046] 2. Terminal: Uses image recognition technology to identify ingredients in a photograph and generates an ingredient list.

[0047] 3. Server: Stores the ingredient list in a database and manages it as the user's current ingredient inventory information.

[0048] Specific example

[0049] User: Upload a photo of the refrigerator to the application.

[0050] Terminal: Analyzes the photo and identifies "tomato, carrot, and chicken breast."

[0051] Server: Adds these identified ingredients to the user's inventory list and saves them as current inventory information.

[0052] Generating customized recipes and menus

[0053] 1. Server: Uses generative AI to create a customized weekly meal plan based on the user's health data, ingredient inventory list, and preferences.

[0054] 2. Terminal: Presents the generated menu and necessary nutrient and calorie information to the user.

[0055] Suggestions for places to buy groceries

[0056] 1. Server: Based on the proposed menu, list the necessary ingredients and identify any missing ingredients.

[0057] 2. Server: Searches for potential grocery purchases from partner e-commerce platforms and generates links to the nearest stores or online shops.

[0058] 3. Terminal: Presents the generated list of potential purchases to the user and provides a link to the online store.

[0059] Specific example

[0060] Server: The server generates a weekly menu using tomatoes, carrots, and chicken breast, and suggests recipes such as "Tomato Chicken Stew."

[0061] Terminal: Displays these recipes, a list of required ingredients, and cooking instructions to the user.

[0062] Health data feedback

[0063] 1. Server: Collects data from wearable devices and food records to analyze the user's health status.

[0064] 2. Server: Generates health feedback and advice based on analysis results, and creates notification messages.

[0065] 3. Device: Send the generated feedback and advice to the user via push notification.

[0066] Specific example

[0067] Server: Verify that the user's activity level has not reached 70% of the target.

[0068] Device: Notifies the user with the message, "Walk another 1000 steps to reach your goal."

[0069] The above describes the embodiments of the present invention. This system allows users to receive customized meal suggestions based on their individual health data and food information, enabling them to efficiently purchase food ingredients and lead a healthy lifestyle.

[0070] The following describes the processing flow.

[0071] User registration and profile settings

[0072] Step 1:

[0073] User: Install the application and, upon first launch, enter your name, age, gender, height, weight, and information about allergies, preferred foods, and health goals.

[0074] Step 2:

[0075] Terminal: Sends the entered information to the server.

[0076] Step 3:

[0077] Server: Stores received information in a database and creates user profiles.

[0078] Integration with wearable devices

[0079] Step 4:

[0080] User: In the application's settings screen, link the wearable device with the application.

[0081] Step 5:

[0082] Device: Once the connection is successfully completed, enable the setting to periodically receive data such as heart rate, steps, and calories burned from the wearable device.

[0083] Step 6:

[0084] Server: Receives device data sent from terminals and stores it in the database.

[0085] Making a list of ingredients in the refrigerator

[0086] Step 7:

[0087] User: Take a photo of the inside of the refrigerator with your smartphone and upload it to the application.

[0088] Step 8:

[0089] Device: Uses AI image recognition technology to identify ingredients in a photograph and creates an ingredient list.

[0090] Step 9:

[0091] Terminal: Sends the generated ingredient list to the server.

[0092] Step 10:

[0093] Server: Stores the received ingredient list in the database and manages it as the user's current inventory information.

[0094] Generating customized recipes and menus

[0095] Step 11:

[0096] Server: Uses generative AI to create a customized weekly meal plan based on the user's health data, ingredient inventory list, preferences, and allergy information.

[0097] Step 12:

[0098] Server: Sends the generated menu to the terminal.

[0099] Step 13:

[0100] Terminal: Displays the received menu and necessary nutrient and calorie information to the user.

[0101] Suggestions for places to buy groceries

[0102] Step 14:

[0103] Server: Identify any missing ingredients based on the proposed menu.

[0104] Step 15:

[0105] Server: Searches for potential purchase options from partner e-commerce platforms for missing ingredients, gathering information on the nearest places to buy them and online stores.

[0106] Step 16:

[0107] Server: Sends a list of potential purchases to the terminal.

[0108] Step 17:

[0109] Terminal: Presents a generated list of potential purchases to the user and provides a link to the online store.

[0110] Health data feedback

[0111] Step 18:

[0112] Server: Collects data from wearable devices and food records to analyze the user's health status.

[0113] Step 19:

[0114] Server: Generates health feedback and advice based on analysis results, and creates notification messages.

[0115] Step 20:

[0116] Server: Sends the generated feedback and advice to the terminal.

[0117] Step 21:

[0118] Device: Send feedback and advice to users via push notifications.

[0119] The above describes the detailed operation at each processing step. Through this detailed processing flow, users can always receive optimal meal suggestions based on the latest health data and ingredient information, efficiently purchase ingredients, and lead a healthy lifestyle.

[0120] (Example 1)

[0121] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0122] Maintaining a healthy diet is difficult in today's busy lifestyle. Furthermore, it's challenging for users to select appropriate foods and recipes based on their own health status and preferences. Even efficiently shopping for ingredients by identifying suitable locations is time-consuming. To address these issues, a system is needed that centrally manages users' health data and ingredient information, and provides optimal meal suggestions and ingredient purchasing locations based on this data.

[0123] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0124] In this invention, the server includes means for inputting the user's personal information and creating a profile; means for working with a wearable device to collect biometric data; means for analyzing images of the inside of a refrigerator taken and identifying ingredients; means for creating a customized menu using a generative AI model based on the identified ingredient information and the user's health data; and means for identifying missing ingredients based on the proposed menu and suggesting where to purchase them. As a result, the user can receive meal suggestions based on their health status and ingredient information, efficiently purchase necessary ingredients, and maintain a healthy lifestyle.

[0125] A "user" is an entity that uses the system to manage individual health data and food information, and to make meal suggestions and purchase food ingredients.

[0126] A "profile" is a set of individual information that includes the user's name, age, gender, height, weight, allergy information, preferred foods, and health goals.

[0127] A "wearable device" is an electronic device worn by a user to collect biometric data such as heart rate, steps taken, and calories burned.

[0128] "Biometric data" refers to data such as a user's heart rate, steps taken, and calories burned, collected through wearable devices and other means.

[0129] "Image analysis" is the process of extracting specific information from captured images, and is particularly used to identify food items inside a refrigerator.

[0130] "Food identification" is a method of identifying food items in a refrigerator through image analysis and listing that information.

[0131] A "generative AI model" is an artificial intelligence model that automatically generates customized menus based on the user's health data and ingredient information.

[0132] A "menu" refers to a meal plan or recipe suggested based on the user's health data and available ingredients.

[0133] "Place of purchase" refers to any location where you can purchase any missing ingredients based on the suggested menu, including online stores and physical stores.

[0134] "Health data" is a general term for a user's physical information, biometric data, and dietary history, and is used to manage their health status.

[0135] "Feedback" refers to improvement suggestions and advice provided based on the user's health status and activity level.

[0136] This invention relates to a system that acquires a user's physical information, identifies food ingredients from captured images, makes meal suggestions based on this information, and indicates where to purchase the necessary ingredients. Specific embodiments of this invention are described below.

[0137] User registration and profile settings

[0138] 1. User: First, the user installs the system's application on their smartphone. Upon first launching the application, they enter their name, age, gender, height, weight, allergy information, preferred foods, and health goals. This creates the user's individual profile.

[0139] 2. Server: The server stores the information entered by the user in a database and creates a profile for each user. A database such as MongoDB is used for this purpose.

[0140] Integration with wearable devices

[0141] 1. User: The user connects the wearable device and the application using Bluetooth in the application's settings screen.

[0142] 2. Terminal: The terminal periodically receives biometric data such as heart rate, steps taken, and calories burned from the wearable device. Bluetooth communication is used for this purpose.

[0143] 3. Server: The server stores incoming data in a database in real time and monitors the user's health status. A database such as PostgreSQL is used for this purpose.

[0144] Making a list of ingredients in the refrigerator

[0145] 1. User: The user takes a photo of the inside of the refrigerator with their smartphone and uploads it to the application.

[0146] 2. Terminal: The terminal uses image recognition technology (e.g., TENSORFLOW®) to identify ingredients in captured images and generates an ingredient list.

[0147] 3. Server: The server stores the ingredient list in a database and manages it as the user's current ingredient inventory information. A database such as MySQL® is used for this purpose.

[0148] Specific example

[0149] When a user uploads a photo of their refrigerator to the application, the device analyzes the photo and identifies "tomatoes, carrots, and chicken breast." The server adds these identified ingredients to the user's inventory list and saves it as current inventory information.

[0150] Generating customized recipes and menus

[0151] 1. Server: The server uses a generative AI model (e.g., OpenAI®'s GPT) to create a customized weekly meal plan based on the user's health data, ingredient inventory list, and preferences.

[0152] 2. Terminal: The terminal presents the user with the generated menu and information on necessary nutrients and calories.

[0153] Example of a prompt

[0154] Send the following prompt to the generating AI: "Create a one-week meal plan based on User A's health data and inventory information."

[0155] Suggestions for places to buy groceries

[0156] 1. Server: The server identifies any missing ingredients based on the proposed menu and suggests where to purchase them.

[0157] 2. Server: The server searches for potential grocery purchases from partner e-commerce platforms and generates links to the nearest purchase locations and online stores.

[0158] 3. Terminal: The terminal presents the user with a generated list of potential purchases and provides links to online stores.

[0159] Specific example

[0160] The server generates a week's worth of meal plans based on the user's health data and food inventory, suggesting recipes such as "Tomato Chicken Stew." The terminal displays these recipes, a list of required ingredients, and cooking instructions to the user.

[0161] Health data feedback

[0162] 1. Server: The server aggregates data from wearable devices and food records, and analyzes the user's health status.

[0163] 2. Server: The server generates health feedback and advice based on the analysis results and creates notification messages.

[0164] 3. Device: The device will send the generated feedback and advice to the user via push notifications.

[0165] Specific example

[0166] The server confirms that the user's activity level has not reached 70% of the target, and the device notifies the user, "Walk another 1000 steps to reach your goal."

[0167] The above describes the embodiments of the present invention. This system allows users to receive customized meal suggestions based on their health condition and food information, enabling them to efficiently purchase food ingredients and lead a healthy lifestyle.

[0168] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0169] Step 1:

[0170] User: Install the application on your smartphone and enter your personal information (name, age, gender, height, weight, allergy information, favorite foods, health goals) upon first launch.

[0171] Input: User's personal information

[0172] Output: Individual user profiles

[0173] Specific action: The user enters information into each field and presses the "Save" button.

[0174] Step 2:

[0175] Server: Stores the entered personal information in the database and creates a profile for each user.

[0176] Input: User's personal information

[0177] Output: Saved user profile

[0178] Specific operation: The server uses MongoDB to store personal information received from users in JSON format.

[0179] Step 3:

[0180] User: Link the wearable device and application in the settings screen.

[0181] Input: Connection information for wearable devices

[0182] Output: Connection established with wearable device

[0183] Specific steps: The user opens the Bluetooth settings, selects the wearable device, and presses the pairing button.

[0184] Step 4:

[0185] Terminal: Regularly receives data such as heart rate, steps taken, and calories burned from wearable devices.

[0186] Input: Data from a wearable device

[0187] Output: Biometric data temporarily stored on the device

[0188] Specific operation: The device uses Bluetooth communication to retrieve data and save it to local storage.

[0189] Step 5:

[0190] Server: Receives data from wearable devices via terminals and saves it to a database in real time.

[0191] Input: Biometric data transmitted from the device

[0192] Output: Biometric data stored in the database

[0193] Specific operation: The server uses PostgreSQL to execute INSERT statements into the database.

[0194] Step 6:

[0195] User: Take a photo of the inside of the refrigerator with your smartphone and upload it to the application.

[0196] Input: Image of the inside of a refrigerator

[0197] Output: Image files uploaded to the application

[0198] Specific action: The user takes a photo using the camera app and presses the upload button in the application.

[0199] Step 7:

[0200] Terminal: Uses image recognition technology to identify ingredients in captured images and generates an ingredient list.

[0201] Input: Uploaded image of the inside of a refrigerator

[0202] Output: List of identified ingredients

[0203] Specific operation: The device uses TensorFlow to perform image analysis and identify food ingredients. Example: "Tomato, carrot, chicken breast".

[0204] Step 8:

[0205] Server: Stores ingredient lists in a database and manages them as the user's current ingredient inventory information.

[0206] Input: Identified ingredient list

[0207] Output: Food inventory information stored in the database

[0208] Specific operation: The server uses MySQL to store ingredient information in a database.

[0209] Step 9:

[0210] Server: Uses a generative AI model to create a customized weekly meal plan based on the user's health data, ingredient inventory list, and preferences.

[0211] Input: User's health data, food inventory list, preferences

[0212] Output: Customized weekly meal plan

[0213] Specific operation: The server sends the following prompt to the generative AI model: "Create a weekly meal plan based on user A's health data and inventory information."

[0214] Step 10:

[0215] Terminal: Displays the generated menu and necessary nutrient and calorie information to the user.

[0216] Input: Customized menu

[0217] Output: Menu information presented to the user

[0218] Specific action: The device displays the menu in a list format on the app's UI.

[0219] Step 11:

[0220] Server: Based on the proposed menu, it identifies any missing ingredients and searches for information to suggest where to purchase them.

[0221] Input: Customized menu

[0222] Output: Information on missing ingredients and where to purchase them.

[0223] Specific operation: The server analyzes the menu data, identifies missing ingredients, and searches for suppliers from affiliated e-commerce platforms.

[0224] Step 12:

[0225] Terminal: Presents a generated list of potential purchases to the user and provides a link to the online store.

[0226] Input: Purchase candidate list

[0227] Output: A list of purchase options and links presented to the user.

[0228] Specific action: The device displays a list of ingredients containing a "purchase link" within the app.

[0229] Step 13:

[0230] Server: Collects data from wearable devices and food records to analyze the user's health status.

[0231] Input: Wearable device and meal record data

[0232] Output: Analysis results

[0233] Specific operation: The server uses a Python script to aggregate and analyze data.

[0234] Step 14:

[0235] Server: Generates health feedback and advice based on analysis results, and creates notification messages.

[0236] Input: Analysis results

[0237] Output: Feedback and advice

[0238] Specific operation: The server checks the user's unfulfilled goal and generates a message such as, "You are 1000 steps away from achieving your goal."

[0239] Step 15:

[0240] Device: Generated feedback and advice are sent to the user via push notifications.

[0241] Input: Feedback and advice

[0242] Output: Push notification to the user

[0243] Specific action: The device sends a notification using the smartphone's push notification function.

[0244] The above outlines the specific processing steps of this system's program. This allows users to receive personalized meal suggestions based on their individual health data and ingredient information, enabling them to efficiently purchase ingredients and manage their health.

[0245] (Application Example 1)

[0246] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0247] In modern times, efficient and healthy dietary management is a crucial challenge for many people. However, in our busy lives, it's not easy to select appropriate recipes based on individual health conditions and current food availability, and to find suppliers for necessary ingredients. Furthermore, there is a lack of efficient feedback systems that link health data and dietary data. To address these challenges, there is a need for a system that utilizes users' health data, proposes individually customized recipes based on ingredient information, and allows for the quick acquisition of necessary ingredients.

[0248] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0249] In this invention, the server includes means for communicating with a device for acquiring physical information, means for identifying ingredients from captured images, means for suggesting meals based on the identified ingredient information and the user's physical information, means for generating customized recipes based on the user's health data and ingredient information using a generative AI model, and means for identifying missing ingredients and suggesting purchases in cooperation with partner food delivery services. This enables the user to smoothly manage their diet efficiently and healthily.

[0250] 1. A "device for acquiring physical information" refers to a device that uses wearable devices, smartphones, etc., to acquire health data such as the user's heart rate, steps taken, and calories burned.

[0251] 2. "Means of communication" refers to methods and protocols for sending and receiving data over a network.

[0252] 3. "Means for identifying food ingredients from captured images" refers to technologies that use image recognition technology to identify food ingredients from photographs taken with a camera.

[0253] 4. "Identified food information" refers to information such as the name and quantity of food items identified through image recognition.

[0254] 5. "User's physical information" refers to health-related data such as the user's age, gender, heart rate, and steps taken.

[0255] 6. "Means of providing meal suggestions" refers to a method of generating appropriate meal menus and recipes based on the user's physical information and identified ingredient information.

[0256] 7. A "generative AI model" refers to an artificial intelligence model that generates customized recipes based on the user's health data and ingredient information.

[0257] 8. A "customized recipe" refers to a specific recipe generated based on the user's individual health condition and ingredient availability.

[0258] 9. "Means of identifying missing ingredients" refers to techniques for identifying ingredients that are not present in the refrigerator based on the proposed recipe.

[0259] 10. "Partner food delivery services" refers to service providers that allow customers to order ingredients and food online and have them delivered.

[0260] This invention is a system that suggests customized recipes based on the user's physical information and food information, and indicates where to purchase the necessary ingredients. This system is implemented using the following hardware and software.

[0261] Hardware and software to be used

[0262] 1. Hardware

[0263] Wearable devices: Devices that acquire physical information such as the user's heart rate, steps taken, and calories burned.

[0264] Smartphone: A device used to take pictures of food items inside the refrigerator and upload them to an application.

[0265] 2. Software

[0266] Image recognition technology: This technology identifies food items from images taken with a smartphone. An example is the Python library `face_recognition`.

[0267] Generative AI Model: An artificial intelligence model that generates customized recipes based on the user's health data and ingredient information. It utilizes APIs such as OpenAI's GPT-4®.

[0268] System Operation Description

[0269] User registration and profile settings

[0270] Users install the application and, upon first launch, enter their name, age, gender, height, weight, allergy information, preferred foods, and health goals. This allows the user's individual profile to be saved on the server.

[0271] Integration with wearable devices

[0272] The server periodically receives data such as heart rate, steps taken, and calories burned from wearable devices and stores it in a database in real time. This allows the user's health status to be constantly monitored.

[0273] Making a list of ingredients in the refrigerator

[0274] The user takes a photo of the inside of their refrigerator with their smartphone and uploads it to the application. The server uses image recognition technology to identify the food items in the photo and generates a list of those items. These identified items are added to the user's inventory list and managed as current inventory information.

[0275] Generating customized recipes and menus

[0276] The server uses a generative AI model to create a customized weekly meal plan based on the user's health data, ingredient inventory list, and preferences. The generated meal plan, along with necessary nutrient and calorie information, is presented to the user via their smartphone.

[0277] Specific example:

[0278] If the user's health condition has not reached the target health condition, utilize the generative AI model to propose recipes.

[0279] Example of the prompt text to be used:

[0280] User information: {'name': 'User 1', 'age': 30, 'gender':'male', 'height': 175, 'weight': 70, 'allergies': ['dairy products'], 'preferences': ['vegetables', 'chicken']}

[0281] Health data: {'heart_rate': 70,'steps': 8000, 'calories_burned': 500}

[0282] Ingredients in the refrigerator: ['tomato', 'carrot', 'chicken']

[0283] Based on these, please propose customized recipes.

[0284] Proposal for ingredient purchase locations

[0285] The server lists the required ingredients based on the proposed menu and identifies the missing ingredients. Next, it searches for ingredient purchase candidates from the partnered food delivery service and generates links to the nearest purchase locations and online stores. These lists of purchase candidates are presented to the user through the smartphone, and links to the online stores are provided.

[0286] Feedback on health data

[0287] The server aggregates the data from wearable devices and meal records and analyzes the user's health condition. Then, based on the analysis results, it generates health feedback and advice and creates notification messages. The generated feedback and advice are communicated to the user via push notification through the smartphone terminal.

[0288] This allows users to quickly receive personalized meal suggestions based on their health data and ingredient information, enabling them to efficiently purchase ingredients and live a healthy lifestyle.

[0289] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0290] Step 1:

[0291] The user installs the application and sets up a profile upon first launch.

[0292] Input: Name, age, gender, height, weight, allergy information, favorite foods, health goals.

[0293] Data processing: Use this data to generate individual user profiles.

[0294] Output: The user's profile is saved to the server's database.

[0295] Step 2:

[0296] The server periodically receives health data from wearable devices.

[0297] Input: Data such as heart rate, steps taken, and calories burned, transmitted from a wearable device.

[0298] Data processing: Received data is recorded in a database in real time to monitor the user's health status.

[0299] Output: The latest health data will be updated.

[0300] Step 3:

[0301] Users take photos of the food in their refrigerator with their smartphones and upload the images to the app.

[0302] Input: A photo of the ingredients in the refrigerator.

[0303] Data processing: The server uses image recognition technology to identify the ingredients in the photo.

[0304] Output: A list of recognized ingredients is generated and saved in the database.

[0305] Step 4:

[0306] The server generates a customized recipe based on the user's health data and the ingredient inventory list.

[0307] Input: Prompt text (user profile, health data, identified ingredient list) for the generation AI model.

[0308] Data processing: Use a generation AI model (e.g., GPT-4) to generate a customized recipe.

[0309] Output: The generated recipe and the necessary nutrient and calorie information are generated.

[0310] Step 5:

[0311] The server lists up the necessary ingredients based on the proposed recipe and identifies the missing ingredients.

[0312] Input: The generated recipe and the current ingredient inventory information.

[0313] Data processing: Identify and list up the missing ingredients.

[0314] Output: A list of missing ingredients is generated.

[0315] Step 6:

[0316] The server searches for grocery purchase options from partner food delivery services and generates links to the nearest purchase locations and online stores.

[0317] Input: List of missing ingredients.

[0318] Data processing: Search the database of food delivery services to obtain information on potential purchases.

[0319] Output: A list of potential purchases and links to online stores are generated.

[0320] Step 7:

[0321] The device presents the user with a generated recipe, information on necessary nutrients and calories, and a list of recommended purchases.

[0322] Input: Generated recipe, required nutrient and calorie information, a list of suggested purchases, and links to online stores.

[0323] Data processing: Converts information into a format that is easy for users to understand.

[0324] Output: It will be displayed on the smartphone screen.

[0325] Step 8:

[0326] The server aggregates data from wearable devices and food records to analyze the user's health status.

[0327] Input: User's health data, dietary record data.

[0328] Data processing: Use analytical algorithms to evaluate the user's health status.

[0329] Output: Health feedback and advice are generated.

[0330] Step 9:

[0331] The device will send the generated feedback and advice to the user via push notifications.

[0332] Input: Feedback and advice sent from the server.

[0333] Data processing: Convert to a format that can be displayed as a push notification.

[0334] Output: It will be displayed as a push notification on your smartphone.

[0335] Through the above processing steps, users can quickly obtain customized meal suggestions based on their health data and food information, enabling them to efficiently purchase ingredients and lead a healthy lifestyle.

[0336] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0337] This invention relates to a system that acquires a user's physical information and emotional state, and presents meal suggestions and locations for purchasing ingredients. The following describes a specific form for realizing this system.

[0338] User registration and profile settings

[0339] 1. User: Install the application and, upon first launch, enter your name, age, gender, height, weight, and information about allergies, preferred foods, and health goals. This information will be used to create your individual profile.

[0340] 2. Server: Saves the entered information to the database and creates a profile for each user.

[0341] Integration with wearable devices

[0342] 1. User: Link the wearable device and application in the settings screen.

[0343] 2. Terminal: Regularly receives data such as heart rate, steps taken, and calories burned from a wearable device.

[0344] 3. Server: Receives device data sent from terminals and stores it in the database.

[0345] Making a list of ingredients in the refrigerator

[0346] 1. User: Take a photo of the inside of the refrigerator with your smartphone and upload it to the application.

[0347] 2. Terminal: Uses image recognition technology to identify ingredients in a photograph and creates an ingredient list.

[0348] 3. Server: Stores the ingredient list in a database and manages it as the user's current inventory information.

[0349] Specific example

[0350] User: Upload a photo of the refrigerator to the application.

[0351] Terminal: Analyzes the photo and identifies "tomato, carrot, and chicken breast."

[0352] Server: Identify these ingredients and add them to the user's inventory list.

[0353] Generating customized recipes and menus

[0354] 1. Server: Uses generative AI to create a customized weekly meal plan based on the user's health data, ingredient inventory list, preferences, and allergy information.

[0355] 2. Terminal: Presents the generated menu and necessary nutrient and calorie information to the user.

[0356] Emotional recognition and adaptation of meal suggestions using an emotional engine

[0357] 1. User: Inputs facial expressions and voice through the application. Also, notifies emotional states by inputting text.

[0358] 2. Device: The emotion engine analyzes the user's facial expressions, voice, or text data to recognize their current emotional state.

[0359] 3. Server: Analyzes recognized emotional information in conjunction with health data and adapts meal suggestions based on the user's emotions.

[0360] Specific example

[0361] User: Type the text "I'm tired today" into the chatbot within the application.

[0362] Terminal: The emotion engine analyzes this text data and recognizes that the user is tired.

[0363] Server: Suggest highly nutritious ingredients and easy-to-prepare dishes to reduce user fatigue.

[0364] Suggestions for places to buy groceries

[0365] 1. Server: Identify any missing ingredients based on the proposed menu.

[0366] 2. Server: For any missing ingredients, it searches for potential purchase options from partner e-commerce platforms and gathers information on the nearest places to buy them and online stores.

[0367] 3. Terminal: Presents the generated list of potential purchases to the user and provides a link to the online store.

[0368] Feedback on health data and emotional data

[0369] 1. Server: Collects data from wearable devices, meal records, and emotional data to analyze the user's health status.

[0370] 2. Server: Generates health feedback and advice, emotion-based recommendations, and notification messages based on the analysis results.

[0371] 3. Device: Send feedback and advice to users via push notifications.

[0372] Specific example

[0373] Server: Verify that the user's activity level has not reached 70% of the target.

[0374] The device notifies the user that "You can reach your goal by walking 1000 more steps." If it detects that the user's stress level is high, it suggests, "Try some relaxing herbal tea."

[0375] The above describes the embodiments of the present invention. In this form, users can receive not only individual health data and food information, but also optimal meal suggestions tailored to their mood at any given time, enabling them to efficiently purchase food ingredients and lead a healthy and balanced life.

[0376] The following describes the processing flow.

[0377] User registration and profile settings

[0378] Step 1:

[0379] User: Install the application and, upon first launch, enter your name, age, gender, height, weight, and information about allergies, preferred foods, and health goals.

[0380] Step 2:

[0381] Terminal: Sends the entered information to the server.

[0382] Step 3:

[0383] Server: Stores received information in a database and creates user profiles.

[0384] Integration with wearable devices

[0385] Step 4:

[0386] User: Link the wearable device and application in the settings screen.

[0387] Step 5:

[0388] Device: Once the connection is successfully completed, enable the setting to periodically receive data such as heart rate, steps, and calories burned from the wearable device.

[0389] Step 6:

[0390] Server: Receives device data sent from terminals and stores it in the database.

[0391] Making a list of ingredients in the refrigerator

[0392] Step 7:

[0393] User: Take a photo of the inside of the refrigerator with your smartphone and upload it to the application.

[0394] Step 8:

[0395] Device: Uses AI image recognition technology to identify ingredients in a photograph and creates an ingredient list.

[0396] Step 9:

[0397] Terminal: Sends the generated ingredient list to the server.

[0398] Step 10:

[0399] Server: Stores the received ingredient list in the database and manages it as the user's current inventory information.

[0400] Generating customized recipes and menus

[0401] Step 11:

[0402] Server: Uses generative AI to create a customized weekly meal plan based on the user's health data, ingredient inventory list, preferences, and allergy information.

[0403] Step 12:

[0404] Server: Sends the generated menu to the terminal.

[0405] Step 13:

[0406] Terminal: Displays the received menu and necessary nutrient and calorie information to the user.

[0407] Emotional recognition and adaptation of meal suggestions using an emotional engine

[0408] Step 14:

[0409] User: Notifies emotional states by inputting facial expressions, voice, or text through the application.

[0410] Step 15:

[0411] Device: The emotion engine analyzes the user's facial expressions, voice, or text data to recognize their current emotional state.

[0412] Step 16:

[0413] Server: Analyzes recognized emotional information in conjunction with health data and adapts meal suggestions based on the user's emotions.

[0414] Suggestions for places to buy groceries

[0415] Step 17:

[0416] Server: Identify any missing ingredients based on the proposed menu.

[0417] Step 18:

[0418] Server: Searches for potential purchase options from partner e-commerce platforms for missing ingredients, gathering information on the nearest places to buy them and online stores.

[0419] Step 19:

[0420] Server: Sends a list of potential purchases to the terminal.

[0421] Step 20:

[0422] Terminal: Presents a generated list of potential purchases to the user and provides a link to the online store.

[0423] Feedback on health data and emotional data

[0424] Step 21:

[0425] Server: Collects data from wearable devices, meal records, and emotional data to analyze the user's health status.

[0426] Step 22:

[0427] Server: Generates health feedback and advice, emotion-based recommendations, and notification messages based on analysis results.

[0428] Step 23:

[0429] Server: Sends the generated feedback and advice to the terminal.

[0430] Step 24:

[0431] Device: Send feedback and advice to users via push notifications.

[0432] The above describes the specific actions taken at each processing step. This detailed processing flow allows users to receive optimal meal suggestions based on the latest health data and ingredient information, efficiently purchase ingredients, and lead a healthy and balanced life. In addition, it provides even more personalized support by offering suggestions that take into account the user's emotional state.

[0433] (Example 2)

[0434] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0435] Conventional meal suggestion systems based solely on the user's physical data and ingredient information, failing to consider the user's emotional state. Furthermore, suggestions for ingredient sourcing locations were limited, failing to integrate information from online stores and physical retail outlets. Therefore, there was a need to provide users with optimal and efficient meal suggestions and ingredient purchasing methods.

[0436] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0437] In this invention, the server includes means for communicating with equipment for acquiring physical data, means for identifying food ingredients from captured content, means for suggesting meals based on the identified food ingredient data, the user's physical data, and emotional state, and means for identifying missing ingredients based on the suggested meal and indicating where to obtain them. This enables meal suggestions that take into account the user's emotional state in addition to their physical data and food ingredient information, and further enables the provision of food ingredient acquisition locations by integrating information from e-commerce platforms and physical stores, thereby providing optimal and efficient meal suggestions and means for purchasing ingredients.

[0438] "Physical data" refers to information about the user's body, including data such as height, weight, heart rate, steps taken, and calories burned.

[0439] "Equipment" refers to wearable devices, sensor devices, and other devices used to acquire the user's physical data.

[0440] "Content" refers to images and photographs taken by users, including, in particular, food items in a refrigerator and cooked meals.

[0441] "Food ingredient data" refers to information such as the type, quantity, and availability of food items identified from the content.

[0442] "Emotional state" refers to information that indicates the user's current psychological and emotional state, including, for example, fatigue, stress, and satisfaction.

[0443] "Meal suggestions" refers to the act of suggesting optimal meal menus and recipes based on the user's physical data and emotional state.

[0444] "Place of acquisition" refers to the location where the ingredients needed for the proposed meal can be purchased, including online stores and physical stores.

[0445] An "e-commerce platform" refers to a website or application that facilitates the buying and selling of goods online, and examples include e-commerce sites.

[0446] A "physical store" refers to an actual store or supermarket where users can visit in person to purchase groceries.

[0447] This invention relates to a system that acquires a user's physical data and emotional state, and presents personalized meal suggestions and locations for obtaining ingredients. The following describes specific embodiments of this system.

[0448] Overall structure

[0449] This system consists of a device (wearable device) for acquiring physical data, the user's smartphone (terminal), and a server connected to them. The server receives and analyzes the data and provides meal suggestions and locations for data acquisition.

[0450] User registration and profile settings

[0451] Users first install the application on their smartphone and, upon initial launch, enter their name, age, gender, height, weight, allergy information, preferred foods, and health goals. This creates an individual profile. The server stores the information entered by the user in a database and builds the profile. The application interface is designed to allow users to easily enter the necessary information.

[0452] Integration with wearable devices

[0453] The user pairs a wearable device (e.g., a wristwatch-type heart rate monitor) with the application via the smartphone's settings screen. The wearable device periodically sends data such as heart rate, steps taken, and calories burned to the device. The device periodically sends this data to a server. The server stores the received data in a database and manages the user's health data.

[0454] Making a list of ingredients in the refrigerator

[0455] The user takes a photo of the inside of their refrigerator with their smartphone and uploads it to the application. The device uses image recognition technology (e.g., Amazon Rekognition) to identify the food items in the photo and create a list of those items. The server stores the identified food list in a database and manages it as the user's current inventory information.

[0456] Specific example

[0457] When a user uploads a photo of their refrigerator to the application, the device analyzes the image and identifies "tomatoes, carrots, and chicken breast." The server saves these ingredients to a database and updates the inventory list.

[0458] Generating customized recipes and menus

[0459] The server uses generative AI (e.g., OpenAI's GPT-4) to create a customized weekly meal plan based on the user's health data, ingredient inventory list, preferences, and allergy information. The terminal then presents the generated meal plan along with necessary nutrient and calorie information to the user.

[0460] Examples of prompt statements

[0461] "I'm a woman in my 30s, aiming to lose weight. I have tomatoes, carrots, and chicken breast in my refrigerator. Please create a low-calorie, nutritionally balanced meal plan for one week."

[0462] Emotional recognition and adaptation of meal suggestions using an emotional engine

[0463] Users notify their emotional state through the application by inputting facial expressions, voice, or text. The emotion engine analyzes this input data to recognize the user's current emotional state. The server then comprehensively analyzes the recognized emotional state and physical data to provide meal suggestions based on the user's emotions.

[0464] Specific example

[0465] When a user types "I'm tired today" into the in-app chatbot, the device's emotion engine analyzes this text data and recognizes that the user is tired. The server then suggests highly nutritious ingredients and easy-to-prepare dishes.

[0466] Suggestions for locations to obtain ingredients

[0467] The server identifies any missing ingredients based on the suggested menu and searches for purchase options from partner e-commerce platforms and physical stores. The terminal presents the generated list of purchase options to the user and provides links to online stores.

[0468] Feedback on health data and emotional data

[0469] The server aggregates data from wearable devices, meal logs, and emotional data to analyze the user's health status. Based on the analysis, it generates feedback and advice and notifies the user. The device delivers this feedback and advice to the user via push notifications. For example, it provides specific advice such as, "You can achieve your goal by walking another 1000 steps," or emotionally-based recommendations such as, "Try a relaxing herbal tea."

[0470] The above describes a specific embodiment for carrying out the present invention. This allows users to receive not only individual health data and food information, but also optimal meal suggestions tailored to their mood at any given time, enabling them to efficiently purchase food ingredients and lead a healthy and balanced life.

[0471] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0472] Program execution flow and detailed explanation of each processing step

[0473] User registration and profile settings

[0474] Step 1:

[0475] The user installs the application and launches it for the first time.

[0476] Input: None

[0477] Output: Display of the app's welcome screen

[0478] Specific operation: The user downloads and launches the app. The interface is designed to be intuitive and easy to understand.

[0479] Step 2:

[0480] The user enters their name, age, gender, height, weight, allergy information, favorite foods, and health goals.

[0481] Input: User information (name, age, gender, height, weight, allergy information, favorite foods, health goals)

[0482] Output: User profile data

[0483] Specific action: The user enters the required information into the input form and presses the submit button.

[0484] Step 3:

[0485] The server saves the entered information to the database and builds a user profile.

[0486] Input: User profile data

[0487] Output: Saved to database successfully

[0488] Specific operation: The server receives profile data and stores it in the database. A relational database such as MySQL is used for this database.

[0489] Integration with wearable devices

[0490] Step 1:

[0491] The user pairs the wearable device and the application in the app settings screen.

[0492] Input: Pairing request

[0493] Output: Pairing complete notification

[0494] Specific steps: The user selects the wearable device on their smartphone's Bluetooth settings screen and pairs it.

[0495] Step 2:

[0496] The device periodically receives data such as heart rate, steps taken, and calories burned from the wearable device.

[0497] Input: Device data (heart rate, steps, calories burned)

[0498] Output: Automatic saving of received data at regular intervals.

[0499] Specific operation: The device periodically (e.g., every hour) retrieves data from the wearable device and saves it locally.

[0500] Step 3:

[0501] The server receives device data sent from the terminal and stores it in the database.

[0502] Input: Device data from the terminal

[0503] Output: Saved to database successfully

[0504] Specific operation: At regular intervals (e.g., every 24 hours), the server receives data from the terminal, analyzes it, and saves it to the database.

[0505] Making a list of ingredients in the refrigerator

[0506] Step 1:

[0507] The user takes a photo of the inside of the refrigerator with their smartphone and uploads it to the application.

[0508] Input: Photo of the inside of the refrigerator

[0509] Output: Upload complete notification

[0510] Specific action: The user uses their smartphone camera to take a picture of the inside of the refrigerator and uploads the image to the app.

[0511] Step 2:

[0512] The device uses image recognition technology to identify food items in the photograph.

[0513] Input: Uploaded photo

[0514] Output: Identified ingredient list

[0515] Specific operation: The device uses image recognition APIs such as Amazon Rekognition to automatically analyze and identify food items in a photograph.

[0516] Step 3:

[0517] The server stores a list of identified ingredients in a database and manages it as the user's current inventory information.

[0518] Input: Identified ingredient list

[0519] Output: Saved to database successfully

[0520] Specific operation: The server receives the identified ingredient information and stores it in the database.

[0521] Generating customized recipes and menus

[0522] Step 1:

[0523] The server uses generative AI to create a customized weekly meal plan based on the user's health data, ingredient inventory list, preferences, and allergy information.

[0524] Input: Health data, food inventory list, preferences and allergy information

[0525] Output: Customized menu

[0526] Specific operation: The server sends prompt messages to the AI ​​model (e.g., OpenAI's GPT-4) and receives the generated menu.

[0527] Step 2:

[0528] The device displays the generated menu and information on necessary nutrients and calories to the user.

[0529] Input: Customized menu

[0530] Output: Presentation of menu information

[0531] Specific operation: The terminal displays the menu information received from the server on its screen, along with information on necessary nutrients and calories.

[0532] Emotional recognition and adaptation of meal suggestions using an emotional engine

[0533] Step 1:

[0534] Users can input facial expressions and voices through the application, or input text to notify their emotional state.

[0535] Input: Facial expressions, voice, or text data

[0536] Output: Notification of emotional state

[0537] Specific operation: Users input their emotional state through the app using their facial photo, voice, or text data.

[0538] Step 2:

[0539] The device uses an emotion engine to analyze facial expressions, voice, or text to recognize the user's current emotional state.

[0540] Input: Emotional data (facial expressions, voice, text)

[0541] Output: Recognized emotional state

[0542] Specific operation: The device uses an emotion recognition engine (e.g., Microsoft® Azure® Cognitive Services) to analyze input data and recognize the emotional state.

[0543] Step 3:

[0544] The server integrates and analyzes recognized emotional states and physical data to provide meal suggestions based on the user's emotions.

[0545] Input: Recognized emotional state, physical data

[0546] Output: Meal suggestions based on emotional state

[0547] Specific operation: Based on the analysis results, the server generates meal suggestions that include highly nutritious ingredients and easy-to-prepare dishes.

[0548] Suggestions for locations to obtain ingredients

[0549] Step 1:

[0550] The server identifies any missing ingredients based on the suggested menu.

[0551] Input: Suggested menu, ingredient inventory list

[0552] Output: List of missing ingredients

[0553] Specific operation: The server compares the menu with the inventory list to identify any missing ingredients.

[0554] Step 2:

[0555] The server searches for potential purchase options for ingredients that are currently out of stock, from partner e-commerce platforms and physical stores.

[0556] Input: List of missing ingredients

[0557] Output: Purchase candidate list

[0558] Specific operation: The server uses APIs from e-commerce platforms and physical stores to search for purchasing options for missing ingredients.

[0559] Step 3:

[0560] The device displays a list of potential purchases to the user and provides a link to the online store.

[0561] Input: Purchase candidate list

[0562] Output: Display of purchase candidate list

[0563] Specific operation: The device displays a list of potential purchases on the screen and provides the user with a link to each option.

[0564] Feedback on health data and emotional data

[0565] Step 1:

[0566] The server collects data from wearable devices, meal logs, and emotional data to analyze the user's health status.

[0567] Input: Wearable device data, food log data, emotional data

[0568] Output: Analysis results of health status

[0569] Specific operation: The server uses Python or similar languages ​​to execute analysis scripts and collect and analyze user health data.

[0570] Step 2:

[0571] The server generates feedback and advice based on the analysis results and notifies the user.

[0572] Input: Analysis results of health status

[0573] Output: Feedback and advice

[0574] Specific operation: The server generates appropriate advice and feedback for the user based on the analysis results and creates notification messages.

[0575] Step 3:

[0576] The device will send feedback and advice to the user via push notifications.

[0577] Input: Feedback and advice

[0578] Output: Notification message

[0579] Specific actions: The device uses push notifications to inform the user of specific advice, such as "You can reach your goal by walking 1000 more steps." It also provides emotion-based recommendations, such as "Try a relaxing herbal tea."

[0580] (Application Example 2)

[0581] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0582] In modern society, maintaining good health requires individuals to plan appropriate meals and efficiently purchase the necessary ingredients. However, it takes considerable time and effort for users to choose appropriate meals, identify necessary ingredients, and purchase them themselves. Furthermore, while a user's emotional state often significantly influences their meal choices, there are limited systems that can appropriately reflect this in their meal suggestions. In addition, although the use of online stores and food delivery services has increased, there is a lack of systems that centrally manage these services and provide users with the most suitable purchasing options. A system that solves these problems is needed.

[0583] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0584] In this invention, the server includes means for communicating with a device for acquiring physical information, means for identifying ingredients from captured images, means for suggesting meals based on the identified ingredient information, the user's physical information, and emotional state, and means for identifying necessary ingredients based on the suggested meal, indicating where to purchase them, and providing links to food delivery services. This allows the user to receive optimal meal suggestions based on their physical information and emotional state, and to purchase necessary ingredients quickly and efficiently. Furthermore, they can quickly obtain ingredients and prepared meals through food delivery, supporting their eating habits.

[0585] "Physical information" refers to data that quantifies the user's physical condition, such as heart rate, steps taken, and calories burned.

[0586] "Emotional state" refers to information that indicates the user's current psychological state or mood, analyzed from the user's facial expressions, voice, or text data.

[0587] "Meal suggestions" is a process that generates and presents meal menus suitable for the user based on the user's physical information and emotional state.

[0588] "Food identification" is the process of using AI technology to identify food items from images taken by the user and analyzing their types.

[0589] "Presenting a place to purchase" refers to the process of showing users information about online stores or physical shops where they can purchase the ingredients they are missing.

[0590] A "food delivery link" is an online link that allows users to order the necessary ingredients and dishes based on a suggested meal menu.

[0591] This invention relates to a system that acquires a user's physical information and emotional state, and presents meal suggestions and locations for purchasing ingredients. An embodiment of this system is described in detail below.

[0592] User registration and profile settings

[0593] Users first install the application on their smartphone and, upon initial launch, enter their name, age, gender, height, weight, allergy information, preferred foods, and health goals. The server stores this information in a database and creates a profile for each user.

[0594] Integration with wearable devices

[0595] The user connects the wearable device and the application through the application's settings screen. The terminal periodically receives data such as heart rate, steps taken, and calories burned from the wearable device. The server stores the received device data in a database.

[0596] Making a list of ingredients in the refrigerator

[0597] The user takes a photo of the inside of their refrigerator with their smartphone and uploads it to the application. The device uses image recognition technology to identify the food items in the photo and creates a list of those items. The server stores the list of items in a database and manages it as the user's current inventory information.

[0598] Generating customized recipes and menus

[0599] The server uses an AI model to generate a customized weekly meal plan based on the user's health data, ingredient inventory list, preferences, and allergy information. The terminal then displays the generated meal plan along with information on necessary nutrients and calories to the user.

[0600] Emotional recognition and adaptation of meal suggestions using an emotional engine

[0601] Users notify their emotional state by inputting facial expressions, voice, or text through the application. The device uses an emotion engine to analyze the user's facial expressions, voice, or text data to recognize their current emotional state. The server analyzes the recognized emotional information in conjunction with health data and adapts meal suggestions based on the user's emotions.

[0602] Suggestions for places to buy groceries

[0603] The server identifies any missing ingredients based on the suggested menu. It also searches for purchase options from partner e-commerce platforms and the nearest supermarket, and provides links to online stores for those missing ingredients.

[0604] Feedback on health data and emotional data

[0605] The server aggregates data from wearable devices, meal records, and emotional data to analyze the user's health status. Based on the analysis, it generates health feedback and advice, emotional recommendations, and creates notification messages. The device then delivers the feedback and advice to the user via push notifications.

[0606] Hardware and software used

[0607] Hardware: Smartphones, wearable devices, refrigerators with cameras

[0608] Software: TensorFlow, OpenCV, Flask, Twilio, Generative AI Models

[0609] Specific example

[0610] The user uploads a photo of their refrigerator to the application, and image analysis identifies "tomatoes, carrots, and chicken breast." If the user enters the text "I'm tired today," the emotion engine analyzes this and recognizes that the user is tired. Based on this information, the server suggests highly nutritious ingredients and easy-to-prepare dishes. For example, the generative AI model is prompted with the message, "The user is currently feeling tired; please suggest an optimal dinner," and then generates suggestions.

[0611] In this way, this system allows users to receive optimal meal suggestions tailored to their physical information and emotional state, and to quickly and efficiently purchase the necessary ingredients. Furthermore, because meals can be easily obtained through food delivery, the system supports users in leading a healthy lifestyle.

[0612] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0613] Step 1: User registration and profile setup

[0614] When a user first launches an application installed on their smartphone, they enter their name, age, gender, height, weight, allergy information, preferred foods, and health goals. This information is the input data, which the server stores in a database to create a profile for each user. The information stored in the database is the output. This initiates individual data management for each user.

[0615] Step 2: Integration with wearable devices

[0616] The user connects the wearable device to the application through the application's settings screen. The terminal periodically receives data such as heart rate, steps taken, and calories burned from the wearable device. The input is biometric data transmitted from the wearable device, which the server receives and stores in a database. The stored biometric data becomes the output. This allows the user's physical information to be obtained in real time.

[0617] Step 3: Make a list of the ingredients in your refrigerator.

[0618] The user takes a photo of the inside of their refrigerator with their smartphone and uploads it to the application. The application receives the image and uses OpenCV and TensorFlow to identify the food items in the photo using image recognition technology. The output is a list of identified food items, which the server saves to a database. This process manages the current food inventory information.

[0619] Step 4: Generating customized recipes and menus

[0620] The server retrieves the user's health data, ingredient inventory list, preferences, and allergy information, and generates a week's worth of customized meal plans by inputting prompts into a generative AI model. For example, by inputting the prompt, "The user is currently feeling tired; please suggest an optimal dinner," the generative AI model will suggest a meal plan. The generated meal plan is the output, and the terminal presents it to the user. The meal suggestions are displayed with specific nutritional and calorie information.

[0621] Step 5: Emotional recognition and adaptation of meal suggestions by the emotional engine

[0622] Users notify the system of their emotional state by inputting facial expressions, voice, or text through the application. The terminal uses an emotion engine to analyze the user's current emotional state from the facial expressions, voice, and text data. This analysis result becomes the input data, which the server acquires and analyzes together with the user's health data. Based on the analysis result, the system provides meal suggestions appropriate to the user's emotional state, and these suggestions become the output.

[0623] Step 6: Suggesting places to buy groceries

[0624] The server identifies missing ingredients based on the suggested menu. These missing ingredients are the input data, and the server searches for purchase options from e-commerce platforms and the nearest supermarket. The output is a list of purchase options, which the terminal presents to the user, providing links to online stores. This allows the user to efficiently purchase the missing ingredients.

[0625] Step 7: Feedback on health and emotional data

[0626] The server collects data from wearable devices, meal records, and emotional data to analyze the user's health status. The input is the various collected data, and the server generates feedback and advice based on its analysis. This is the output, and the device communicates it to the user via push notifications, etc. For example, messages such as "You can achieve your goal by walking 1000 more steps" or emotional recommendations such as "Try a relaxing herbal tea" might be displayed.

[0627] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0628] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0629] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0630] [Second Embodiment]

[0631] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0632] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0633] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0634] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0635] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0636] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0637] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0638] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0639] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0640] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0641] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0642] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0643] This invention relates to a system that acquires a user's physical information, identifies food ingredients from captured images, makes meal suggestions based on this information, and indicates where to purchase the necessary ingredients. The following describes a specific form for realizing this system.

[0644] User registration and profile settings

[0645] 1. User: Install the application and, upon first launch, enter your name, age, gender, height, weight, and information about allergies, preferred foods, and health goals. This information will be used to create your individual profile.

[0646] 2. Server: Saves the entered information to the database and creates a profile for each user.

[0647] Integration with wearable devices

[0648] 1. User: Link the wearable device and application in the settings screen.

[0649] 2. Terminal: Regularly receives data such as heart rate, steps taken, and calories burned from wearable devices.

[0650] 3. Server: Saves received data to the database in real time and monitors the user's health status.

[0651] Making a list of ingredients in the refrigerator

[0652] 1. User: Take a photo of the inside of the refrigerator with your smartphone and upload it to the application.

[0653] 2. Terminal: Uses image recognition technology to identify ingredients in a photograph and generates an ingredient list.

[0654] 3. Server: Stores the ingredient list in a database and manages it as the user's current ingredient inventory information.

[0655] Specific example

[0656] User: Upload a photo of the refrigerator to the application.

[0657] Terminal: Analyzes the photo and identifies "tomato, carrot, and chicken breast."

[0658] Server: Adds these identified ingredients to the user's inventory list and saves them as current inventory information.

[0659] Generating customized recipes and menus

[0660] 1. Server: Uses generative AI to create a customized weekly meal plan based on the user's health data, ingredient inventory list, and preferences.

[0661] 2. Terminal: Presents the generated menu and necessary nutrient and calorie information to the user.

[0662] Suggestions for places to buy groceries

[0663] 1. Server: Based on the proposed menu, list the necessary ingredients and identify any missing ingredients.

[0664] 2. Server: Searches for potential grocery purchases from partner e-commerce platforms and generates links to the nearest stores or online shops.

[0665] 3. Terminal: Presents the generated list of potential purchases to the user and provides a link to the online store.

[0666] Specific example

[0667] Server: The server generates a weekly menu using tomatoes, carrots, and chicken breast, and suggests recipes such as "Tomato Chicken Stew."

[0668] Terminal: Displays these recipes, a list of required ingredients, and cooking instructions to the user.

[0669] Health data feedback

[0670] 1. Server: Collects data from wearable devices and food records to analyze the user's health status.

[0671] 2. Server: Generates health feedback and advice based on analysis results, and creates notification messages.

[0672] 3. Device: Send the generated feedback and advice to the user via push notification.

[0673] Specific example

[0674] Server: Verify that the user's activity level has not reached 70% of the target.

[0675] Device: Notifies the user with the message, "Walk another 1000 steps to reach your goal."

[0676] The above describes the embodiments of the present invention. This system allows users to receive customized meal suggestions based on their individual health data and food information, enabling them to efficiently purchase food ingredients and lead a healthy lifestyle.

[0677] The following describes the processing flow.

[0678] User registration and profile settings

[0679] Step 1:

[0680] User: Install the application and, upon first launch, enter your name, age, gender, height, weight, and information about allergies, preferred foods, and health goals.

[0681] Step 2:

[0682] Terminal: Sends the entered information to the server.

[0683] Step 3:

[0684] Server: Stores received information in a database and creates user profiles.

[0685] Integration with wearable devices

[0686] Step 4:

[0687] User: In the application's settings screen, link the wearable device with the application.

[0688] Step 5:

[0689] Device: Once the connection is successfully completed, enable the setting to periodically receive data such as heart rate, steps, and calories burned from the wearable device.

[0690] Step 6:

[0691] Server: Receives device data sent from terminals and stores it in the database.

[0692] Making a list of ingredients in the refrigerator

[0693] Step 7:

[0694] User: Take a photo of the inside of the refrigerator with your smartphone and upload it to the application.

[0695] Step 8:

[0696] Device: Uses AI image recognition technology to identify ingredients in a photograph and creates an ingredient list.

[0697] Step 9:

[0698] Terminal: Sends the generated ingredient list to the server.

[0699] Step 10:

[0700] Server: Stores the received ingredient list in the database and manages it as the user's current inventory information.

[0701] Generating customized recipes and menus

[0702] Step 11:

[0703] Server: Uses generative AI to create a customized weekly meal plan based on the user's health data, ingredient inventory list, preferences, and allergy information.

[0704] Step 12:

[0705] Server: Sends the generated menu to the terminal.

[0706] Step 13:

[0707] Terminal: Displays the received menu and necessary nutrient and calorie information to the user.

[0708] Suggestions for places to buy groceries

[0709] Step 14:

[0710] Server: Identify any missing ingredients based on the proposed menu.

[0711] Step 15:

[0712] Server: Searches for potential purchase options from partner e-commerce platforms for missing ingredients, gathering information on the nearest places to buy them and online stores.

[0713] Step 16:

[0714] Server: Sends a list of potential purchases to the terminal.

[0715] Step 17:

[0716] Terminal: Presents a generated list of potential purchases to the user and provides a link to the online store.

[0717] Health data feedback

[0718] Step 18:

[0719] Server: Collects data from wearable devices and food records to analyze the user's health status.

[0720] Step 19:

[0721] Server: Generates health feedback and advice based on analysis results, and creates notification messages.

[0722] Step 20:

[0723] Server: Sends the generated feedback and advice to the terminal.

[0724] Step 21:

[0725] Device: Send feedback and advice to users via push notifications.

[0726] The above describes the detailed operation at each processing step. Through this detailed processing flow, users can always receive optimal meal suggestions based on the latest health data and ingredient information, efficiently purchase ingredients, and lead a healthy lifestyle.

[0727] (Example 1)

[0728] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0729] Maintaining a healthy diet is difficult in today's busy lifestyle. Furthermore, it's challenging for users to select appropriate foods and recipes based on their own health status and preferences. Even efficiently shopping for ingredients by identifying suitable locations is time-consuming. To address these issues, a system is needed that centrally manages users' health data and ingredient information, and provides optimal meal suggestions and ingredient purchasing locations based on this data.

[0730] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0731] In this invention, the server includes means for inputting the user's personal information and creating a profile; means for working with a wearable device to collect biometric data; means for analyzing images of the inside of a refrigerator taken and identifying ingredients; means for creating a customized menu using a generative AI model based on the identified ingredient information and the user's health data; and means for identifying missing ingredients based on the proposed menu and suggesting where to purchase them. As a result, the user can receive meal suggestions based on their health status and ingredient information, efficiently purchase necessary ingredients, and maintain a healthy lifestyle.

[0732] A "user" is an entity that uses the system to manage individual health data and food information, and to make meal suggestions and purchase food ingredients.

[0733] A "profile" is a set of individual information that includes the user's name, age, gender, height, weight, allergy information, preferred foods, and health goals.

[0734] A "wearable device" is an electronic device worn by a user to collect biometric data such as heart rate, steps taken, and calories burned.

[0735] "Biometric data" refers to data such as a user's heart rate, steps taken, and calories burned, collected through wearable devices and other means.

[0736] "Image analysis" is the process of extracting specific information from captured images, and is particularly used to identify food items inside a refrigerator.

[0737] "Food identification" is a method of identifying food items in a refrigerator through image analysis and listing that information.

[0738] A "generative AI model" is an artificial intelligence model that automatically generates customized menus based on the user's health data and ingredient information.

[0739] A "menu" refers to a meal plan or recipe suggested based on the user's health data and available ingredients.

[0740] "Place of purchase" refers to any location where you can purchase any missing ingredients based on the suggested menu, including online stores and physical stores.

[0741] "Health data" is a general term for a user's physical information, biometric data, and dietary history, and is used to manage their health status.

[0742] "Feedback" refers to improvement suggestions and advice provided based on the user's health status and activity level.

[0743] This invention relates to a system that acquires a user's physical information, identifies food ingredients from captured images, makes meal suggestions based on this information, and indicates where to purchase the necessary ingredients. Specific embodiments of this invention are described below.

[0744] User registration and profile settings

[0745] 1. User: First, the user installs the system's application on their smartphone. Upon first launching the application, they enter their name, age, gender, height, weight, allergy information, preferred foods, and health goals. This creates the user's individual profile.

[0746] 2. Server: The server stores the information entered by the user in a database and creates a profile for each user. A database such as MongoDB is used for this purpose.

[0747] Integration with wearable devices

[0748] 1. User: The user connects the wearable device and the application using Bluetooth in the application's settings screen.

[0749] 2. Terminal: The terminal periodically receives biometric data such as heart rate, steps taken, and calories burned from the wearable device. Bluetooth communication is used for this purpose.

[0750] 3. Server: The server stores incoming data in a database in real time and monitors the user's health status. A database such as PostgreSQL is used for this purpose.

[0751] Making a list of ingredients in the refrigerator

[0752] 1. User: The user takes a photo of the inside of the refrigerator with their smartphone and uploads it to the application.

[0753] 2. Terminal: The terminal uses image recognition technology (e.g., TensorFlow) to identify ingredients in the captured image and generate a list of ingredients.

[0754] 3. Server: The server stores the ingredient list in a database and manages it as the user's current ingredient inventory information. A database such as MySQL is used for this purpose.

[0755] Specific example

[0756] When a user uploads a photo of their refrigerator to the application, the device analyzes the photo and identifies "tomatoes, carrots, and chicken breast." The server adds these identified ingredients to the user's inventory list and saves it as current inventory information.

[0757] Generating customized recipes and menus

[0758] 1. Server: The server uses a generative AI model (e.g., OpenAI's GPT) to create a customized weekly meal plan based on the user's health data, ingredient inventory list, and preferences.

[0759] 2. Terminal: The terminal presents the user with the generated menu and information on necessary nutrients and calories.

[0760] Example of a prompt

[0761] Send the following prompt to the generating AI: "Create a one-week meal plan based on User A's health data and inventory information."

[0762] Suggestions for places to buy groceries

[0763] 1. Server: The server identifies any missing ingredients based on the proposed menu and suggests where to purchase them.

[0764] 2. Server: The server searches for potential grocery purchases from partner e-commerce platforms and generates links to the nearest purchase locations and online stores.

[0765] 3. Terminal: The terminal presents the user with a generated list of potential purchases and provides links to online stores.

[0766] Specific example

[0767] The server generates a week's worth of meal plans based on the user's health data and food inventory, suggesting recipes such as "Tomato Chicken Stew." The terminal displays these recipes, a list of required ingredients, and cooking instructions to the user.

[0768] Health data feedback

[0769] 1. Server: The server aggregates data from wearable devices and food records, and analyzes the user's health status.

[0770] 2. Server: The server generates health feedback and advice based on the analysis results and creates notification messages.

[0771] 3. Device: The device will send the generated feedback and advice to the user via push notifications.

[0772] Specific example

[0773] The server confirms that the user's activity level has not reached 70% of the target, and the device notifies the user, "Walk another 1000 steps to reach your goal."

[0774] The above describes the embodiments of the present invention. This system allows users to receive customized meal suggestions based on their health condition and food information, enabling them to efficiently purchase food ingredients and lead a healthy lifestyle.

[0775] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0776] Step 1:

[0777] User: Install the application on your smartphone and enter your personal information (name, age, gender, height, weight, allergy information, favorite foods, health goals) upon first launch.

[0778] Input: User's personal information

[0779] Output: Individual user profiles

[0780] Specific action: The user enters information into each field and presses the "Save" button.

[0781] Step 2:

[0782] Server: Stores the entered personal information in the database and creates a profile for each user.

[0783] Input: User's personal information

[0784] Output: Saved user profile

[0785] Specific operation: The server uses MongoDB to store personal information received from users in JSON format.

[0786] Step 3:

[0787] User: Link the wearable device and application in the settings screen.

[0788] Input: Connection information for wearable devices

[0789] Output: Connection established with wearable device

[0790] Specific steps: The user opens the Bluetooth settings, selects the wearable device, and presses the pairing button.

[0791] Step 4:

[0792] Terminal: Regularly receives data such as heart rate, steps taken, and calories burned from wearable devices.

[0793] Input: Data from a wearable device

[0794] Output: Biometric data temporarily stored on the device

[0795] Specific operation: The device uses Bluetooth communication to retrieve data and save it to local storage.

[0796] Step 5:

[0797] Server: Receives data from wearable devices via terminals and saves it to a database in real time.

[0798] Input: Biometric data transmitted from the device

[0799] Output: Biometric data stored in the database

[0800] Specific operation: The server uses PostgreSQL to execute INSERT statements into the database.

[0801] Step 6:

[0802] User: Take a photo of the inside of the refrigerator with your smartphone and upload it to the application.

[0803] Input: Image of the inside of a refrigerator

[0804] Output: Image files uploaded to the application

[0805] Specific action: The user takes a photo using the camera app and presses the upload button in the application.

[0806] Step 7:

[0807] Terminal: Uses image recognition technology to identify ingredients in captured images and generates an ingredient list.

[0808] Input: Uploaded image of the inside of a refrigerator

[0809] Output: List of identified ingredients

[0810] Specific operation: The device uses TensorFlow to perform image analysis and identify food ingredients. Example: "Tomato, carrot, chicken breast".

[0811] Step 8:

[0812] Server: Stores ingredient lists in a database and manages them as the user's current ingredient inventory information.

[0813] Input: Identified ingredient list

[0814] Output: Food inventory information stored in the database

[0815] Specific operation: The server uses MySQL to store ingredient information in a database.

[0816] Step 9:

[0817] Server: Uses a generative AI model to create a customized weekly meal plan based on the user's health data, ingredient inventory list, and preferences.

[0818] Input: User's health data, food inventory list, preferences

[0819] Output: Customized weekly meal plan

[0820] Specific operation: The server sends the following prompt to the generative AI model: "Create a weekly meal plan based on user A's health data and inventory information."

[0821] Step 10:

[0822] Terminal: Displays the generated menu and necessary nutrient and calorie information to the user.

[0823] Input: Customized menu

[0824] Output: Menu information presented to the user

[0825] Specific action: The device displays the menu in a list format on the app's UI.

[0826] Step 11:

[0827] Server: Based on the proposed menu, it identifies any missing ingredients and searches for information to suggest where to purchase them.

[0828] Input: Customized menu

[0829] Output: Information on missing ingredients and where to purchase them.

[0830] Specific operation: The server analyzes the menu data, identifies missing ingredients, and searches for suppliers from affiliated e-commerce platforms.

[0831] Step 12:

[0832] Terminal: Presents a generated list of potential purchases to the user and provides a link to the online store.

[0833] Input: Purchase candidate list

[0834] Output: A list of purchase options and links presented to the user.

[0835] Specific action: The device displays a list of ingredients containing a "purchase link" within the app.

[0836] Step 13:

[0837] Server: Collects data from wearable devices and food records to analyze the user's health status.

[0838] Input: Wearable device and meal record data

[0839] Output: Analysis results

[0840] Specific operation: The server uses a Python script to aggregate and analyze data.

[0841] Step 14:

[0842] Server: Generates health feedback and advice based on analysis results, and creates notification messages.

[0843] Input: Analysis results

[0844] Output: Feedback and advice

[0845] Specific operation: The server checks the user's unfulfilled goal and generates a message such as, "You are 1000 steps away from achieving your goal."

[0846] Step 15:

[0847] Device: Generated feedback and advice are sent to the user via push notifications.

[0848] Input: Feedback and advice

[0849] Output: Push notification to the user

[0850] Specific action: The device sends a notification using the smartphone's push notification function.

[0851] The above outlines the specific processing steps of this system's program. This allows users to receive personalized meal suggestions based on their individual health data and ingredient information, enabling them to efficiently purchase ingredients and manage their health.

[0852] (Application Example 1)

[0853] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0854] In modern times, efficient and healthy dietary management is a crucial challenge for many people. However, in our busy lives, it's not easy to select appropriate recipes based on individual health conditions and current food availability, and to find suppliers for necessary ingredients. Furthermore, there is a lack of efficient feedback systems that link health data and dietary data. To address these challenges, there is a need for a system that utilizes users' health data, proposes individually customized recipes based on ingredient information, and allows for the quick acquisition of necessary ingredients.

[0855] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0856] In this invention, the server includes means for communicating with a device for acquiring physical information, means for identifying ingredients from captured images, means for suggesting meals based on the identified ingredient information and the user's physical information, means for generating customized recipes based on the user's health data and ingredient information using a generative AI model, and means for identifying missing ingredients and suggesting purchases in cooperation with partner food delivery services. This enables the user to smoothly manage their diet efficiently and healthily.

[0857] 1. A "device for acquiring physical information" refers to a device that uses wearable devices, smartphones, etc., to acquire health data such as the user's heart rate, steps taken, and calories burned.

[0858] 2. "Means of communication" refers to methods and protocols for sending and receiving data over a network.

[0859] 3. "Means for identifying food ingredients from captured images" refers to technologies that use image recognition technology to identify food ingredients from photographs taken with a camera.

[0860] 4. "Identified food information" refers to information such as the name and quantity of food items identified through image recognition.

[0861] 5. "User's physical information" refers to health-related data such as the user's age, gender, heart rate, and steps taken.

[0862] 6. "Means of providing meal suggestions" refers to a method of generating appropriate meal menus and recipes based on the user's physical information and identified ingredient information.

[0863] 7. A "generative AI model" refers to an artificial intelligence model that generates customized recipes based on the user's health data and ingredient information.

[0864] 8. A "customized recipe" refers to a specific recipe generated based on the user's individual health condition and ingredient availability.

[0865] 9. "Means of identifying missing ingredients" refers to techniques for identifying ingredients that are not present in the refrigerator based on the proposed recipe.

[0866] 10. "Partner food delivery services" refers to service providers that allow customers to order ingredients and food online and have them delivered.

[0867] This invention is a system that suggests customized recipes based on the user's physical information and food information, and indicates where to purchase the necessary ingredients. This system is implemented using the following hardware and software.

[0868] Hardware and software to be used

[0869] 1. Hardware

[0870] Wearable devices: Devices that acquire physical information such as the user's heart rate, steps taken, and calories burned.

[0871] Smartphone: A device used to take pictures of food items inside the refrigerator and upload them to an application.

[0872] 2. Software

[0873] Image recognition technology: This technology identifies food items from images taken with a smartphone. An example is the Python library `face_recognition`.

[0874] Generative AI Model: An artificial intelligence model that generates customized recipes based on the user's health data and ingredient information. OpenAI's GPT-4 API, among others, is used.

[0875] System Operation Description

[0876] User registration and profile settings

[0877] Users install the application and, upon first launch, enter their name, age, gender, height, weight, allergy information, preferred foods, and health goals. This allows the user's individual profile to be saved on the server.

[0878] Integration with wearable devices

[0879] The server periodically receives data such as heart rate, steps taken, and calories burned from wearable devices and stores it in a database in real time. This allows the user's health status to be constantly monitored.

[0880] Making a list of ingredients in the refrigerator

[0881] The user takes a photo of the inside of their refrigerator with their smartphone and uploads it to the application. The server uses image recognition technology to identify the food items in the photo and generates a list of those items. These identified items are added to the user's inventory list and managed as current inventory information.

[0882] Generating customized recipes and menus

[0883] The server uses a generative AI model to create a customized weekly meal plan based on the user's health data, ingredient inventory list, and preferences. The generated meal plan, along with necessary nutrient and calorie information, is presented to the user via their smartphone.

[0884] Specific example:

[0885] If the user's health status does not meet the target health level, a generative AI model will be used to suggest recipes.

[0886] Examples of prompt statements to use:

[0887] User information: {'name': 'User 1', 'age': 30, 'gender': 'male', 'height': 175, 'weight': 70, 'allergies': ['dairy products'], 'preferences': ['vegetables', 'chicken']}

[0888] Health data: {'heart_rate': 70, 'steps': 8000, 'calories_burned': 500}

[0889] Ingredients in the refrigerator: ['Tomato', 'Carrot', 'Chicken']

[0890] Based on this, please propose a customized recipe.

[0891] Suggestions for places to buy groceries

[0892] The server lists the necessary ingredients based on the suggested menu and identifies any missing ingredients. Next, it searches for ingredient purchase options from partner food delivery services and generates links to the nearest purchase locations or online stores. These purchase option lists are presented to the user via their smartphone, along with links to online stores.

[0893] Health data feedback

[0894] The server aggregates data from wearable devices and food records to analyze the user's health status. Based on the analysis, it generates health feedback and advice, and creates notification messages. The generated feedback and advice are delivered to the user via push notifications on their smartphone.

[0895] This allows users to quickly receive personalized meal suggestions based on their health data and ingredient information, enabling them to efficiently purchase ingredients and live a healthy lifestyle.

[0896] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0897] Step 1:

[0898] The user installs the application and sets up a profile upon first launch.

[0899] Input: Name, age, gender, height, weight, allergy information, favorite foods, health goals.

[0900] Data processing: Use this data to generate individual user profiles.

[0901] Output: The user's profile is saved to the server's database.

[0902] Step 2:

[0903] The server periodically receives health data from wearable devices.

[0904] Input: Data such as heart rate, steps taken, and calories burned, transmitted from a wearable device.

[0905] Data processing: Received data is recorded in a database in real time to monitor the user's health status.

[0906] Output: The latest health data will be updated.

[0907] Step 3:

[0908] Users take photos of the food in their refrigerator with their smartphones and upload the images to the app.

[0909] Input: A photo showing the food inside a refrigerator.

[0910] Data processing: The server uses image recognition technology to identify ingredients in the photograph.

[0911] Output: A list of recognized ingredients is generated and saved to the database.

[0912] Step 4:

[0913] The server generates customized recipes based on the user's health data and ingredient inventory list.

[0914] Input: Prompt text for the generated AI model (user profile, health data, list of identified foods).

[0915] Data processing: Generate customized recipes using a generative AI model (e.g., GPT-4).

[0916] Output: The generated recipe and necessary nutrient and calorie information are produced.

[0917] Step 5:

[0918] The server lists the necessary ingredients based on the proposed recipe and identifies any missing ingredients.

[0919] Input: Generated recipe and current ingredient inventory information.

[0920] Data processing: Identify and list any missing ingredients.

[0921] Output: A list of missing ingredients is generated.

[0922] Step 6:

[0923] The server searches for grocery purchase options from partner food delivery services and generates links to the nearest purchase locations and online stores.

[0924] Input: List of missing ingredients.

[0925] Data processing: Search the database of food delivery services to obtain information on potential purchases.

[0926] Output: A list of potential purchases and links to online stores are generated.

[0927] Step 7:

[0928] The device presents the user with a generated recipe, information on necessary nutrients and calories, and a list of recommended purchases.

[0929] Input: Generated recipe, required nutrient and calorie information, a list of suggested purchases, and links to online stores.

[0930] Data processing: Converts information into a format that is easy for users to understand.

[0931] Output: It will be displayed on the smartphone screen.

[0932] Step 8:

[0933] The server aggregates data from wearable devices and food records to analyze the user's health status.

[0934] Input: User's health data, dietary record data.

[0935] Data processing: Use analytical algorithms to evaluate the user's health status.

[0936] Output: Health feedback and advice are generated.

[0937] Step 9:

[0938] The device will send the generated feedback and advice to the user via push notifications.

[0939] Input: Feedback and advice sent from the server.

[0940] Data processing: Convert to a format that can be displayed as a push notification.

[0941] Output: It will be displayed as a push notification on your smartphone.

[0942] Through the above processing steps, users can quickly obtain customized meal suggestions based on their health data and ingredient information, enabling them to efficiently purchase ingredients and lead a healthy lifestyle.

[0943] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0944] This invention relates to a system that acquires a user's physical information and emotional state, and presents meal suggestions and locations for purchasing ingredients. The following describes a specific form for realizing this system.

[0945] User registration and profile settings

[0946] 1. User: Install the application and, upon first launch, enter your name, age, gender, height, weight, and information about allergies, preferred foods, and health goals. This information will be used to create your individual profile.

[0947] 2. Server: Saves the entered information to the database and creates a profile for each user.

[0948] Integration with wearable devices

[0949] 1. User: Link the wearable device and application in the settings screen.

[0950] 2. Terminal: Regularly receives data such as heart rate, steps taken, and calories burned from a wearable device.

[0951] 3. Server: Receives device data sent from terminals and stores it in the database.

[0952] Making a list of ingredients in the refrigerator

[0953] 1. User: Take a photo of the inside of the refrigerator with your smartphone and upload it to the application.

[0954] 2. Terminal: Uses image recognition technology to identify ingredients in a photograph and creates an ingredient list.

[0955] 3. Server: Stores the ingredient list in a database and manages it as the user's current inventory information.

[0956] Specific example

[0957] User: Upload a photo of the refrigerator to the application.

[0958] Terminal: Analyzes the photo and identifies "tomato, carrot, and chicken breast."

[0959] Server: Identify these ingredients and add them to the user's inventory list.

[0960] Generating customized recipes and menus

[0961] 1. Server: Uses generative AI to create a customized weekly meal plan based on the user's health data, ingredient inventory list, preferences, and allergy information.

[0962] 2. Terminal: Presents the generated menu and necessary nutrient and calorie information to the user.

[0963] Emotional recognition and adaptation of meal suggestions using an emotional engine

[0964] 1. User: Inputs facial expressions and voice through the application. Also, notifies emotional states by inputting text.

[0965] 2. Device: The emotion engine analyzes the user's facial expressions, voice, or text data to recognize their current emotional state.

[0966] 3. Server: Analyzes recognized emotional information in conjunction with health data and adapts meal suggestions based on the user's emotions.

[0967] Specific example

[0968] User: Type the text "I'm tired today" into the chatbot within the application.

[0969] Terminal: The emotion engine analyzes this text data and recognizes that the user is tired.

[0970] Server: Suggest highly nutritious ingredients and easy-to-prepare dishes to reduce user fatigue.

[0971] Suggestions for places to buy groceries

[0972] 1. Server: Identify any missing ingredients based on the proposed menu.

[0973] 2. Server: For any missing ingredients, it searches for potential purchase options from partner e-commerce platforms and gathers information on the nearest places to buy them and online stores.

[0974] 3. Terminal: Presents the generated list of potential purchases to the user and provides a link to the online store.

[0975] Feedback on health data and emotional data

[0976] 1. Server: Collects data from wearable devices, meal records, and emotional data to analyze the user's health status.

[0977] 2. Server: Generates health feedback and advice, emotion-based recommendations, and notification messages based on the analysis results.

[0978] 3. Device: Send feedback and advice to users via push notifications.

[0979] Specific example

[0980] Server: Verify that the user's activity level has not reached 70% of the target.

[0981] The device notifies the user that "You can reach your goal by walking 1000 more steps." If it detects that the user's stress level is high, it suggests, "Try some relaxing herbal tea."

[0982] The above describes the embodiments of the present invention. In this form, users can receive not only individual health data and food information, but also optimal meal suggestions tailored to their mood at any given time, enabling them to efficiently purchase food ingredients and lead a healthy and balanced life.

[0983] The following describes the processing flow.

[0984] User registration and profile settings

[0985] Step 1:

[0986] User: Install the application and, upon first launch, enter your name, age, gender, height, weight, and information about allergies, preferred foods, and health goals.

[0987] Step 2:

[0988] Terminal: Sends the entered information to the server.

[0989] Step 3:

[0990] Server: Stores received information in a database and creates user profiles.

[0991] Integration with wearable devices

[0992] Step 4:

[0993] User: Link the wearable device and application in the settings screen.

[0994] Step 5:

[0995] Device: Once the connection is successfully completed, enable the setting to periodically receive data such as heart rate, steps, and calories burned from the wearable device.

[0996] Step 6:

[0997] Server: Receives device data sent from terminals and stores it in the database.

[0998] Making a list of ingredients in the refrigerator

[0999] Step 7:

[1000] User: Take a photo of the inside of the refrigerator with your smartphone and upload it to the application.

[1001] Step 8:

[1002] Device: Uses AI image recognition technology to identify ingredients in a photograph and creates an ingredient list.

[1003] Step 9:

[1004] Terminal: Sends the generated ingredient list to the server.

[1005] Step 10:

[1006] Server: Stores the received ingredient list in the database and manages it as the user's current inventory information.

[1007] Generating customized recipes and menus

[1008] Step 11:

[1009] Server: Uses generative AI to create a customized weekly meal plan based on the user's health data, ingredient inventory list, preferences, and allergy information.

[1010] Step 12:

[1011] Server: Sends the generated menu to the terminal.

[1012] Step 13:

[1013] Terminal: Displays the received menu and necessary nutrient and calorie information to the user.

[1014] Emotional recognition and adaptation of meal suggestions using an emotional engine

[1015] Step 14:

[1016] User: Notifies emotional states by inputting facial expressions, voice, or text through the application.

[1017] Step 15:

[1018] Device: The emotion engine analyzes the user's facial expressions, voice, or text data to recognize their current emotional state.

[1019] Step 16:

[1020] Server: Analyzes recognized emotional information in conjunction with health data and adapts meal suggestions based on the user's emotions.

[1021] Suggestions for places to buy groceries

[1022] Step 17:

[1023] Server: Identify any missing ingredients based on the proposed menu.

[1024] Step 18:

[1025] Server: Searches for potential purchase options from partner e-commerce platforms for missing ingredients, gathering information on the nearest places to buy them and online stores.

[1026] Step 19:

[1027] Server: Sends a list of potential purchases to the terminal.

[1028] Step 20:

[1029] Terminal: Presents a generated list of potential purchases to the user and provides a link to the online store.

[1030] Feedback on health data and emotional data

[1031] Step 21:

[1032] Server: Collects data from wearable devices, meal records, and emotional data to analyze the user's health status.

[1033] Step 22:

[1034] Server: Generates health feedback and advice, emotion-based recommendations, and notification messages based on analysis results.

[1035] Step 23:

[1036] Server: Sends the generated feedback and advice to the terminal.

[1037] Step 24:

[1038] Device: Send feedback and advice to users via push notifications.

[1039] The above describes the specific actions taken at each processing step. This detailed processing flow allows users to receive optimal meal suggestions based on the latest health data and ingredient information, efficiently purchase ingredients, and lead a healthy and balanced life. In addition, it provides even more personalized support by offering suggestions that take into account the user's emotional state.

[1040] (Example 2)

[1041] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[1042] Conventional meal suggestion systems based solely on the user's physical data and ingredient information, failing to consider the user's emotional state. Furthermore, suggestions for ingredient sourcing locations were limited, failing to integrate information from online stores and physical retail outlets. Therefore, there was a need to provide users with optimal and efficient meal suggestions and ingredient purchasing methods.

[1043] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1044] In this invention, the server includes means for communicating with equipment for acquiring physical data, means for identifying food ingredients from captured content, means for suggesting meals based on the identified food ingredient data, the user's physical data, and emotional state, and means for identifying missing ingredients based on the suggested meal and indicating where to obtain them. This enables meal suggestions that take into account the user's emotional state in addition to their physical data and food ingredient information, and further enables the provision of food ingredient acquisition locations by integrating information from e-commerce platforms and physical stores, thereby providing optimal and efficient meal suggestions and means for purchasing ingredients.

[1045] "Physical data" refers to information about the user's body, including data such as height, weight, heart rate, steps taken, and calories burned.

[1046] "Equipment" refers to wearable devices, sensor devices, and other devices used to acquire the user's physical data.

[1047] "Content" refers to images and photographs taken by users, including, in particular, food items in a refrigerator and cooked meals.

[1048] "Food ingredient data" refers to information such as the type, quantity, and availability of food items identified from the content.

[1049] "Emotional state" refers to information that indicates the user's current psychological and emotional state, including, for example, fatigue, stress, and satisfaction.

[1050] "Meal suggestions" refers to the act of suggesting optimal meal menus and recipes based on the user's physical data and emotional state.

[1051] "Place of acquisition" refers to the location where the ingredients needed for the proposed meal can be purchased, including online stores and physical stores.

[1052] An "e-commerce platform" refers to a website or application that facilitates the buying and selling of goods online, and examples include e-commerce sites.

[1053] A "physical store" refers to an actual store or supermarket where users can visit in person to purchase groceries.

[1054] This invention relates to a system that acquires a user's physical data and emotional state, and presents personalized meal suggestions and locations for obtaining ingredients. The following describes specific embodiments of this system.

[1055] Overall structure

[1056] This system consists of a device (wearable device) for acquiring physical data, the user's smartphone (terminal), and a server connected to them. The server receives and analyzes the data and provides meal suggestions and locations for data acquisition.

[1057] User registration and profile settings

[1058] Users first install the application on their smartphone and, upon initial launch, enter their name, age, gender, height, weight, allergy information, preferred foods, and health goals. This creates an individual profile. The server stores the information entered by the user in a database and builds the profile. The application interface is designed to allow users to easily enter the necessary information.

[1059] Integration with wearable devices

[1060] The user pairs a wearable device (e.g., a wristwatch-type heart rate monitor) with the application via the smartphone's settings screen. The wearable device periodically sends data such as heart rate, steps taken, and calories burned to the device. The device periodically sends this data to a server. The server stores the received data in a database and manages the user's health data.

[1061] Making a list of ingredients in the refrigerator

[1062] The user takes a photo of the inside of their refrigerator with their smartphone and uploads it to the application. The device uses image recognition technology (e.g., Amazon Rekognition) to identify the food items in the photo and create a list of those items. The server stores the identified food list in a database and manages it as the user's current inventory information.

[1063] Specific example

[1064] When a user uploads a photo of their refrigerator to the application, the device analyzes the image and identifies "tomatoes, carrots, and chicken breast." The server saves these ingredients to a database and updates the inventory list.

[1065] Generating customized recipes and menus

[1066] The server uses generative AI (e.g., OpenAI's GPT-4) to create a customized weekly meal plan based on the user's health data, ingredient inventory list, preferences, and allergy information. The terminal then presents the generated meal plan along with necessary nutrient and calorie information to the user.

[1067] Examples of prompt statements

[1068] "I'm a woman in my 30s, aiming to lose weight. I have tomatoes, carrots, and chicken breast in my refrigerator. Please create a low-calorie, nutritionally balanced meal plan for one week."

[1069] Emotional recognition and adaptation of meal suggestions using an emotional engine

[1070] Users notify their emotional state through the application by inputting facial expressions, voice, or text. The emotion engine analyzes this input data to recognize the user's current emotional state. The server then comprehensively analyzes the recognized emotional state and physical data to provide meal suggestions based on the user's emotions.

[1071] Specific example

[1072] When a user types "I'm tired today" into the in-app chatbot, the device's emotion engine analyzes this text data and recognizes that the user is tired. The server then suggests highly nutritious ingredients and easy-to-prepare dishes.

[1073] Suggestions for locations to obtain ingredients

[1074] The server identifies any missing ingredients based on the suggested menu and searches for purchase options from partner e-commerce platforms and physical stores. The terminal presents the generated list of purchase options to the user and provides links to online stores.

[1075] Feedback on health data and emotional data

[1076] The server aggregates data from wearable devices, meal logs, and emotional data to analyze the user's health status. Based on the analysis, it generates feedback and advice and notifies the user. The device delivers this feedback and advice to the user via push notifications. For example, it provides specific advice such as, "You can achieve your goal by walking another 1000 steps," or emotionally-based recommendations such as, "Try a relaxing herbal tea."

[1077] The above describes a specific embodiment for carrying out the present invention. This allows users to receive not only individual health data and food information, but also optimal meal suggestions tailored to their mood at any given time, enabling them to efficiently purchase food ingredients and lead a healthy and balanced life.

[1078] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1079] Program execution flow and detailed explanation of each processing step

[1080] User registration and profile settings

[1081] Step 1:

[1082] The user installs the application and launches it for the first time.

[1083] Input: None

[1084] Output: Display of the app's welcome screen

[1085] Specific operation: The user downloads and launches the app. The interface is designed to be intuitive and easy to understand.

[1086] Step 2:

[1087] The user enters their name, age, gender, height, weight, allergy information, favorite foods, and health goals.

[1088] Input: User information (name, age, gender, height, weight, allergy information, favorite foods, health goals)

[1089] Output: User profile data

[1090] Specific action: The user enters the required information into the input form and presses the submit button.

[1091] Step 3:

[1092] The server saves the entered information to the database and builds a user profile.

[1093] Input: User profile data

[1094] Output: Saved to database successfully

[1095] Specific operation: The server receives profile data and stores it in the database. A relational database such as MySQL is used for this database.

[1096] Integration with wearable devices

[1097] Step 1:

[1098] The user pairs the wearable device and the application in the app settings screen.

[1099] Input: Pairing request

[1100] Output: Pairing complete notification

[1101] Specific steps: The user selects the wearable device on their smartphone's Bluetooth settings screen and pairs it.

[1102] Step 2:

[1103] The device periodically receives data such as heart rate, steps taken, and calories burned from the wearable device.

[1104] Input: Device data (heart rate, steps, calories burned)

[1105] Output: Automatic saving of received data at regular intervals.

[1106] Specific operation: The device periodically (e.g., every hour) retrieves data from the wearable device and saves it locally.

[1107] Step 3:

[1108] The server receives device data sent from the terminal and stores it in the database.

[1109] Input: Device data from the terminal

[1110] Output: Saved to database successfully

[1111] Specific operation: At regular intervals (e.g., every 24 hours), the server receives data from the terminal, analyzes it, and saves it to the database.

[1112] Making a list of ingredients in the refrigerator

[1113] Step 1:

[1114] The user takes a photo of the inside of the refrigerator with their smartphone and uploads it to the application.

[1115] Input: Photo of the inside of the refrigerator

[1116] Output: Upload complete notification

[1117] Specific action: The user uses their smartphone camera to take a picture of the inside of the refrigerator and uploads the image to the app.

[1118] Step 2:

[1119] The device uses image recognition technology to identify food items in the photograph.

[1120] Input: Uploaded photo

[1121] Output: Identified ingredient list

[1122] Specific operation: The device uses image recognition APIs such as Amazon Rekognition to automatically analyze and identify food items in a photograph.

[1123] Step 3:

[1124] The server stores a list of identified ingredients in a database and manages it as the user's current inventory information.

[1125] Input: Identified ingredient list

[1126] Output: Saved to database successfully

[1127] Specific operation: The server receives the identified ingredient information and stores it in the database.

[1128] Generating customized recipes and menus

[1129] Step 1:

[1130] The server uses generative AI to create a customized weekly meal plan based on the user's health data, ingredient inventory list, preferences, and allergy information.

[1131] Input: Health data, food inventory list, preferences and allergy information

[1132] Output: Customized menu

[1133] Specific operation: The server sends prompt messages to the AI ​​model (e.g., OpenAI's GPT-4) and receives the generated menu.

[1134] Step 2:

[1135] The device displays the generated menu and information on necessary nutrients and calories to the user.

[1136] Input: Customized menu

[1137] Output: Presentation of menu information

[1138] Specific operation: The terminal displays the menu information received from the server on its screen, along with information on necessary nutrients and calories.

[1139] Emotional recognition and adaptation of meal suggestions using an emotional engine

[1140] Step 1:

[1141] Users can input facial expressions and voices through the application, or input text to notify their emotional state.

[1142] Input: Facial expressions, voice, or text data

[1143] Output: Notification of emotional state

[1144] Specific operation: Users input their emotional state through the app using their facial photo, voice, or text data.

[1145] Step 2:

[1146] The device uses an emotion engine to analyze facial expressions, voice, or text to recognize the user's current emotional state.

[1147] Input: Emotional data (facial expressions, voice, text)

[1148] Output: Recognized emotional state

[1149] Specific operation: The device uses an emotion recognition engine (e.g., Microsoft Azure Cognitive Services) to analyze input data and recognize the emotional state.

[1150] Step 3:

[1151] The server integrates and analyzes recognized emotional states and physical data to provide meal suggestions based on the user's emotions.

[1152] Input: Recognized emotional state, physical data

[1153] Output: Meal suggestions based on emotional state

[1154] Specific operation: Based on the analysis results, the server generates meal suggestions that include highly nutritious ingredients and easy-to-prepare dishes.

[1155] Suggestions for locations to obtain ingredients

[1156] Step 1:

[1157] The server identifies any missing ingredients based on the suggested menu.

[1158] Input: Suggested menu, ingredient inventory list

[1159] Output: List of missing ingredients

[1160] Specific operation: The server compares the menu with the inventory list to identify any missing ingredients.

[1161] Step 2:

[1162] The server searches for potential purchase options for ingredients that are currently out of stock, from partner e-commerce platforms and physical stores.

[1163] Input: List of missing ingredients

[1164] Output: Purchase candidate list

[1165] Specific operation: The server uses APIs from e-commerce platforms and physical stores to search for purchasing options for missing ingredients.

[1166] Step 3:

[1167] The device displays a list of potential purchases to the user and provides a link to the online store.

[1168] Input: Purchase candidate list

[1169] Output: Display of purchase candidate list

[1170] Specific operation: The device displays a list of potential purchases on the screen and provides the user with a link to each option.

[1171] Feedback on health data and emotional data

[1172] Step 1:

[1173] The server collects data from wearable devices, meal logs, and emotional data to analyze the user's health status.

[1174] Input: Wearable device data, food log data, emotional data

[1175] Output: Analysis results of health status

[1176] Specific operation: The server uses Python or similar languages ​​to execute analysis scripts and collect and analyze user health data.

[1177] Step 2:

[1178] The server generates feedback and advice based on the analysis results and notifies the user.

[1179] Input: Analysis results of health status

[1180] Output: Feedback and advice

[1181] Specific operation: The server generates appropriate advice and feedback for the user based on the analysis results and creates notification messages.

[1182] Step 3:

[1183] The device will send feedback and advice to the user via push notifications.

[1184] Input: Feedback and advice

[1185] Output: Notification message

[1186] Specific actions: The device uses push notifications to inform the user of specific advice, such as "You can reach your goal by walking 1000 more steps." It also provides emotion-based recommendations, such as "Try a relaxing herbal tea."

[1187] (Application Example 2)

[1188] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[1189] In modern society, maintaining good health requires individuals to plan appropriate meals and efficiently purchase the necessary ingredients. However, it takes considerable time and effort for users to choose appropriate meals, identify necessary ingredients, and purchase them themselves. Furthermore, while a user's emotional state often significantly influences their meal choices, there are limited systems that can appropriately reflect this in their meal suggestions. In addition, although the use of online stores and food delivery services has increased, there is a lack of systems that centrally manage these services and provide users with the most suitable purchasing options. A system that solves these problems is needed.

[1190] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[1191] In this invention, the server includes means for communicating with a device for acquiring physical information, means for identifying ingredients from captured images, means for suggesting meals based on the identified ingredient information, the user's physical information, and emotional state, and means for identifying necessary ingredients based on the suggested meal, indicating where to purchase them, and providing links to food delivery services. This allows the user to receive optimal meal suggestions based on their physical information and emotional state, and to purchase necessary ingredients quickly and efficiently. Furthermore, they can quickly obtain ingredients and prepared meals through food delivery, supporting their eating habits.

[1192] "Physical information" refers to data that quantifies the user's physical condition, such as heart rate, steps taken, and calories burned.

[1193] "Emotional state" refers to information that indicates the user's current psychological state or mood, analyzed from the user's facial expressions, voice, or text data.

[1194] "Meal suggestions" is a process that generates and presents meal menus suitable for the user based on the user's physical information and emotional state.

[1195] "Food identification" is the process of using AI technology to identify food items from images taken by the user and analyzing their types.

[1196] "Presenting a place to purchase" refers to the process of showing users information about online stores or physical shops where they can purchase the ingredients they are missing.

[1197] A "food delivery link" is an online link that allows users to order the necessary ingredients and dishes based on a suggested meal menu.

[1198] This invention relates to a system that acquires a user's physical information and emotional state, and presents meal suggestions and locations for purchasing ingredients. An embodiment of this system is described in detail below.

[1199] User registration and profile settings

[1200] Users first install the application on their smartphone and, upon initial launch, enter their name, age, gender, height, weight, allergy information, preferred foods, and health goals. The server stores this information in a database and creates a profile for each user.

[1201] Integration with wearable devices

[1202] The user connects the wearable device and the application through the application's settings screen. The terminal periodically receives data such as heart rate, steps taken, and calories burned from the wearable device. The server stores the received device data in a database.

[1203] Making a list of ingredients in the refrigerator

[1204] The user takes a photo of the inside of their refrigerator with their smartphone and uploads it to the application. The device uses image recognition technology to identify the food items in the photo and creates a list of those items. The server stores the list of items in a database and manages it as the user's current inventory information.

[1205] Generating customized recipes and menus

[1206] The server uses an AI model to generate a customized weekly meal plan based on the user's health data, ingredient inventory list, preferences, and allergy information. The terminal then displays the generated meal plan along with information on necessary nutrients and calories to the user.

[1207] Emotional recognition and adaptation of meal suggestions using an emotional engine

[1208] Users notify their emotional state by inputting facial expressions, voice, or text through the application. The device uses an emotion engine to analyze the user's facial expressions, voice, or text data to recognize their current emotional state. The server analyzes the recognized emotional information in conjunction with health data and adapts meal suggestions based on the user's emotions.

[1209] Suggestions for places to buy groceries

[1210] The server identifies any missing ingredients based on the suggested menu. It also searches for purchase options from partner e-commerce platforms and the nearest supermarket, and provides links to online stores for those missing ingredients.

[1211] Feedback on health data and emotional data

[1212] The server aggregates data from wearable devices, meal records, and emotional data to analyze the user's health status. Based on the analysis, it generates health feedback and advice, emotional recommendations, and creates notification messages. The device then delivers the feedback and advice to the user via push notifications.

[1213] Hardware and software used

[1214] Hardware: Smartphones, wearable devices, refrigerators with cameras

[1215] Software: TensorFlow, OpenCV, Flask, Twilio, Generative AI Models

[1216] Specific example

[1217] The user uploads a photo of their refrigerator to the application, and image analysis identifies "tomatoes, carrots, and chicken breast." If the user enters the text "I'm tired today," the emotion engine analyzes this and recognizes that the user is tired. Based on this information, the server suggests highly nutritious ingredients and easy-to-prepare dishes. For example, the generative AI model is prompted with the message, "The user is currently feeling tired; please suggest an optimal dinner," and then generates suggestions.

[1218] In this way, this system allows users to receive optimal meal suggestions tailored to their physical information and emotional state, and to quickly and efficiently purchase the necessary ingredients. Furthermore, because meals can be easily obtained through food delivery, the system supports users in leading a healthy lifestyle.

[1219] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1220] Step 1: User registration and profile setup

[1221] When a user first launches an application installed on their smartphone, they enter their name, age, gender, height, weight, allergy information, preferred foods, and health goals. This information is the input data, which the server stores in a database to create a profile for each user. The information stored in the database is the output. This initiates individual data management for each user.

[1222] Step 2: Integration with wearable devices

[1223] The user connects the wearable device to the application through the application's settings screen. The terminal periodically receives data such as heart rate, steps taken, and calories burned from the wearable device. The input is biometric data transmitted from the wearable device, which the server receives and stores in a database. The stored biometric data becomes the output. This allows the user's physical information to be obtained in real time.

[1224] Step 3: Make a list of the ingredients in your refrigerator.

[1225] The user takes a photo of the inside of their refrigerator with their smartphone and uploads it to the application. The application receives the image and uses OpenCV and TensorFlow to identify the food items in the photo using image recognition technology. The output is a list of identified food items, which the server saves to a database. This process manages the current food inventory information.

[1226] Step 4: Generating customized recipes and menus

[1227] The server retrieves the user's health data, ingredient inventory list, preferences, and allergy information, and generates a week's worth of customized meal plans by inputting prompts into a generative AI model. For example, by inputting the prompt, "The user is currently feeling tired; please suggest an optimal dinner," the generative AI model will suggest a meal plan. The generated meal plan is the output, and the terminal presents it to the user. The meal suggestions are displayed with specific nutritional and calorie information.

[1228] Step 5: Emotional recognition and adaptation of meal suggestions by the emotional engine

[1229] Users notify the system of their emotional state by inputting facial expressions, voice, or text through the application. The terminal uses an emotion engine to analyze the user's current emotional state from the facial expressions, voice, and text data. This analysis result becomes the input data, which the server acquires and analyzes together with the user's health data. Based on the analysis result, the system provides meal suggestions appropriate to the user's emotional state, and these suggestions become the output.

[1230] Step 6: Suggesting places to buy groceries

[1231] The server identifies missing ingredients based on the suggested menu. These missing ingredients are the input data, and the server searches for purchase options from e-commerce platforms and the nearest supermarket. The output is a list of purchase options, which the terminal presents to the user, providing links to online stores. This allows the user to efficiently purchase the missing ingredients.

[1232] Step 7: Feedback on health and emotional data

[1233] The server collects data from wearable devices, meal records, and emotional data to analyze the user's health status. The input is the various collected data, and the server generates feedback and advice based on its analysis. This is the output, and the device communicates it to the user via push notifications, etc. For example, messages such as "You can achieve your goal by walking 1000 more steps" or emotional recommendations such as "Try a relaxing herbal tea" might be displayed.

[1234] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1235] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1236] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[1237] [Third Embodiment]

[1238] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[1239] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1240] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1241] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[1242] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1243] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1244] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1245] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1246] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1247] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1248] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1249] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[1250] This invention relates to a system that acquires a user's physical information, identifies food ingredients from captured images, makes meal suggestions based on this information, and indicates where to purchase the necessary ingredients. The following describes a specific form for realizing this system.

[1251] User registration and profile settings

[1252] 1. User: Install the application and, upon first launch, enter your name, age, gender, height, weight, and information about allergies, preferred foods, and health goals. This information will be used to create your individual profile.

[1253] 2. Server: Saves the entered information to the database and creates a profile for each user.

[1254] Integration with wearable devices

[1255] 1. User: Link the wearable device and application in the settings screen.

[1256] 2. Terminal: Regularly receives data such as heart rate, steps taken, and calories burned from wearable devices.

[1257] 3. Server: Saves received data to the database in real time and monitors the user's health status.

[1258] Making a list of ingredients in the refrigerator

[1259] 1. User: Take a photo of the inside of the refrigerator with your smartphone and upload it to the application.

[1260] 2. Terminal: Uses image recognition technology to identify ingredients in a photograph and generates an ingredient list.

[1261] 3. Server: Stores the ingredient list in a database and manages it as the user's current ingredient inventory information.

[1262] Specific example

[1263] User: Upload a photo of the refrigerator to the application.

[1264] Terminal: Analyzes the photo and identifies "tomato, carrot, and chicken breast."

[1265] Server: Adds these identified ingredients to the user's inventory list and saves them as current inventory information.

[1266] Generating customized recipes and menus

[1267] 1. Server: Uses generative AI to create a customized weekly meal plan based on the user's health data, ingredient inventory list, and preferences.

[1268] 2. Terminal: Presents the generated menu and necessary nutrient and calorie information to the user.

[1269] Suggestions for places to buy groceries

[1270] 1. Server: Based on the proposed menu, list the necessary ingredients and identify any missing ingredients.

[1271] 2. Server: Searches for potential grocery purchases from partner e-commerce platforms and generates links to the nearest stores or online shops.

[1272] 3. Terminal: Presents the generated list of potential purchases to the user and provides a link to the online store.

[1273] Specific example

[1274] Server: The server generates a weekly menu using tomatoes, carrots, and chicken breast, and suggests recipes such as "Tomato Chicken Stew."

[1275] Terminal: Displays these recipes, a list of required ingredients, and cooking instructions to the user.

[1276] Health data feedback

[1277] 1. Server: Collects data from wearable devices and food records to analyze the user's health status.

[1278] 2. Server: Generates health feedback and advice based on analysis results, and creates notification messages.

[1279] 3. Device: Send the generated feedback and advice to the user via push notification.

[1280] Specific example

[1281] Server: Verify that the user's activity level has not reached 70% of the target.

[1282] Device: Notifies the user with the message, "Walk another 1000 steps to reach your goal."

[1283] The above describes the embodiments of the present invention. This system allows users to receive customized meal suggestions based on their individual health data and food information, enabling them to efficiently purchase food ingredients and lead a healthy lifestyle.

[1284] The following describes the processing flow.

[1285] User registration and profile settings

[1286] Step 1:

[1287] User: Install the application and, upon first launch, enter your name, age, gender, height, weight, and information about allergies, preferred foods, and health goals.

[1288] Step 2:

[1289] Terminal: Sends the entered information to the server.

[1290] Step 3:

[1291] Server: Stores received information in a database and creates user profiles.

[1292] Integration with wearable devices

[1293] Step 4:

[1294] User: In the application's settings screen, link the wearable device with the application.

[1295] Step 5:

[1296] Device: Once the connection is successfully completed, enable the setting to periodically receive data such as heart rate, steps, and calories burned from the wearable device.

[1297] Step 6:

[1298] Server: Receives device data sent from terminals and stores it in the database.

[1299] Making a list of ingredients in the refrigerator

[1300] Step 7:

[1301] User: Take a photo of the inside of the refrigerator with your smartphone and upload it to the application.

[1302] Step 8:

[1303] Device: Uses AI image recognition technology to identify ingredients in a photograph and creates an ingredient list.

[1304] Step 9:

[1305] Terminal: Sends the generated ingredient list to the server.

[1306] Step 10:

[1307] Server: Stores the received ingredient list in the database and manages it as the user's current inventory information.

[1308] Generating customized recipes and menus

[1309] Step 11:

[1310] Server: Uses generative AI to create a customized weekly meal plan based on the user's health data, ingredient inventory list, preferences, and allergy information.

[1311] Step 12:

[1312] Server: Sends the generated menu to the terminal.

[1313] Step 13:

[1314] Terminal: Displays the received menu and necessary nutrient and calorie information to the user.

[1315] Suggestions for places to buy groceries

[1316] Step 14:

[1317] Server: Identify any missing ingredients based on the proposed menu.

[1318] Step 15:

[1319] Server: Searches for potential purchase options from partner e-commerce platforms for missing ingredients, gathering information on the nearest places to buy them and online stores.

[1320] Step 16:

[1321] Server: Sends a list of potential purchases to the terminal.

[1322] Step 17:

[1323] Terminal: Presents a generated list of potential purchases to the user and provides a link to the online store.

[1324] Health data feedback

[1325] Step 18:

[1326] Server: Collects data from wearable devices and food records to analyze the user's health status.

[1327] Step 19:

[1328] Server: Generates health feedback and advice based on analysis results, and creates notification messages.

[1329] Step 20:

[1330] Server: Sends the generated feedback and advice to the terminal.

[1331] Step 21:

[1332] Device: Send feedback and advice to users via push notifications.

[1333] The above describes the detailed operation at each processing step. Through this detailed processing flow, users can always receive optimal meal suggestions based on the latest health data and ingredient information, efficiently purchase ingredients, and lead a healthy lifestyle.

[1334] (Example 1)

[1335] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1336] Maintaining a healthy diet is difficult in today's busy lifestyle. Furthermore, it's challenging for users to select appropriate foods and recipes based on their own health status and preferences. Even efficiently shopping for ingredients by identifying suitable locations is time-consuming. To address these issues, a system is needed that centrally manages users' health data and ingredient information, and provides optimal meal suggestions and ingredient purchasing locations based on this data.

[1337] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1338] In this invention, the server includes means for inputting the user's personal information and creating a profile; means for working with a wearable device to collect biometric data; means for analyzing images of the inside of a refrigerator taken and identifying ingredients; means for creating a customized menu using a generative AI model based on the identified ingredient information and the user's health data; and means for identifying missing ingredients based on the proposed menu and suggesting where to purchase them. As a result, the user can receive meal suggestions based on their health status and ingredient information, efficiently purchase necessary ingredients, and maintain a healthy lifestyle.

[1339] A "user" is an entity that uses the system to manage individual health data and food information, and to make meal suggestions and purchase food ingredients.

[1340] A "profile" is a set of individual information that includes the user's name, age, gender, height, weight, allergy information, preferred foods, and health goals.

[1341] A "wearable device" is an electronic device worn by a user to collect biometric data such as heart rate, steps taken, and calories burned.

[1342] "Biometric data" refers to data such as a user's heart rate, steps taken, and calories burned, collected through wearable devices and other means.

[1343] "Image analysis" is the process of extracting specific information from captured images, and is particularly used to identify food items inside a refrigerator.

[1344] "Food identification" is a method of identifying food items in a refrigerator through image analysis and listing that information.

[1345] A "generative AI model" is an artificial intelligence model that automatically generates customized menus based on the user's health data and ingredient information.

[1346] A "menu" refers to a meal plan or recipe suggested based on the user's health data and available ingredients.

[1347] "Place of purchase" refers to any location where you can purchase any missing ingredients based on the suggested menu, including online stores and physical stores.

[1348] "Health data" is a general term for a user's physical information, biometric data, and dietary history, and is used to manage their health status.

[1349] "Feedback" refers to improvement suggestions and advice provided based on the user's health status and activity level.

[1350] This invention relates to a system that acquires a user's physical information, identifies food ingredients from captured images, makes meal suggestions based on this information, and indicates where to purchase the necessary ingredients. Specific embodiments of this invention are described below.

[1351] User registration and profile settings

[1352] 1. User: First, the user installs the system's application on their smartphone. Upon first launching the application, they enter their name, age, gender, height, weight, allergy information, preferred foods, and health goals. This creates the user's individual profile.

[1353] 2. Server: The server stores the information entered by the user in a database and creates a profile for each user. A database such as MongoDB is used for this purpose.

[1354] Integration with wearable devices

[1355] 1. User: The user connects the wearable device and the application using Bluetooth in the application's settings screen.

[1356] 2. Terminal: The terminal periodically receives biometric data such as heart rate, steps taken, and calories burned from the wearable device. Bluetooth communication is used for this purpose.

[1357] 3. Server: The server stores incoming data in a database in real time and monitors the user's health status. A database such as PostgreSQL is used for this purpose.

[1358] Making a list of ingredients in the refrigerator

[1359] 1. User: The user takes a photo of the inside of the refrigerator with their smartphone and uploads it to the application.

[1360] 2. Terminal: The terminal uses image recognition technology (e.g., TensorFlow) to identify ingredients in the captured image and generate a list of ingredients.

[1361] 3. Server: The server stores the ingredient list in a database and manages it as the user's current ingredient inventory information. A database such as MySQL is used for this purpose.

[1362] Specific example

[1363] When a user uploads a photo of their refrigerator to the application, the device analyzes the photo and identifies "tomatoes, carrots, and chicken breast." The server adds these identified ingredients to the user's inventory list and saves it as current inventory information.

[1364] Generating customized recipes and menus

[1365] 1. Server: The server uses a generative AI model (e.g., OpenAI's GPT) to create a customized weekly meal plan based on the user's health data, ingredient inventory list, and preferences.

[1366] 2. Terminal: The terminal presents the user with the generated menu and information on necessary nutrients and calories.

[1367] Example of a prompt

[1368] Send the following prompt to the generating AI: "Create a one-week meal plan based on User A's health data and inventory information."

[1369] Suggestions for places to buy groceries

[1370] 1. Server: The server identifies any missing ingredients based on the proposed menu and suggests where to purchase them.

[1371] 2. Server: The server searches for potential grocery purchases from partner e-commerce platforms and generates links to the nearest purchase locations and online stores.

[1372] 3. Terminal: The terminal presents the user with a generated list of potential purchases and provides links to online stores.

[1373] Specific example

[1374] The server generates a week's worth of meal plans based on the user's health data and food inventory, suggesting recipes such as "Tomato Chicken Stew." The terminal displays these recipes, a list of required ingredients, and cooking instructions to the user.

[1375] Health data feedback

[1376] 1. Server: The server aggregates data from wearable devices and food records, and analyzes the user's health status.

[1377] 2. Server: The server generates health feedback and advice based on the analysis results and creates notification messages.

[1378] 3. Device: The device will send the generated feedback and advice to the user via push notifications.

[1379] Specific example

[1380] The server confirms that the user's activity level has not reached 70% of the target, and the device notifies the user, "Walk another 1000 steps to reach your goal."

[1381] The above describes the embodiments of the present invention. This system allows users to receive customized meal suggestions based on their health condition and food information, enabling them to efficiently purchase food ingredients and lead a healthy lifestyle.

[1382] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1383] Step 1:

[1384] User: Install the application on your smartphone and enter your personal information (name, age, gender, height, weight, allergy information, favorite foods, health goals) upon first launch.

[1385] Input: User's personal information

[1386] Output: Individual user profiles

[1387] Specific action: The user enters information into each field and presses the "Save" button.

[1388] Step 2:

[1389] Server: Stores the entered personal information in the database and creates a profile for each user.

[1390] Input: User's personal information

[1391] Output: Saved user profile

[1392] Specific operation: The server uses MongoDB to store personal information received from users in JSON format.

[1393] Step 3:

[1394] User: Link the wearable device and application in the settings screen.

[1395] Input: Connection information for wearable devices

[1396] Output: Connection established with wearable device

[1397] Specific steps: The user opens the Bluetooth settings, selects the wearable device, and presses the pairing button.

[1398] Step 4:

[1399] Terminal: Regularly receives data such as heart rate, steps taken, and calories burned from wearable devices.

[1400] Input: Data from a wearable device

[1401] Output: Biometric data temporarily stored on the device

[1402] Specific operation: The device uses Bluetooth communication to retrieve data and save it to local storage.

[1403] Step 5:

[1404] Server: Receives data from wearable devices via terminals and saves it to a database in real time.

[1405] Input: Biometric data transmitted from the device

[1406] Output: Biometric data stored in the database

[1407] Specific operation: The server uses PostgreSQL to execute INSERT statements into the database.

[1408] Step 6:

[1409] User: Take a photo of the inside of the refrigerator with your smartphone and upload it to the application.

[1410] Input: Image of the inside of a refrigerator

[1411] Output: Image files uploaded to the application

[1412] Specific action: The user takes a photo using the camera app and presses the upload button in the application.

[1413] Step 7:

[1414] Terminal: Uses image recognition technology to identify ingredients in captured images and generates an ingredient list.

[1415] Input: Uploaded image of the inside of a refrigerator

[1416] Output: List of identified ingredients

[1417] Specific operation: The device uses TensorFlow to perform image analysis and identify food ingredients. Example: "Tomato, carrot, chicken breast".

[1418] Step 8:

[1419] Server: Stores ingredient lists in a database and manages them as the user's current ingredient inventory information.

[1420] Input: Identified ingredient list

[1421] Output: Food inventory information stored in the database

[1422] Specific operation: The server uses MySQL to store ingredient information in a database.

[1423] Step 9:

[1424] Server: Uses a generative AI model to create a customized weekly meal plan based on the user's health data, ingredient inventory list, and preferences.

[1425] Input: User's health data, food inventory list, preferences

[1426] Output: Customized weekly meal plan

[1427] Specific operation: The server sends the following prompt to the generative AI model: "Create a weekly meal plan based on user A's health data and inventory information."

[1428] Step 10:

[1429] Terminal: Displays the generated menu and necessary nutrient and calorie information to the user.

[1430] Input: Customized menu

[1431] Output: Menu information presented to the user

[1432] Specific action: The device displays the menu in a list format on the app's UI.

[1433] Step 11:

[1434] Server: Based on the proposed menu, it identifies any missing ingredients and searches for information to suggest where to purchase them.

[1435] Input: Customized menu

[1436] Output: Information on missing ingredients and where to purchase them.

[1437] Specific operation: The server analyzes the menu data, identifies missing ingredients, and searches for suppliers from affiliated e-commerce platforms.

[1438] Step 12:

[1439] Terminal: Presents a generated list of potential purchases to the user and provides a link to the online store.

[1440] Input: Purchase candidate list

[1441] Output: A list of purchase options and links presented to the user.

[1442] Specific action: The device displays a list of ingredients containing a "purchase link" within the app.

[1443] Step 13:

[1444] Server: Collects data from wearable devices and food records to analyze the user's health status.

[1445] Input: Wearable device and meal record data

[1446] Output: Analysis results

[1447] Specific operation: The server uses a Python script to aggregate and analyze data.

[1448] Step 14:

[1449] Server: Generates health feedback and advice based on analysis results, and creates notification messages.

[1450] Input: Analysis results

[1451] Output: Feedback and advice

[1452] Specific operation: The server checks the user's unfulfilled goal and generates a message such as, "You are 1000 steps away from achieving your goal."

[1453] Step 15:

[1454] Device: Generated feedback and advice are sent to the user via push notifications.

[1455] Input: Feedback and advice

[1456] Output: Push notification to the user

[1457] Specific action: The device sends a notification using the smartphone's push notification function.

[1458] The above outlines the specific processing steps of this system's program. This allows users to receive personalized meal suggestions based on their individual health data and ingredient information, enabling them to efficiently purchase ingredients and manage their health.

[1459] (Application Example 1)

[1460] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1461] In modern times, efficient and healthy dietary management is a crucial challenge for many people. However, in our busy lives, it's not easy to select appropriate recipes based on individual health conditions and current food availability, and to find suppliers for necessary ingredients. Furthermore, there is a lack of efficient feedback systems that link health data and dietary data. To address these challenges, there is a need for a system that utilizes users' health data, proposes individually customized recipes based on ingredient information, and allows for the quick acquisition of necessary ingredients.

[1462] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1463] In this invention, the server includes means for communicating with a device for acquiring physical information, means for identifying ingredients from captured images, means for suggesting meals based on the identified ingredient information and the user's physical information, means for generating customized recipes based on the user's health data and ingredient information using a generative AI model, and means for identifying missing ingredients and suggesting purchases in cooperation with partner food delivery services. This enables the user to smoothly manage their diet efficiently and healthily.

[1464] 1. A "device for acquiring physical information" refers to a device that uses wearable devices, smartphones, etc., to acquire health data such as the user's heart rate, steps taken, and calories burned.

[1465] 2. "Means of communication" refers to methods and protocols for sending and receiving data over a network.

[1466] 3. "Means for identifying food ingredients from captured images" refers to technologies that use image recognition technology to identify food ingredients from photographs taken with a camera.

[1467] 4. "Identified food information" refers to information such as the name and quantity of food items identified through image recognition.

[1468] 5. "User's physical information" refers to health-related data such as the user's age, gender, heart rate, and steps taken.

[1469] 6. "Means of providing meal suggestions" refers to a method of generating appropriate meal menus and recipes based on the user's physical information and identified ingredient information.

[1470] 7. A "generative AI model" refers to an artificial intelligence model that generates customized recipes based on the user's health data and ingredient information.

[1471] 8. A "customized recipe" refers to a specific recipe generated based on the user's individual health condition and ingredient availability.

[1472] 9. "Means of identifying missing ingredients" refers to techniques for identifying ingredients that are not present in the refrigerator based on the proposed recipe.

[1473] 10. "Partner food delivery services" refers to service providers that allow customers to order ingredients and food online and have them delivered.

[1474] This invention is a system that suggests customized recipes based on the user's physical information and food information, and indicates where to purchase the necessary ingredients. This system is implemented using the following hardware and software.

[1475] Hardware and software to be used

[1476] 1. Hardware

[1477] Wearable devices: Devices that acquire physical information such as the user's heart rate, steps taken, and calories burned.

[1478] Smartphone: A device used to take pictures of food items inside the refrigerator and upload them to an application.

[1479] 2. Software

[1480] Image recognition technology: This technology identifies food items from images taken with a smartphone. An example is the Python library `face_recognition`.

[1481] Generative AI Model: An artificial intelligence model that generates customized recipes based on the user's health data and ingredient information. OpenAI's GPT-4 API, among others, is used.

[1482] System Operation Description

[1483] User registration and profile settings

[1484] Users install the application and, upon first launch, enter their name, age, gender, height, weight, allergy information, preferred foods, and health goals. This allows the user's individual profile to be saved on the server.

[1485] Integration with wearable devices

[1486] The server periodically receives data such as heart rate, steps taken, and calories burned from wearable devices and stores it in a database in real time. This allows the user's health status to be constantly monitored.

[1487] Making a list of ingredients in the refrigerator

[1488] The user takes a photo of the inside of their refrigerator with their smartphone and uploads it to the application. The server uses image recognition technology to identify the food items in the photo and generates a list of those items. These identified items are added to the user's inventory list and managed as current inventory information.

[1489] Generating customized recipes and menus

[1490] The server uses a generative AI model to create a customized weekly meal plan based on the user's health data, ingredient inventory list, and preferences. The generated meal plan, along with necessary nutrient and calorie information, is presented to the user via their smartphone.

[1491] Specific example:

[1492] If the user's health status does not meet the target health level, a generative AI model will be used to suggest recipes.

[1493] Examples of prompt statements to use:

[1494] User information: {'name': 'User 1', 'age': 30, 'gender': 'male', 'height': 175, 'weight': 70, 'allergies': ['dairy products'], 'preferences': ['vegetables', 'chicken']}

[1495] Health data: {'heart_rate': 70, 'steps': 8000, 'calories_burned': 500}

[1496] Ingredients in the refrigerator: ['Tomato', 'Carrot', 'Chicken']

[1497] Based on this, please propose a customized recipe.

[1498] Suggestions for places to buy groceries

[1499] The server lists the necessary ingredients based on the suggested menu and identifies any missing ingredients. Next, it searches for ingredient purchase options from partner food delivery services and generates links to the nearest purchase locations or online stores. These purchase option lists are presented to the user via their smartphone, along with links to online stores.

[1500] Health data feedback

[1501] The server aggregates data from wearable devices and food records to analyze the user's health status. Based on the analysis, it generates health feedback and advice, and creates notification messages. The generated feedback and advice are delivered to the user via push notifications on their smartphone.

[1502] This allows users to quickly receive personalized meal suggestions based on their health data and ingredient information, enabling them to efficiently purchase ingredients and live a healthy lifestyle.

[1503] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1504] Step 1:

[1505] The user installs the application and sets up a profile upon first launch.

[1506] Input: Name, age, gender, height, weight, allergy information, favorite foods, health goals.

[1507] Data processing: Use this data to generate individual user profiles.

[1508] Output: The user's profile is saved to the server's database.

[1509] Step 2:

[1510] The server periodically receives health data from wearable devices.

[1511] Input: Data such as heart rate, steps taken, and calories burned, transmitted from a wearable device.

[1512] Data processing: Received data is recorded in a database in real time to monitor the user's health status.

[1513] Output: The latest health data will be updated.

[1514] Step 3:

[1515] Users take photos of the food in their refrigerator with their smartphones and upload the images to the app.

[1516] Input: A photo showing the food inside a refrigerator.

[1517] Data processing: The server uses image recognition technology to identify ingredients in the photograph.

[1518] Output: A list of recognized ingredients is generated and saved to the database.

[1519] Step 4:

[1520] The server generates customized recipes based on the user's health data and ingredient inventory list.

[1521] Input: Prompt text for the generated AI model (user profile, health data, list of identified foods).

[1522] Data processing: Generate customized recipes using a generative AI model (e.g., GPT-4).

[1523] Output: The generated recipe and necessary nutrient and calorie information are produced.

[1524] Step 5:

[1525] The server lists the necessary ingredients based on the proposed recipe and identifies any missing ingredients.

[1526] Input: Generated recipe and current ingredient inventory information.

[1527] Data processing: Identify and list any missing ingredients.

[1528] Output: A list of missing ingredients is generated.

[1529] Step 6:

[1530] The server searches for grocery purchase options from partner food delivery services and generates links to the nearest purchase locations and online stores.

[1531] Input: List of missing ingredients.

[1532] Data processing: Search the database of food delivery services to obtain information on potential purchases.

[1533] Output: A list of potential purchases and links to online stores are generated.

[1534] Step 7:

[1535] The device presents the user with a generated recipe, information on necessary nutrients and calories, and a list of recommended purchases.

[1536] Input: Generated recipe, required nutrient and calorie information, a list of suggested purchases, and links to online stores.

[1537] Data processing: Converts information into a format that is easy for users to understand.

[1538] Output: It will be displayed on the smartphone screen.

[1539] Step 8:

[1540] The server aggregates data from wearable devices and food records to analyze the user's health status.

[1541] Input: User's health data, dietary record data.

[1542] Data processing: Use analytical algorithms to evaluate the user's health status.

[1543] Output: Health feedback and advice are generated.

[1544] Step 9:

[1545] The device will send the generated feedback and advice to the user via push notifications.

[1546] Input: Feedback and advice sent from the server.

[1547] Data processing: Convert to a format that can be displayed as a push notification.

[1548] Output: It will be displayed as a push notification on your smartphone.

[1549] Through the above processing steps, users can quickly obtain customized meal suggestions based on their health data and food information, enabling them to efficiently purchase ingredients and lead a healthy lifestyle.

[1550] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1551] This invention relates to a system that acquires a user's physical information and emotional state, and presents meal suggestions and locations for purchasing ingredients. The following describes a specific form for realizing this system.

[1552] User registration and profile settings

[1553] 1. User: Install the application and, upon first launch, enter your name, age, gender, height, weight, and information about allergies, preferred foods, and health goals. This information will be used to create your individual profile.

[1554] 2. Server: Saves the entered information to the database and creates a profile for each user.

[1555] Integration with wearable devices

[1556] 1. User: Link the wearable device and application in the settings screen.

[1557] 2. Terminal: Regularly receives data such as heart rate, steps taken, and calories burned from a wearable device.

[1558] 3. Server: Receives device data sent from terminals and stores it in the database.

[1559] Making a list of ingredients in the refrigerator

[1560] 1. User: Take a photo of the inside of the refrigerator with your smartphone and upload it to the application.

[1561] 2. Terminal: Uses image recognition technology to identify ingredients in a photograph and creates an ingredient list.

[1562] 3. Server: Stores the ingredient list in a database and manages it as the user's current inventory information.

[1563] Specific example

[1564] User: Upload a photo of the refrigerator to the application.

[1565] Terminal: Analyzes the photo and identifies "tomato, carrot, and chicken breast."

[1566] Server: Identify these ingredients and add them to the user's inventory list.

[1567] Generating customized recipes and menus

[1568] 1. Server: Uses generative AI to create a customized weekly meal plan based on the user's health data, ingredient inventory list, preferences, and allergy information.

[1569] 2. Terminal: Presents the generated menu and necessary nutrient and calorie information to the user.

[1570] Emotional recognition and adaptation of meal suggestions using an emotional engine

[1571] 1. User: Inputs facial expressions and voice through the application. Also, notifies emotional states by inputting text.

[1572] 2. Device: The emotion engine analyzes the user's facial expressions, voice, or text data to recognize their current emotional state.

[1573] 3. Server: Analyzes recognized emotional information in conjunction with health data and adapts meal suggestions based on the user's emotions.

[1574] Specific example

[1575] User: Type the text "I'm tired today" into the chatbot within the application.

[1576] Terminal: The emotion engine analyzes this text data and recognizes that the user is tired.

[1577] Server: Suggest highly nutritious ingredients and easy-to-prepare dishes to reduce user fatigue.

[1578] Suggestions for places to buy groceries

[1579] 1. Server: Identify any missing ingredients based on the proposed menu.

[1580] 2. Server: For any missing ingredients, it searches for potential purchase options from partner e-commerce platforms and gathers information on the nearest places to buy them and online stores.

[1581] 3. Terminal: Presents the generated list of potential purchases to the user and provides a link to the online store.

[1582] Feedback on health data and emotional data

[1583] 1. Server: Collects data from wearable devices, meal records, and emotional data to analyze the user's health status.

[1584] 2. Server: Generates health feedback and advice, emotion-based recommendations, and notification messages based on the analysis results.

[1585] 3. Device: Send feedback and advice to users via push notifications.

[1586] Specific example

[1587] Server: Verify that the user's activity level has not reached 70% of the target.

[1588] The device notifies the user that "You can reach your goal by walking 1000 more steps." If it detects that the user's stress level is high, it suggests, "Try some relaxing herbal tea."

[1589] The above describes the embodiments of the present invention. In this form, users can receive not only individual health data and food information, but also optimal meal suggestions tailored to their mood at any given time, enabling them to efficiently purchase food ingredients and lead a healthy and balanced life.

[1590] The following describes the processing flow.

[1591] User registration and profile settings

[1592] Step 1:

[1593] User: Install the application and, upon first launch, enter your name, age, gender, height, weight, and information about allergies, preferred foods, and health goals.

[1594] Step 2:

[1595] Terminal: Sends the entered information to the server.

[1596] Step 3:

[1597] Server: Stores received information in a database and creates user profiles.

[1598] Integration with wearable devices

[1599] Step 4:

[1600] User: Link the wearable device and application in the settings screen.

[1601] Step 5:

[1602] Device: Once the connection is successfully completed, enable the setting to periodically receive data such as heart rate, steps, and calories burned from the wearable device.

[1603] Step 6:

[1604] Server: Receives device data sent from terminals and stores it in the database.

[1605] Making a list of ingredients in the refrigerator

[1606] Step 7:

[1607] User: Take a photo of the inside of the refrigerator with your smartphone and upload it to the application.

[1608] Step 8:

[1609] Device: Uses AI image recognition technology to identify ingredients in a photograph and creates an ingredient list.

[1610] Step 9:

[1611] Terminal: Sends the generated ingredient list to the server.

[1612] Step 10:

[1613] Server: Stores the received ingredient list in the database and manages it as the user's current inventory information.

[1614] Generating customized recipes and menus

[1615] Step 11:

[1616] Server: Uses generative AI to create a customized weekly meal plan based on the user's health data, ingredient inventory list, preferences, and allergy information.

[1617] Step 12:

[1618] Server: Sends the generated menu to the terminal.

[1619] Step 13:

[1620] Terminal: Displays the received menu and necessary nutrient and calorie information to the user.

[1621] Emotional recognition and adaptation of meal suggestions using an emotional engine

[1622] Step 14:

[1623] User: Notifies emotional states by inputting facial expressions, voice, or text through the application.

[1624] Step 15:

[1625] Device: The emotion engine analyzes the user's facial expressions, voice, or text data to recognize their current emotional state.

[1626] Step 16:

[1627] Server: Analyzes recognized emotional information in conjunction with health data and adapts meal suggestions based on the user's emotions.

[1628] Suggestions for places to buy groceries

[1629] Step 17:

[1630] Server: Identify any missing ingredients based on the proposed menu.

[1631] Step 18:

[1632] Server: Searches for potential purchase options from partner e-commerce platforms for missing ingredients, gathering information on the nearest places to buy them and online stores.

[1633] Step 19:

[1634] Server: Sends a list of potential purchases to the terminal.

[1635] Step 20:

[1636] Terminal: Presents a generated list of potential purchases to the user and provides a link to the online store.

[1637] Feedback on health data and emotional data

[1638] Step 21:

[1639] Server: Collects data from wearable devices, meal records, and emotional data to analyze the user's health status.

[1640] Step 22:

[1641] Server: Generates health feedback and advice, emotion-based recommendations, and notification messages based on analysis results.

[1642] Step 23:

[1643] Server: Sends the generated feedback and advice to the terminal.

[1644] Step 24:

[1645] Device: Send feedback and advice to users via push notifications.

[1646] The above describes the specific actions taken at each processing step. This detailed processing flow allows users to receive optimal meal suggestions based on the latest health data and ingredient information, efficiently purchase ingredients, and lead a healthy and balanced life. In addition, it provides even more personalized support by offering suggestions that take into account the user's emotional state.

[1647] (Example 2)

[1648] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1649] Conventional meal suggestion systems based solely on the user's physical data and ingredient information, failing to consider the user's emotional state. Furthermore, suggestions for ingredient sourcing locations were limited, failing to integrate information from online stores and physical retail outlets. Therefore, there was a need to provide users with optimal and efficient meal suggestions and ingredient purchasing methods.

[1650] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1651] In this invention, the server includes means for communicating with equipment for acquiring physical data, means for identifying food ingredients from captured content, means for suggesting meals based on the identified food ingredient data, the user's physical data, and emotional state, and means for identifying missing ingredients based on the suggested meal and indicating where to obtain them. This enables meal suggestions that take into account the user's emotional state in addition to their physical data and food ingredient information, and further enables the provision of food ingredient acquisition locations by integrating information from e-commerce platforms and physical stores, thereby providing optimal and efficient meal suggestions and means for purchasing ingredients.

[1652] "Physical data" refers to information about the user's body, including data such as height, weight, heart rate, steps taken, and calories burned.

[1653] "Equipment" refers to wearable devices, sensor devices, and other devices used to acquire the user's physical data.

[1654] "Content" refers to images and photographs taken by users, including, in particular, food items in a refrigerator and cooked meals.

[1655] "Food ingredient data" refers to information such as the type, quantity, and availability of food items identified from the content.

[1656] "Emotional state" refers to information that indicates the user's current psychological and emotional state, including, for example, fatigue, stress, and satisfaction.

[1657] "Meal suggestions" refers to the act of suggesting optimal meal menus and recipes based on the user's physical data and emotional state.

[1658] "Place of acquisition" refers to the location where the ingredients needed for the proposed meal can be purchased, including online stores and physical stores.

[1659] An "e-commerce platform" refers to a website or application that facilitates the buying and selling of goods online, and examples include e-commerce sites.

[1660] A "physical store" refers to an actual store or supermarket where users can visit in person to purchase groceries.

[1661] This invention relates to a system that acquires a user's physical data and emotional state, and presents personalized meal suggestions and locations for obtaining ingredients. The following describes specific embodiments of this system.

[1662] Overall structure

[1663] This system consists of a device (wearable device) for acquiring physical data, the user's smartphone (terminal), and a server connected to them. The server receives and analyzes the data and provides meal suggestions and locations for data acquisition.

[1664] User registration and profile settings

[1665] Users first install the application on their smartphone and, upon initial launch, enter their name, age, gender, height, weight, allergy information, preferred foods, and health goals. This creates an individual profile. The server stores the information entered by the user in a database and builds the profile. The application interface is designed to allow users to easily enter the necessary information.

[1666] Integration with wearable devices

[1667] The user pairs a wearable device (e.g., a wristwatch-type heart rate monitor) with the application via the smartphone's settings screen. The wearable device periodically sends data such as heart rate, steps taken, and calories burned to the device. The device periodically sends this data to a server. The server stores the received data in a database and manages the user's health data.

[1668] Making a list of ingredients in the refrigerator

[1669] The user takes a photo of the inside of their refrigerator with their smartphone and uploads it to the application. The device uses image recognition technology (e.g., Amazon Rekognition) to identify the food items in the photo and create a list of those items. The server stores the identified food list in a database and manages it as the user's current inventory information.

[1670] Specific example

[1671] When a user uploads a photo of their refrigerator to the application, the device analyzes the image and identifies "tomatoes, carrots, and chicken breast." The server saves these ingredients to a database and updates the inventory list.

[1672] Generating customized recipes and menus

[1673] The server uses generative AI (e.g., OpenAI's GPT-4) to create a customized weekly meal plan based on the user's health data, ingredient inventory list, preferences, and allergy information. The terminal then presents the generated meal plan along with necessary nutrient and calorie information to the user.

[1674] Examples of prompt statements

[1675] "I'm a woman in my 30s, aiming to lose weight. I have tomatoes, carrots, and chicken breast in my refrigerator. Please create a low-calorie, nutritionally balanced meal plan for one week."

[1676] Emotional recognition and adaptation of meal suggestions using an emotional engine

[1677] Users notify their emotional state through the application by inputting facial expressions, voice, or text. The emotion engine analyzes this input data to recognize the user's current emotional state. The server then comprehensively analyzes the recognized emotional state and physical data to provide meal suggestions based on the user's emotions.

[1678] Specific example

[1679] When a user types "I'm tired today" into the in-app chatbot, the device's emotion engine analyzes this text data and recognizes that the user is tired. The server then suggests highly nutritious ingredients and easy-to-prepare dishes.

[1680] Suggestions for locations to obtain ingredients

[1681] The server identifies any missing ingredients based on the suggested menu and searches for purchase options from partner e-commerce platforms and physical stores. The terminal presents the generated list of purchase options to the user and provides links to online stores.

[1682] Feedback on health data and emotional data

[1683] The server aggregates data from wearable devices, meal logs, and emotional data to analyze the user's health status. Based on the analysis, it generates feedback and advice and notifies the user. The device delivers this feedback and advice to the user via push notifications. For example, it provides specific advice such as, "You can achieve your goal by walking another 1000 steps," or emotionally-based recommendations such as, "Try a relaxing herbal tea."

[1684] The above describes a specific embodiment for carrying out the present invention. This allows users to receive not only individual health data and food information, but also optimal meal suggestions tailored to their mood at any given time, enabling them to efficiently purchase food ingredients and lead a healthy and balanced life.

[1685] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1686] Program execution flow and detailed explanation of each processing step

[1687] User registration and profile settings

[1688] Step 1:

[1689] The user installs the application and launches it for the first time.

[1690] Input: None

[1691] Output: Display of the app's welcome screen

[1692] Specific operation: The user downloads and launches the app. The interface is designed to be intuitive and easy to understand.

[1693] Step 2:

[1694] The user enters their name, age, gender, height, weight, allergy information, favorite foods, and health goals.

[1695] Input: User information (name, age, gender, height, weight, allergy information, favorite foods, health goals)

[1696] Output: User profile data

[1697] Specific action: The user enters the required information into the input form and presses the submit button.

[1698] Step 3:

[1699] The server saves the entered information to the database and builds a user profile.

[1700] Input: User profile data

[1701] Output: Saved to database successfully

[1702] Specific operation: The server receives profile data and stores it in the database. A relational database such as MySQL is used for this database.

[1703] Integration with wearable devices

[1704] Step 1:

[1705] The user pairs the wearable device and the application in the app settings screen.

[1706] Input: Pairing request

[1707] Output: Pairing complete notification

[1708] Specific steps: The user selects the wearable device on their smartphone's Bluetooth settings screen and pairs it.

[1709] Step 2:

[1710] The device periodically receives data such as heart rate, steps taken, and calories burned from the wearable device.

[1711] Input: Device data (heart rate, steps, calories burned)

[1712] Output: Automatic saving of received data at regular intervals.

[1713] Specific operation: The device periodically (e.g., every hour) retrieves data from the wearable device and saves it locally.

[1714] Step 3:

[1715] The server receives device data sent from the terminal and stores it in the database.

[1716] Input: Device data from the terminal

[1717] Output: Saved to database successfully

[1718] Specific operation: At regular intervals (e.g., every 24 hours), the server receives data from the terminal, analyzes it, and saves it to the database.

[1719] Making a list of ingredients in the refrigerator

[1720] Step 1:

[1721] The user takes a photo of the inside of the refrigerator with their smartphone and uploads it to the application.

[1722] Input: Photo of the inside of the refrigerator

[1723] Output: Upload complete notification

[1724] Specific action: The user uses their smartphone camera to take a picture of the inside of the refrigerator and uploads the image to the app.

[1725] Step 2:

[1726] The device uses image recognition technology to identify food items in the photograph.

[1727] Input: Uploaded photo

[1728] Output: Identified ingredient list

[1729] Specific operation: The device uses image recognition APIs such as Amazon Rekognition to automatically analyze and identify food items in a photograph.

[1730] Step 3:

[1731] The server stores a list of identified ingredients in a database and manages it as the user's current inventory information.

[1732] Input: Identified ingredient list

[1733] Output: Saved to database successfully

[1734] Specific operation: The server receives the identified ingredient information and stores it in the database.

[1735] Generating customized recipes and menus

[1736] Step 1:

[1737] The server uses generative AI to create a customized weekly meal plan based on the user's health data, ingredient inventory list, preferences, and allergy information.

[1738] Input: Health data, food inventory list, preferences and allergy information

[1739] Output: Customized menu

[1740] Specific operation: The server sends prompt messages to the AI ​​model (e.g., OpenAI's GPT-4) and receives the generated menu.

[1741] Step 2:

[1742] The device displays the generated menu and information on necessary nutrients and calories to the user.

[1743] Input: Customized menu

[1744] Output: Presentation of menu information

[1745] Specific operation: The terminal displays the menu information received from the server on its screen, along with information on necessary nutrients and calories.

[1746] Emotional recognition and adaptation of meal suggestions using an emotional engine

[1747] Step 1:

[1748] Users can input facial expressions and voices through the application, or input text to notify their emotional state.

[1749] Input: Facial expressions, voice, or text data

[1750] Output: Notification of emotional state

[1751] Specific operation: Users input their emotional state through the app using their facial photo, voice, or text data.

[1752] Step 2:

[1753] The device uses an emotion engine to analyze facial expressions, voice, or text to recognize the user's current emotional state.

[1754] Input: Emotional data (facial expressions, voice, text)

[1755] Output: Recognized emotional state

[1756] Specific operation: The device uses an emotion recognition engine (e.g., Microsoft Azure Cognitive Services) to analyze input data and recognize the emotional state.

[1757] Step 3:

[1758] The server integrates and analyzes recognized emotional states and physical data to provide meal suggestions based on the user's emotions.

[1759] Input: Recognized emotional state, physical data

[1760] Output: Meal suggestions based on emotional state

[1761] Specific operation: Based on the analysis results, the server generates meal suggestions that include highly nutritious ingredients and easy-to-prepare dishes.

[1762] Suggestions for locations to obtain ingredients

[1763] Step 1:

[1764] The server identifies any missing ingredients based on the suggested menu.

[1765] Input: Suggested menu, ingredient inventory list

[1766] Output: List of missing ingredients

[1767] Specific operation: The server compares the menu with the inventory list to identify any missing ingredients.

[1768] Step 2:

[1769] The server searches for potential purchase options for ingredients that are currently out of stock, from partner e-commerce platforms and physical stores.

[1770] Input: List of missing ingredients

[1771] Output: Purchase candidate list

[1772] Specific operation: The server uses APIs from e-commerce platforms and physical stores to search for purchasing options for missing ingredients.

[1773] Step 3:

[1774] The device displays a list of potential purchases to the user and provides a link to the online store.

[1775] Input: Purchase candidate list

[1776] Output: Display of purchase candidate list

[1777] Specific operation: The device displays a list of potential purchases on the screen and provides the user with a link to each option.

[1778] Feedback on health data and emotional data

[1779] Step 1:

[1780] The server collects data from wearable devices, meal logs, and emotional data to analyze the user's health status.

[1781] Input: Wearable device data, food log data, emotional data

[1782] Output: Analysis results of health status

[1783] Specific operation: The server uses Python or similar languages ​​to execute analysis scripts and collect and analyze user health data.

[1784] Step 2:

[1785] The server generates feedback and advice based on the analysis results and notifies the user.

[1786] Input: Analysis results of health status

[1787] Output: Feedback and advice

[1788] Specific operation: The server generates appropriate advice and feedback for the user based on the analysis results and creates notification messages.

[1789] Step 3:

[1790] The device will send feedback and advice to the user via push notifications.

[1791] Input: Feedback and advice

[1792] Output: Notification message

[1793] Specific actions: The device uses push notifications to inform the user of specific advice, such as "You can reach your goal by walking 1000 more steps." It also provides emotion-based recommendations, such as "Try a relaxing herbal tea."

[1794] (Application Example 2)

[1795] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1796] In modern society, maintaining good health requires individuals to plan appropriate meals and efficiently purchase the necessary ingredients. However, it takes considerable time and effort for users to choose appropriate meals, identify necessary ingredients, and purchase them themselves. Furthermore, while a user's emotional state often significantly influences their meal choices, there are limited systems that can appropriately reflect this in their meal suggestions. In addition, although the use of online stores and food delivery services has increased, there is a lack of systems that centrally manage these services and provide users with the most suitable purchasing options. A system that solves these problems is needed.

[1797] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[1798] In this invention, the server includes means for communicating with a device for acquiring physical information, means for identifying ingredients from captured images, means for suggesting meals based on the identified ingredient information, the user's physical information, and emotional state, and means for identifying necessary ingredients based on the suggested meal, indicating where to purchase them, and providing links to food delivery services. This allows the user to receive optimal meal suggestions based on their physical information and emotional state, and to purchase necessary ingredients quickly and efficiently. Furthermore, they can quickly obtain ingredients and prepared meals through food delivery, supporting their eating habits.

[1799] "Physical information" refers to data that quantifies the user's physical condition, such as heart rate, steps taken, and calories burned.

[1800] "Emotional state" refers to information that indicates the user's current psychological state or mood, analyzed from the user's facial expressions, voice, or text data.

[1801] "Meal suggestions" is a process that generates and presents meal menus suitable for the user based on the user's physical information and emotional state.

[1802] "Food identification" is the process of using AI technology to identify food items from images taken by the user and analyzing their types.

[1803] "Presenting a place to purchase" refers to the process of showing users information about online stores or physical shops where they can purchase the ingredients they are missing.

[1804] A "food delivery link" is an online link that allows users to order the necessary ingredients and dishes based on a suggested meal menu.

[1805] This invention relates to a system that acquires a user's physical information and emotional state, and presents meal suggestions and locations for purchasing ingredients. An embodiment of this system is described in detail below.

[1806] User registration and profile settings

[1807] Users first install the application on their smartphone and, upon initial launch, enter their name, age, gender, height, weight, allergy information, preferred foods, and health goals. The server stores this information in a database and creates a profile for each user.

[1808] Integration with wearable devices

[1809] The user connects the wearable device and the application through the application's settings screen. The terminal periodically receives data such as heart rate, steps taken, and calories burned from the wearable device. The server stores the received device data in a database.

[1810] Making a list of ingredients in the refrigerator

[1811] The user takes a photo of the inside of their refrigerator with their smartphone and uploads it to the application. The device uses image recognition technology to identify the food items in the photo and creates a list of those items. The server stores the list of items in a database and manages it as the user's current inventory information.

[1812] Generating customized recipes and menus

[1813] The server uses an AI model to generate a customized weekly meal plan based on the user's health data, ingredient inventory list, preferences, and allergy information. The terminal then displays the generated meal plan along with information on necessary nutrients and calories to the user.

[1814] Emotional recognition and adaptation of meal suggestions using an emotional engine

[1815] Users notify their emotional state by inputting facial expressions, voice, or text through the application. The device uses an emotion engine to analyze the user's facial expressions, voice, or text data to recognize their current emotional state. The server analyzes the recognized emotional information in conjunction with health data and adapts meal suggestions based on the user's emotions.

[1816] Suggestions for places to buy groceries

[1817] The server identifies any missing ingredients based on the suggested menu. It also searches for purchase options from partner e-commerce platforms and the nearest supermarket, and provides links to online stores for those missing ingredients.

[1818] Feedback on health data and emotional data

[1819] The server aggregates data from wearable devices, meal records, and emotional data to analyze the user's health status. Based on the analysis, it generates health feedback and advice, emotional recommendations, and creates notification messages. The device then delivers the feedback and advice to the user via push notifications.

[1820] Hardware and software used

[1821] Hardware: Smartphones, wearable devices, refrigerators with cameras

[1822] Software: TensorFlow, OpenCV, Flask, Twilio, Generative AI Models

[1823] Specific example

[1824] The user uploads a photo of their refrigerator to the application, and image analysis identifies "tomatoes, carrots, and chicken breast." If the user enters the text "I'm tired today," the emotion engine analyzes this and recognizes that the user is tired. Based on this information, the server suggests highly nutritious ingredients and easy-to-prepare dishes. For example, the generative AI model is prompted with the message, "The user is currently feeling tired; please suggest an optimal dinner," and then generates suggestions.

[1825] In this way, this system allows users to receive optimal meal suggestions tailored to their physical information and emotional state, and to quickly and efficiently purchase the necessary ingredients. Furthermore, because meals can be easily obtained through food delivery, the system supports users in leading a healthy lifestyle.

[1826] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1827] Step 1: User registration and profile setup

[1828] When a user first launches an application installed on their smartphone, they enter their name, age, gender, height, weight, allergy information, preferred foods, and health goals. This information is the input data, which the server stores in a database to create a profile for each user. The information stored in the database is the output. This initiates individual data management for each user.

[1829] Step 2: Integration with wearable devices

[1830] The user connects the wearable device to the application through the application's settings screen. The terminal periodically receives data such as heart rate, steps taken, and calories burned from the wearable device. The input is biometric data transmitted from the wearable device, which the server receives and stores in a database. The stored biometric data becomes the output. This allows the user's physical information to be obtained in real time.

[1831] Step 3: Make a list of the ingredients in your refrigerator.

[1832] The user takes a photo of the inside of their refrigerator with their smartphone and uploads it to the application. The application receives the image and uses OpenCV and TensorFlow to identify the food items in the photo using image recognition technology. The output is a list of identified food items, which the server saves to a database. This process manages the current food inventory information.

[1833] Step 4: Generating customized recipes and menus

[1834] The server retrieves the user's health data, ingredient inventory list, preferences, and allergy information, and generates a week's worth of customized meal plans by inputting prompts into a generative AI model. For example, by inputting the prompt, "The user is currently feeling tired; please suggest an optimal dinner," the generative AI model will suggest a meal plan. The generated meal plan is the output, and the terminal presents it to the user. The meal suggestions are displayed with specific nutritional and calorie information.

[1835] Step 5: Emotional recognition and adaptation of meal suggestions by the emotional engine

[1836] Users notify the system of their emotional state by inputting facial expressions, voice, or text through the application. The terminal uses an emotion engine to analyze the user's current emotional state from the facial expressions, voice, and text data. This analysis result becomes the input data, which the server acquires and analyzes together with the user's health data. Based on the analysis result, the system provides meal suggestions appropriate to the user's emotional state, and these suggestions become the output.

[1837] Step 6: Suggesting places to buy groceries

[1838] The server identifies missing ingredients based on the suggested menu. These missing ingredients are the input data, and the server searches for purchase options from e-commerce platforms and the nearest supermarket. The output is a list of purchase options, which the terminal presents to the user, providing links to online stores. This allows the user to efficiently purchase the missing ingredients.

[1839] Step 7: Feedback on health and emotional data

[1840] The server collects data from wearable devices, meal records, and emotional data to analyze the user's health status. The input is the various collected data, and the server generates feedback and advice based on its analysis. This is the output, and the device communicates it to the user via push notifications, etc. For example, messages such as "You can achieve your goal by walking 1000 more steps" or emotional recommendations such as "Try a relaxing herbal tea" might be displayed.

[1841] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1842] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1843] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[1844] [Fourth Embodiment]

[1845] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[1846] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1847] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1848] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[1849] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1850] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1851] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1852] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[1853] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1854] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1855] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1856] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1857] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1858] This invention relates to a system that acquires a user's physical information, identifies food ingredients from captured images, makes meal suggestions based on this information, and indicates where to purchase the necessary ingredients. The following describes a specific form for realizing this system.

[1859] User registration and profile settings

[1860] 1. User: Install the application and, upon first launch, enter your name, age, gender, height, weight, and information about allergies, preferred foods, and health goals. This information will be used to create your individual profile.

[1861] 2. Server: Saves the entered information to the database and creates a profile for each user.

[1862] Integration with wearable devices

[1863] 1. User: Link the wearable device and application in the settings screen.

[1864] 2. Terminal: Regularly receives data such as heart rate, steps taken, and calories burned from wearable devices.

[1865] 3. Server: Saves received data to the database in real time and monitors the user's health status.

[1866] Making a list of ingredients in the refrigerator

[1867] 1. User: Take a photo of the inside of the refrigerator with your smartphone and upload it to the application.

[1868] 2. Terminal: Uses image recognition technology to identify ingredients in a photograph and generates an ingredient list.

[1869] 3. Server: Stores the ingredient list in a database and manages it as the user's current ingredient inventory information.

[1870] Specific example

[1871] User: Upload a photo of the refrigerator to the application.

[1872] Terminal: Analyzes the photo and identifies "tomato, carrot, and chicken breast."

[1873] Server: Adds these identified ingredients to the user's inventory list and saves them as current inventory information.

[1874] Generating customized recipes and menus

[1875] 1. Server: Uses generative AI to create a customized weekly meal plan based on the user's health data, ingredient inventory list, and preferences.

[1876] 2. Terminal: Presents the generated menu and necessary nutrient and calorie information to the user.

[1877] Suggestions for places to buy groceries

[1878] 1. Server: Based on the proposed menu, list the necessary ingredients and identify any missing ingredients.

[1879] 2. Server: Searches for potential grocery purchases from partner e-commerce platforms and generates links to the nearest stores or online shops.

[1880] 3. Terminal: Presents the generated list of potential purchases to the user and provides a link to the online store.

[1881] Specific example

[1882] Server: The server generates a weekly menu using tomatoes, carrots, and chicken breast, and suggests recipes such as "Tomato Chicken Stew."

[1883] Terminal: Displays these recipes, a list of required ingredients, and cooking instructions to the user.

[1884] Health data feedback

[1885] 1. Server: Collects data from wearable devices and food records to analyze the user's health status.

[1886] 2. Server: Generates health feedback and advice based on analysis results, and creates notification messages.

[1887] 3. Device: Send the generated feedback and advice to the user via push notification.

[1888] Specific example

[1889] Server: Verify that the user's activity level has not reached 70% of the target.

[1890] Device: Notifies the user with the message, "Walk another 1000 steps to reach your goal."

[1891] The above describes the embodiments of the present invention. This system allows users to receive customized meal suggestions based on their individual health data and food information, enabling them to efficiently purchase food ingredients and lead a healthy lifestyle.

[1892] The following describes the processing flow.

[1893] User registration and profile settings

[1894] Step 1:

[1895] User: Install the application and, upon first launch, enter your name, age, gender, height, weight, and information about allergies, preferred foods, and health goals.

[1896] Step 2:

[1897] Terminal: Sends the entered information to the server.

[1898] Step 3:

[1899] Server: Stores received information in a database and creates user profiles.

[1900] Integration with wearable devices

[1901] Step 4:

[1902] User: In the application's settings screen, link the wearable device with the application.

[1903] Step 5:

[1904] Device: Once the connection is successfully completed, enable the setting to periodically receive data such as heart rate, steps, and calories burned from the wearable device.

[1905] Step 6:

[1906] Server: Receives device data sent from terminals and stores it in the database.

[1907] Making a list of ingredients in the refrigerator

[1908] Step 7:

[1909] User: Take a photo of the inside of the refrigerator with your smartphone and upload it to the application.

[1910] Step 8:

[1911] Device: Uses AI image recognition technology to identify ingredients in a photograph and creates an ingredient list.

[1912] Step 9:

[1913] Terminal: Sends the generated ingredient list to the server.

[1914] Step 10:

[1915] Server: Stores the received ingredient list in the database and manages it as the user's current inventory information.

[1916] Generating customized recipes and menus

[1917] Step 11:

[1918] Server: Uses generative AI to create a customized weekly meal plan based on the user's health data, ingredient inventory list, preferences, and allergy information.

[1919] Step 12:

[1920] Server: Sends the generated menu to the terminal.

[1921] Step 13:

[1922] Terminal: Displays the received menu and necessary nutrient and calorie information to the user.

[1923] Suggestions for places to buy groceries

[1924] Step 14:

[1925] Server: Identify any missing ingredients based on the proposed menu.

[1926] Step 15:

[1927] Server: Searches for potential purchase options from partner e-commerce platforms for missing ingredients, gathering information on the nearest places to buy them and online stores.

[1928] Step 16:

[1929] Server: Sends a list of potential purchases to the terminal.

[1930] Step 17:

[1931] Terminal: Presents a generated list of potential purchases to the user and provides a link to the online store.

[1932] Health data feedback

[1933] Step 18:

[1934] Server: Collects data from wearable devices and food records to analyze the user's health status.

[1935] Step 19:

[1936] Server: Generates health feedback and advice based on analysis results, and creates notification messages.

[1937] Step 20:

[1938] Server: Sends the generated feedback and advice to the terminal.

[1939] Step 21:

[1940] Device: Send feedback and advice to users via push notifications.

[1941] The above describes the detailed operation at each processing step. Through this detailed processing flow, users can always receive optimal meal suggestions based on the latest health data and ingredient information, efficiently purchase ingredients, and lead a healthy lifestyle.

[1942] (Example 1)

[1943] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1944] Maintaining a healthy diet is difficult in today's busy lifestyle. Furthermore, it's challenging for users to select appropriate foods and recipes based on their own health status and preferences. Even efficiently shopping for ingredients by identifying suitable locations is time-consuming. To address these issues, a system is needed that centrally manages users' health data and ingredient information, and provides optimal meal suggestions and ingredient purchasing locations based on this data.

[1945] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1946] In this invention, the server includes means for inputting the user's personal information and creating a profile; means for working with a wearable device to collect biometric data; means for analyzing images of the inside of a refrigerator taken and identifying ingredients; means for creating a customized menu using a generative AI model based on the identified ingredient information and the user's health data; and means for identifying missing ingredients based on the proposed menu and suggesting where to purchase them. As a result, the user can receive meal suggestions based on their health status and ingredient information, efficiently purchase necessary ingredients, and maintain a healthy lifestyle.

[1947] A "user" is an entity that uses the system to manage individual health data and food information, and to make meal suggestions and purchase food ingredients.

[1948] A "profile" is a set of individual information that includes the user's name, age, gender, height, weight, allergy information, preferred foods, and health goals.

[1949] A "wearable device" is an electronic device worn by a user to collect biometric data such as heart rate, steps taken, and calories burned.

[1950] "Biometric data" refers to data such as a user's heart rate, steps taken, and calories burned, collected through wearable devices and other means.

[1951] "Image analysis" is the process of extracting specific information from captured images, and is particularly used to identify food items inside a refrigerator.

[1952] "Food identification" is a method of identifying food items in a refrigerator through image analysis and listing that information.

[1953] A "generative AI model" is an artificial intelligence model that automatically generates customized menus based on the user's health data and ingredient information.

[1954] A "menu" refers to a meal plan or recipe suggested based on the user's health data and available ingredients.

[1955] "Place of purchase" refers to any location where you can purchase any missing ingredients based on the suggested menu, including online stores and physical stores.

[1956] "Health data" is a general term for a user's physical information, biometric data, and dietary history, and is used to manage their health status.

[1957] "Feedback" refers to improvement suggestions and advice provided based on the user's health status and activity level.

[1958] This invention relates to a system that acquires a user's physical information, identifies food ingredients from captured images, makes meal suggestions based on this information, and indicates where to purchase the necessary ingredients. Specific embodiments of this invention are described below.

[1959] User registration and profile settings

[1960] 1. User: First, the user installs the system's application on their smartphone. Upon first launching the application, they enter their name, age, gender, height, weight, allergy information, preferred foods, and health goals. This creates the user's individual profile.

[1961] 2. Server: The server stores the information entered by the user in a database and creates a profile for each user. A database such as MongoDB is used for this purpose.

[1962] Integration with wearable devices

[1963] 1. User: The user connects the wearable device and the application using Bluetooth in the application's settings screen.

[1964] 2. Terminal: The terminal periodically receives biometric data such as heart rate, steps taken, and calories burned from the wearable device. Bluetooth communication is used for this purpose.

[1965] 3. Server: The server stores incoming data in a database in real time and monitors the user's health status. A database such as PostgreSQL is used for this purpose.

[1966] Making a list of ingredients in the refrigerator

[1967] 1. User: The user takes a photo of the inside of the refrigerator with their smartphone and uploads it to the application.

[1968] 2. Terminal: The terminal uses image recognition technology (e.g., TensorFlow) to identify ingredients in the captured image and generate a list of ingredients.

[1969] 3. Server: The server stores the ingredient list in a database and manages it as the user's current ingredient inventory information. A database such as MySQL is used for this purpose.

[1970] Specific example

[1971] When a user uploads a photo of their refrigerator to the application, the device analyzes the photo and identifies "tomatoes, carrots, and chicken breast." The server adds these identified ingredients to the user's inventory list and saves it as current inventory information.

[1972] Generating customized recipes and menus

[1973] 1. Server: The server uses a generative AI model (e.g., OpenAI's GPT) to create a customized weekly meal plan based on the user's health data, ingredient inventory list, and preferences.

[1974] 2. Terminal: The terminal presents the user with the generated menu and information on necessary nutrients and calories.

[1975] Example of a prompt

[1976] Send the following prompt to the generating AI: "Create a one-week meal plan based on User A's health data and inventory information."

[1977] Suggestions for places to buy groceries

[1978] 1. Server: The server identifies any missing ingredients based on the proposed menu and suggests where to purchase them.

[1979] 2. Server: The server searches for potential grocery purchases from partner e-commerce platforms and generates links to the nearest purchase locations and online stores.

[1980] 3. Terminal: The terminal presents the user with a generated list of potential purchases and provides links to online stores.

[1981] Specific example

[1982] The server generates a week's worth of meal plans based on the user's health data and food inventory, suggesting recipes such as "Tomato Chicken Stew." The terminal displays these recipes, a list of required ingredients, and cooking instructions to the user.

[1983] Health data feedback

[1984] 1. Server: The server aggregates data from wearable devices and food records, and analyzes the user's health status.

[1985] 2. Server: The server generates health feedback and advice based on the analysis results and creates notification messages.

[1986] 3. Device: The device will send the generated feedback and advice to the user via push notifications.

[1987] Specific example

[1988] The server confirms that the user's activity level has not reached 70% of the target, and the device notifies the user, "Walk another 1000 steps to reach your goal."

[1989] The above describes the embodiments of the present invention. This system allows users to receive customized meal suggestions based on their health condition and food information, enabling them to efficiently purchase food ingredients and lead a healthy lifestyle.

[1990] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1991] Step 1:

[1992] User: Install the application on your smartphone and enter your personal information (name, age, gender, height, weight, allergy information, favorite foods, health goals) upon first launch.

[1993] Input: User's personal information

[1994] Output: Individual user profiles

[1995] Specific action: The user enters information into each field and presses the "Save" button.

[1996] Step 2:

[1997] Server: Stores the entered personal information in the database and creates a profile for each user.

[1998] Input: User's personal information

[1999] Output: Saved user profile

[2000] Specific operation: The server uses MongoDB to store personal information received from users in JSON format.

[2001] Step 3:

[2002] User: Link the wearable device and application in the settings screen.

[2003] Input: Connection information for wearable devices

[2004] Output: Connection established with wearable device

[2005] Specific steps: The user opens the Bluetooth settings, selects the wearable device, and presses the pairing button.

[2006] Step 4:

[2007] Terminal: Regularly receives data such as heart rate, steps taken, and calories burned from wearable devices.

[2008] Input: Data from a wearable device

[2009] Output: Biometric data temporarily stored on the device

[2010] Specific operation: The device uses Bluetooth communication to retrieve data and save it to local storage.

[2011] Step 5:

[2012] Server: Receives data from wearable devices via terminals and saves it to a database in real time.

[2013] Input: Biometric data transmitted from the device

[2014] Output: Biometric data stored in the database

[2015] Specific operation: The server uses PostgreSQL to execute INSERT statements into the database.

[2016] Step 6:

[2017] User: Take a photo of the inside of the refrigerator with your smartphone and upload it to the application.

[2018] Input: Image of the inside of a refrigerator

[2019] Output: Image files uploaded to the application

[2020] Specific action: The user takes a photo using the camera app and presses the upload button in the application.

[2021] Step 7:

[2022] Terminal: Uses image recognition technology to identify ingredients in captured images and generates an ingredient list.

[2023] Input: Uploaded image of the inside of a refrigerator

[2024] Output: List of identified ingredients

[2025] Specific operation: The device uses TensorFlow to perform image analysis and identify food ingredients. Example: "Tomato, carrot, chicken breast".

[2026] Step 8:

[2027] Server: Stores ingredient lists in a database and manages them as the user's current ingredient inventory information.

[2028] Input: Identified ingredient list

[2029] Output: Food inventory information stored in the database

[2030] Specific operation: The server uses MySQL to store ingredient information in a database.

[2031] Step 9:

[2032] Server: Uses a generative AI model to create a customized weekly meal plan based on the user's health data, ingredient inventory list, and preferences.

[2033] Input: User's health data, food inventory list, preferences

[2034] Output: Customized weekly meal plan

[2035] Specific operation: The server sends the following prompt to the generative AI model: "Create a weekly meal plan based on user A's health data and inventory information."

[2036] Step 10:

[2037] Terminal: Displays the generated menu and necessary nutrient and calorie information to the user.

[2038] Input: Customized menu

[2039] Output: Menu information presented to the user

[2040] Specific action: The device displays the menu in a list format on the app's UI.

[2041] Step 11:

[2042] Server: Based on the proposed menu, it identifies any missing ingredients and searches for information to suggest where to purchase them.

[2043] Input: Customized menu

[2044] Output: Information on missing ingredients and where to purchase them.

[2045] Specific operation: The server analyzes the menu data, identifies missing ingredients, and searches for suppliers from affiliated e-commerce platforms.

[2046] Step 12:

[2047] Terminal: Presents a generated list of potential purchases to the user and provides a link to the online store.

[2048] Input: Purchase candidate list

[2049] Output: A list of purchase options and links presented to the user.

[2050] Specific action: The device displays a list of ingredients containing a "purchase link" within the app.

[2051] Step 13:

[2052] Server: Collects data from wearable devices and food records to analyze the user's health status.

[2053] Input: Wearable device and meal record data

[2054] Output: Analysis results

[2055] Specific operation: The server uses a Python script to aggregate and analyze data.

[2056] Step 14:

[2057] Server: Generates health feedback and advice based on analysis results, and creates notification messages.

[2058] Input: Analysis results

[2059] Output: Feedback and advice

[2060] Specific operation: The server checks the user's unfulfilled goal and generates a message such as, "You are 1000 steps away from achieving your goal."

[2061] Step 15:

[2062] Device: Generated feedback and advice are sent to the user via push notifications.

[2063] Input: Feedback and advice

[2064] Output: Push notification to the user

[2065] Specific action: The device sends a notification using the smartphone's push notification function.

[2066] The above outlines the specific processing steps of this system's program. This allows users to receive personalized meal suggestions based on their individual health data and ingredient information, enabling them to efficiently purchase ingredients and manage their health.

[2067] (Application Example 1)

[2068] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[2069] In modern times, efficient and healthy dietary management is a crucial challenge for many people. However, in our busy lives, it's not easy to select appropriate recipes based on individual health conditions and current food availability, and to find suppliers for necessary ingredients. Furthermore, there is a lack of efficient feedback systems that link health data and dietary data. To address these challenges, there is a need for a system that utilizes users' health data, proposes individually customized recipes based on ingredient information, and allows for the quick acquisition of necessary ingredients.

[2070] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[2071] In this invention, the server includes means for communicating with a device for acquiring physical information, means for identifying ingredients from captured images, means for suggesting meals based on the identified ingredient information and the user's physical information, means for generating customized recipes based on the user's health data and ingredient information using a generative AI model, and means for identifying missing ingredients and suggesting purchases in cooperation with partner food delivery services. This enables the user to smoothly manage their diet efficiently and healthily.

[2072] 1. A "device for acquiring physical information" refers to a device that uses wearable devices, smartphones, etc., to acquire health data such as the user's heart rate, steps taken, and calories burned.

[2073] 2. "Means of communication" refers to methods and protocols for sending and receiving data over a network.

[2074] 3. "Means for identifying food ingredients from captured images" refers to technologies that use image recognition technology to identify food ingredients from photographs taken with a camera.

[2075] 4. "Identified food information" refers to information such as the name and quantity of food items identified through image recognition.

[2076] 5. "User's physical information" refers to health-related data such as the user's age, gender, heart rate, and steps taken.

[2077] 6. "Means of providing meal suggestions" refers to a method of generating appropriate meal menus and recipes based on the user's physical information and identified ingredient information.

[2078] 7. A "generative AI model" refers to an artificial intelligence model that generates customized recipes based on the user's health data and ingredient information.

[2079] 8. A "customized recipe" refers to a specific recipe generated based on the user's individual health condition and ingredient availability.

[2080] 9. "Means of identifying missing ingredients" refers to techniques for identifying ingredients that are not present in the refrigerator based on the proposed recipe.

[2081] 10. "Partner food delivery services" refers to service providers that allow customers to order ingredients and food online and have them delivered.

[2082] This invention is a system that suggests customized recipes based on the user's physical information and food information, and indicates where to purchase the necessary ingredients. This system is implemented using the following hardware and software.

[2083] Hardware and software to be used

[2084] 1. Hardware

[2085] Wearable devices: Devices that acquire physical information such as the user's heart rate, steps taken, and calories burned.

[2086] Smartphone: A device used to take pictures of food items inside the refrigerator and upload them to an application.

[2087] 2. Software

[2088] Image recognition technology: This technology identifies food items from images taken with a smartphone. An example is the Python library `face_recognition`.

[2089] Generative AI Model: An artificial intelligence model that generates customized recipes based on the user's health data and ingredient information. OpenAI's GPT-4 API, among others, is used.

[2090] System Operation Description

[2091] User registration and profile settings

[2092] Users install the application and, upon first launch, enter their name, age, gender, height, weight, allergy information, preferred foods, and health goals. This allows the user's individual profile to be saved on the server.

[2093] Integration with wearable devices

[2094] The server periodically receives data such as heart rate, steps taken, and calories burned from wearable devices and stores it in a database in real time. This allows the user's health status to be constantly monitored.

[2095] Making a list of ingredients in the refrigerator

[2096] The user takes a photo of the inside of their refrigerator with their smartphone and uploads it to the application. The server uses image recognition technology to identify the food items in the photo and generates a list of those items. These identified items are added to the user's inventory list and managed as current inventory information.

[2097] Generating customized recipes and menus

[2098] The server uses a generative AI model to create a customized weekly meal plan based on the user's health data, ingredient inventory list, and preferences. The generated meal plan, along with necessary nutrient and calorie information, is presented to the user via their smartphone.

[2099] Specific example:

[2100] If the user's health status does not meet the target health level, a generative AI model will be used to suggest recipes.

[2101] Examples of prompt statements to use:

[2102] User information: {'name': 'User 1', 'age': 30, 'gender': 'male', 'height': 175, 'weight': 70, 'allergies': ['dairy products'], 'preferences': ['vegetables', 'chicken']}

[2103] Health data: {'heart_rate': 70, 'steps': 8000, 'calories_burned': 500}

[2104] Ingredients in the refrigerator: ['Tomato', 'Carrot', 'Chicken']

[2105] Based on this, please propose a customized recipe.

[2106] Suggestions for places to buy groceries

[2107] The server lists the necessary ingredients based on the suggested menu and identifies any missing ingredients. Next, it searches for ingredient purchase options from partner food delivery services and generates links to the nearest purchase locations or online stores. These purchase option lists are presented to the user via their smartphone, along with links to online stores.

[2108] Health data feedback

[2109] The server aggregates data from wearable devices and food records to analyze the user's health status. Based on the analysis, it generates health feedback and advice, and creates notification messages. The generated feedback and advice are delivered to the user via push notifications on their smartphone.

[2110] This allows users to quickly receive personalized meal suggestions based on their health data and ingredient information, enabling them to efficiently purchase ingredients and live a healthy lifestyle.

[2111] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[2112] Step 1:

[2113] The user installs the application and sets up a profile upon first launch.

[2114] Input: Name, age, gender, height, weight, allergy information, favorite foods, health goals.

[2115] Data processing: Use this data to generate individual user profiles.

[2116] Output: The user's profile is saved to the server's database.

[2117] Step 2:

[2118] The server periodically receives health data from wearable devices.

[2119] Input: Data such as heart rate, steps taken, and calories burned, transmitted from a wearable device.

[2120] Data processing: Received data is recorded in a database in real time to monitor the user's health status.

[2121] Output: The latest health data will be updated.

[2122] Step 3:

[2123] Users take photos of the food in their refrigerator with their smartphones and upload the images to the app.

[2124] Input: A photo showing the food inside a refrigerator.

[2125] Data processing: The server uses image recognition technology to identify ingredients in the photograph.

[2126] Output: A list of recognized ingredients is generated and saved to the database.

[2127] Step 4:

[2128] The server generates customized recipes based on the user's health data and ingredient inventory list.

[2129] Input: Prompt text for the generated AI model (user profile, health data, list of identified foods).

[2130] Data processing: Generate customized recipes using a generative AI model (e.g., GPT-4).

[2131] Output: The generated recipe and necessary nutrient and calorie information are produced.

[2132] Step 5:

[2133] The server lists the necessary ingredients based on the proposed recipe and identifies any missing ingredients.

[2134] Input: Generated recipe and current ingredient inventory information.

[2135] Data processing: Identify and list any missing ingredients.

[2136] Output: A list of missing ingredients is generated.

[2137] Step 6:

[2138] The server searches for grocery purchase options from partner food delivery services and generates links to the nearest purchase locations and online stores.

[2139] Input: List of missing ingredients.

[2140] Data processing: Search the database of food delivery services to obtain information on potential purchases.

[2141] Output: A list of potential purchases and links to online stores are generated.

[2142] Step 7:

[2143] The device presents the user with a generated recipe, information on necessary nutrients and calories, and a list of recommended purchases.

[2144] Input: Generated recipe, required nutrient and calorie information, a list of suggested purchases, and links to online stores.

[2145] Data processing: Converts information into a format that is easy for users to understand.

[2146] Output: It will be displayed on the smartphone screen.

[2147] Step 8:

[2148] The server aggregates data from wearable devices and food records to analyze the user's health status.

[2149] Input: User's health data, dietary record data.

[2150] Data processing: Use analytical algorithms to evaluate the user's health status.

[2151] Output: Health feedback and advice are generated.

[2152] Step 9:

[2153] The device will send the generated feedback and advice to the user via push notifications.

[2154] Input: Feedback and advice sent from the server.

[2155] Data processing: Convert to a format that can be displayed as a push notification.

[2156] Output: It will be displayed as a push notification on your smartphone.

[2157] Through the above processing steps, users can quickly obtain customized meal suggestions based on their health data and food information, enabling them to efficiently purchase ingredients and lead a healthy lifestyle.

[2158] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[2159] This invention relates to a system that acquires a user's physical information and emotional state, and presents meal suggestions and locations for purchasing ingredients. The following describes a specific form for realizing this system.

[2160] User registration and profile settings

[2161] 1. User: Install the application and, upon first launch, enter your name, age, gender, height, weight, and information about allergies, preferred foods, and health goals. This information will be used to create your individual profile.

[2162] 2. Server: Saves the entered information to the database and creates a profile for each user.

[2163] Integration with wearable devices

[2164] 1. User: Link the wearable device and application in the settings screen.

[2165] 2. Terminal: Regularly receives data such as heart rate, steps taken, and calories burned from a wearable device.

[2166] 3. Server: Receives device data sent from terminals and stores it in the database.

[2167] Making a list of ingredients in the refrigerator

[2168] 1. User: Take a photo of the inside of the refrigerator with your smartphone and upload it to the application.

[2169] 2. Terminal: Uses image recognition technology to identify ingredients in a photograph and creates an ingredient list.

[2170] 3. Server: Stores the ingredient list in a database and manages it as the user's current inventory information.

[2171] Specific example

[2172] User: Upload a photo of the refrigerator to the application.

[2173] Terminal: Analyzes the photo and identifies "tomato, carrot, and chicken breast."

[2174] Server: Identify these ingredients and add them to the user's inventory list.

[2175] Generating customized recipes and menus

[2176] 1. Server: Uses generative AI to create a customized weekly meal plan based on the user's health data, ingredient inventory list, preferences, and allergy information.

[2177] 2. Terminal: Presents the generated menu and necessary nutrient and calorie information to the user.

[2178] Emotional recognition and adaptation of meal suggestions using an emotional engine

[2179] 1. User: Inputs facial expressions and voice through the application. Also, notifies emotional states by inputting text.

[2180] 2. Device: The emotion engine analyzes the user's facial expressions, voice, or text data to recognize their current emotional state.

[2181] 3. Server: Analyzes recognized emotional information in conjunction with health data and adapts meal suggestions based on the user's emotions.

[2182] Specific example

[2183] User: Type the text "I'm tired today" into the chatbot within the application.

[2184] Terminal: The emotion engine analyzes this text data and recognizes that the user is tired.

[2185] Server: Suggest highly nutritious ingredients and easy-to-prepare dishes to reduce user fatigue.

[2186] Suggestions for places to buy groceries

[2187] 1. Server: Identify any missing ingredients based on the proposed menu.

[2188] 2. Server: For any missing ingredients, it searches for potential purchase options from partner e-commerce platforms and gathers information on the nearest places to buy them and online stores.

[2189] 3. Terminal: Presents the generated list of potential purchases to the user and provides a link to the online store.

[2190] Feedback on health data and emotional data

[2191] 1. Server: Collects data from wearable devices, meal records, and emotional data to analyze the user's health status.

[2192] 2. Server: Generates health feedback and advice, emotion-based recommendations, and notification messages based on the analysis results.

[2193] 3. Device: Send feedback and advice to users via push notifications.

[2194] Specific example

[2195] Server: Verify that the user's activity level has not reached 70% of the target.

[2196] The device notifies the user that "You can reach your goal by walking 1000 more steps." If it detects that the user's stress level is high, it suggests, "Try some relaxing herbal tea."

[2197] The above describes the embodiments of the present invention. In this form, users can receive not only individual health data and food information, but also optimal meal suggestions tailored to their mood at any given time, enabling them to efficiently purchase food ingredients and lead a healthy and balanced life.

[2198] The following describes the processing flow.

[2199] User registration and profile settings

[2200] Step 1:

[2201] User: Install the application and, upon first launch, enter your name, age, gender, height, weight, and information about allergies, preferred foods, and health goals.

[2202] Step 2:

[2203] Terminal: Sends the entered information to the server.

[2204] Step 3:

[2205] Server: Stores received information in a database and creates user profiles.

[2206] Integration with wearable devices

[2207] Step 4:

[2208] User: Link the wearable device and application in the settings screen.

[2209] Step 5:

[2210] Device: Once the connection is successfully completed, enable the setting to periodically receive data such as heart rate, steps, and calories burned from the wearable device.

[2211] Step 6:

[2212] Server: Receives device data sent from terminals and stores it in the database.

[2213] Making a list of ingredients in the refrigerator

[2214] Step 7:

[2215] User: Take a photo of the inside of the refrigerator with your smartphone and upload it to the application.

[2216] Step 8:

[2217] Device: Uses AI image recognition technology to identify ingredients in a photograph and creates an ingredient list.

[2218] Step 9:

[2219] Terminal: Sends the generated ingredient list to the server.

[2220] Step 10:

[2221] Server: Stores the received ingredient list in the database and manages it as the user's current inventory information.

[2222] Generating customized recipes and menus

[2223] Step 11:

[2224] Server: Uses generative AI to create a customized weekly meal plan based on the user's health data, ingredient inventory list, preferences, and allergy information.

[2225] Step 12:

[2226] Server: Sends the generated menu to the terminal.

[2227] Step 13:

[2228] Terminal: Displays the received menu and necessary nutrient and calorie information to the user.

[2229] Emotional recognition and adaptation of meal suggestions using an emotional engine

[2230] Step 14:

[2231] User: Notifies emotional states by inputting facial expressions, voice, or text through the application.

[2232] Step 15:

[2233] Device: The emotion engine analyzes the user's facial expressions, voice, or text data to recognize their current emotional state.

[2234] Step 16:

[2235] Server: Analyzes recognized emotional information in conjunction with health data and adapts meal suggestions based on the user's emotions.

[2236] Suggestions for places to buy groceries

[2237] Step 17:

[2238] Server: Identify any missing ingredients based on the proposed menu.

[2239] Step 18:

[2240] Server: Searches for potential purchase options from partner e-commerce platforms for missing ingredients, gathering information on the nearest places to buy them and online stores.

[2241] Step 19:

[2242] Server: Sends a list of potential purchases to the terminal.

[2243] Step 20:

[2244] Terminal: Presents a generated list of potential purchases to the user and provides a link to the online store.

[2245] Feedback on health data and emotional data

[2246] Step 21:

[2247] Server: Collects data from wearable devices, meal records, and emotional data to analyze the user's health status.

[2248] Step 22:

[2249] Server: Generates health feedback and advice, emotion-based recommendations, and notification messages based on analysis results.

[2250] Step 23:

[2251] Server: Sends the generated feedback and advice to the terminal.

[2252] Step 24:

[2253] Device: Send feedback and advice to users via push notifications.

[2254] The above describes the specific actions taken at each processing step. This detailed processing flow allows users to receive optimal meal suggestions based on the latest health data and ingredient information, efficiently purchase ingredients, and lead a healthy and balanced life. In addition, it provides even more personalized support by offering suggestions that take into account the user's emotional state.

[2255] (Example 2)

[2256] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[2257] Conventional meal suggestion systems based solely on the user's physical data and ingredient information, failing to consider the user's emotional state. Furthermore, suggestions for ingredient sourcing locations were limited, failing to integrate information from online stores and physical retail outlets. Therefore, there was a need to provide users with optimal and efficient meal suggestions and ingredient purchasing methods.

[2258] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[2259] In this invention, the server includes means for communicating with equipment for acquiring physical data, means for identifying food ingredients from captured content, means for suggesting meals based on the identified food ingredient data, the user's physical data, and emotional state, and means for identifying missing ingredients based on the suggested meal and indicating where to obtain them. This enables meal suggestions that take into account the user's emotional state in addition to their physical data and food ingredient information, and further enables the provision of food ingredient acquisition locations by integrating information from e-commerce platforms and physical stores, thereby providing optimal and efficient meal suggestions and means for purchasing ingredients.

[2260] "Physical data" refers to information about the user's body, including data such as height, weight, heart rate, steps taken, and calories burned.

[2261] "Equipment" refers to wearable devices, sensor devices, and other devices used to acquire the user's physical data.

[2262] "Content" refers to images and photographs taken by users, including, in particular, food items in a refrigerator and cooked meals.

[2263] "Food ingredient data" refers to information such as the type, quantity, and availability of food items identified from the content.

[2264] "Emotional state" refers to information that indicates the user's current psychological and emotional state, including, for example, fatigue, stress, and satisfaction.

[2265] "Meal suggestions" refers to the act of suggesting optimal meal menus and recipes based on the user's physical data and emotional state.

[2266] "Place of acquisition" refers to the location where the ingredients needed for the proposed meal can be purchased, including online stores and physical stores.

[2267] An "e-commerce platform" refers to a website or application that facilitates the buying and selling of goods online, and examples include e-commerce sites.

[2268] A "physical store" refers to an actual store or supermarket where users can visit in person to purchase groceries.

[2269] This invention relates to a system that acquires a user's physical data and emotional state, and presents personalized meal suggestions and locations for obtaining ingredients. The following describes specific embodiments of this system.

[2270] Overall structure

[2271] This system consists of a device (wearable device) for acquiring physical data, the user's smartphone (terminal), and a server connected to them. The server receives and analyzes the data and provides meal suggestions and locations for data acquisition.

[2272] User registration and profile settings

[2273] Users first install the application on their smartphone and, upon initial launch, enter their name, age, gender, height, weight, allergy information, preferred foods, and health goals. This creates an individual profile. The server stores the information entered by the user in a database and builds the profile. The application interface is designed to allow users to easily enter the necessary information.

[2274] Integration with wearable devices

[2275] The user pairs a wearable device (e.g., a wristwatch-type heart rate monitor) with the application via the smartphone's settings screen. The wearable device periodically sends data such as heart rate, steps taken, and calories burned to the device. The device periodically sends this data to a server. The server stores the received data in a database and manages the user's health data.

[2276] Making a list of ingredients in the refrigerator

[2277] The user takes a photo of the inside of their refrigerator with their smartphone and uploads it to the application. The device uses image recognition technology (e.g., Amazon Rekognition) to identify the food items in the photo and create a list of those items. The server stores the identified food list in a database and manages it as the user's current inventory information.

[2278] Specific example

[2279] When a user uploads a photo of their refrigerator to the application, the device analyzes the image and identifies "tomatoes, carrots, and chicken breast." The server saves these ingredients to a database and updates the inventory list.

[2280] Generating customized recipes and menus

[2281] The server uses generative AI (e.g., OpenAI's GPT-4) to create a customized weekly meal plan based on the user's health data, ingredient inventory list, preferences, and allergy information. The terminal then presents the generated meal plan along with necessary nutrient and calorie information to the user.

[2282] Examples of prompt statements

[2283] "I'm a woman in my 30s, aiming to lose weight. I have tomatoes, carrots, and chicken breast in my refrigerator. Please create a low-calorie, nutritionally balanced meal plan for one week."

[2284] Emotional recognition and adaptation of meal suggestions using an emotional engine

[2285] Users notify their emotional state through the application by inputting facial expressions, voice, or text. The emotion engine analyzes this input data to recognize the user's current emotional state. The server then comprehensively analyzes the recognized emotional state and physical data to provide meal suggestions based on the user's emotions.

[2286] Specific example

[2287] When a user types "I'm tired today" into the in-app chatbot, the device's emotion engine analyzes this text data and recognizes that the user is tired. The server then suggests highly nutritious ingredients and easy-to-prepare dishes.

[2288] Suggestions for locations to obtain ingredients

[2289] The server identifies any missing ingredients based on the suggested menu and searches for purchase options from partner e-commerce platforms and physical stores. The terminal presents the generated list of purchase options to the user and provides links to online stores.

[2290] Feedback on health data and emotional data

[2291] The server aggregates data from wearable devices, meal logs, and emotional data to analyze the user's health status. Based on the analysis, it generates feedback and advice and notifies the user. The device delivers this feedback and advice to the user via push notifications. For example, it provides specific advice such as, "You can achieve your goal by walking another 1000 steps," or emotionally-based recommendations such as, "Try a relaxing herbal tea."

[2292] The above describes a specific embodiment for carrying out the present invention. This allows users to receive not only individual health data and food information, but also optimal meal suggestions tailored to their mood at any given time, enabling them to efficiently purchase food ingredients and lead a healthy and balanced life.

[2293] The flow of the specific processing in Example 2 will be explained using Figure 13.

[2294] Program execution flow and detailed explanation of each processing step

[2295] User registration and profile settings

[2296] Step 1:

[2297] The user installs the application and launches it for the first time.

[2298] Input: None

[2299] Output: Display of the app's welcome screen

[2300] Specific operation: The user downloads and launches the app. The interface is designed to be intuitive and easy to understand.

[2301] Step 2:

[2302] The user enters their name, age, gender, height, weight, allergy information, favorite foods, and health goals.

[2303] Input: User information (name, age, gender, height, weight, allergy information, favorite foods, health goals)

[2304] Output: User profile data

[2305] Specific action: The user enters the required information into the input form and presses the submit button.

[2306] Step 3:

[2307] The server saves the entered information to the database and builds a user profile.

[2308] Input: User profile data

[2309] Output: Saved to database successfully

[2310] Specific operation: The server receives profile data and stores it in the database. A relational database such as MySQL is used for this database.

[2311] Integration with wearable devices

[2312] Step 1:

[2313] The user pairs the wearable device and the application in the app settings screen.

[2314] Input: Pairing request

[2315] Output: Pairing complete notification

[2316] Specific steps: The user selects the wearable device on their smartphone's Bluetooth settings screen and pairs it.

[2317] Step 2:

[2318] The device periodically receives data such as heart rate, steps taken, and calories burned from the wearable device.

[2319] Input: Device data (heart rate, steps, calories burned)

[2320] Output: Automatic saving of received data at regular intervals.

[2321] Specific operation: The device periodically (e.g., every hour) retrieves data from the wearable device and saves it locally.

[2322] Step 3:

[2323] The server receives device data sent from the terminal and stores it in the database.

[2324] Input: Device data from the terminal

[2325] Output: Saved to database successfully

[2326] Specific operation: At regular intervals (e.g., every 24 hours), the server receives data from the terminal, analyzes it, and saves it to the database.

[2327] Making a list of ingredients in the refrigerator

[2328] Step 1:

[2329] The user takes a photo of the inside of the refrigerator with their smartphone and uploads it to the application.

[2330] Input: Photo of the inside of the refrigerator

[2331] Output: Upload complete notification

[2332] Specific action: The user uses their smartphone camera to take a picture of the inside of the refrigerator and uploads the image to the app.

[2333] Step 2:

[2334] The device uses image recognition technology to identify food items in the photograph.

[2335] Input: Uploaded photo

[2336] Output: Identified ingredient list

[2337] Specific operation: The device uses image recognition APIs such as Amazon Rekognition to automatically analyze and identify food items in a photograph.

[2338] Step 3:

[2339] The server stores a list of identified ingredients in a database and manages it as the user's current inventory information.

[2340] Input: Identified ingredient list

[2341] Output: Saved to database successfully

[2342] Specific operation: The server receives the identified ingredient information and stores it in the database.

[2343] Generating customized recipes and menus

[2344] Step 1:

[2345] The server uses generative AI to create a customized weekly meal plan based on the user's health data, ingredient inventory list, preferences, and allergy information.

[2346] Input: Health data, food inventory list, preferences and allergy information

[2347] Output: Customized menu

[2348] Specific operation: The server sends prompt messages to the AI ​​model (e.g., OpenAI's GPT-4) and receives the generated menu.

[2349] Step 2:

[2350] The device displays the generated menu and information on necessary nutrients and calories to the user.

[2351] Input: Customized menu

[2352] Output: Presentation of menu information

[2353] Specific operation: The terminal displays the menu information received from the server on its screen, along with information on necessary nutrients and calories.

[2354] Emotional recognition and adaptation of meal suggestions using an emotional engine

[2355] Step 1:

[2356] Users can input facial expressions and voices through the application, or input text to notify their emotional state.

[2357] Input: Facial expressions, voice, or text data

[2358] Output: Notification of emotional state

[2359] Specific operation: Users input their emotional state through the app using their facial photo, voice, or text data.

[2360] Step 2:

[2361] The device uses an emotion engine to analyze facial expressions, voice, or text to recognize the user's current emotional state.

[2362] Input: Emotional data (facial expressions, voice, text)

[2363] Output: Recognized emotional state

[2364] Specific operation: The device uses an emotion recognition engine (e.g., Microsoft Azure Cognitive Services) to analyze input data and recognize the emotional state.

[2365] Step 3:

[2366] The server integrates and analyzes recognized emotional states and physical data to provide meal suggestions based on the user's emotions.

[2367] Input: Recognized emotional state, physical data

[2368] Output: Meal suggestions based on emotional state

[2369] Specific operation: Based on the analysis results, the server generates meal suggestions that include highly nutritious ingredients and easy-to-prepare dishes.

[2370] Suggestions for locations to obtain ingredients

[2371] Step 1:

[2372] The server identifies any missing ingredients based on the suggested menu.

[2373] Input: Suggested menu, ingredient inventory list

[2374] Output: List of missing ingredients

[2375] Specific operation: The server compares the menu with the inventory list to identify any missing ingredients.

[2376] Step 2:

[2377] The server searches for potential purchase options for ingredients that are currently out of stock, from partner e-commerce platforms and physical stores.

[2378] Input: List of missing ingredients

[2379] Output: Purchase candidate list

[2380] Specific operation: The server uses APIs from e-commerce platforms and physical stores to search for purchasing options for missing ingredients.

[2381] Step 3:

[2382] The device displays a list of potential purchases to the user and provides a link to the online store.

[2383] Input: Purchase candidate list

[2384] Output: Display of purchase candidate list

[2385] Specific operation: The device displays a list of potential purchases on the screen and provides the user with a link to each option.

[2386] Feedback on health data and emotional data

[2387] Step 1:

[2388] The server collects data from wearable devices, meal logs, and emotional data to analyze the user's health status.

[2389] Input: Wearable device data, food log data, emotional data

[2390] Output: Analysis results of health status

[2391] Specific operation: The server uses Python or similar languages ​​to execute analysis scripts and collect and analyze user health data.

[2392] Step 2:

[2393] The server generates feedback and advice based on the analysis results and notifies the user.

[2394] Input: Analysis results of health status

[2395] Output: Feedback and advice

[2396] Specific operation: The server generates appropriate advice and feedback for the user based on the analysis results and creates notification messages.

[2397] Step 3:

[2398] The device will send feedback and advice to the user via push notifications.

[2399] Input: Feedback and advice

[2400] Output: Notification message

[2401] Specific actions: The device uses push notifications to inform the user of specific advice, such as "You can reach your goal by walking 1000 more steps." It also provides emotion-based recommendations, such as "Try a relaxing herbal tea."

[2402] (Application Example 2)

[2403] Next, we will explain application example 2....

Claims

1. A device for acquiring physical information and means for communication, A means of identifying ingredients from captured images, A means for suggesting meals based on identified food ingredient information and the user's physical information, A means of identifying the necessary ingredients based on the proposed meal and indicating where to purchase them, A system that includes this.

2. One method for image recognition involves using a database that stores information on previously purchased food items. The system according to claim 1, which includes means for providing integrated information from online stores and physical stores to indicate the place of purchase.

3. The system according to claim 1, further comprising means for suggesting a health strategy based on the user's current physical information and dietary history.

Images