system

Abstract

We provide the system. [Solution] A means for users to input ingredient information, A means of generating suggested menus based on input ingredient information, A means of providing cooking instructions corresponding to the generated menu in audio and video, A means for users to input their health checkup results, A means for generating suggestions for improving dietary habits based on input health check results, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, and includes steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance as a response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In modern society, it is difficult to maintain a balanced diet in a busy daily life. Also, selecting an appropriate diet based on a health check result requires specialized knowledge, and many people struggle with self-management. Furthermore, information on how to use ingredients and how to cook dishes is often insufficient. As a result, inappropriate eating habits and health problems may occur.

Means for Solving the Problems

[0005] To solve the above problems, the present invention provides a means for the user to input information about ingredients. Next, it provides a means for generating suggested menus based on the input ingredient information. Furthermore, it provides a means for providing cooking procedures corresponding to the generated menus in audio and video. It also provides a means for the user to input their health checkup results and a means for generating suggestions for improving their diet based on the input health checkup results. In this way, the system makes it possible for the user to easily select a balanced and appropriate meal and manage their health.

[0006] A "user" refers to an individual who uses the system.

[0007] "Ingredient information" refers to data about the types and quantities of ingredients owned by the user.

[0008] A "menu" refers to a recipe for a dish prepared using specific ingredients.

[0009] "Cooking instructions" refers to information about the steps and methods for preparing a dish.

[0010] "Sound" refers to a means of transmitting information using sound.

[0011] "Video" refers to a means of conveying information using visual images.

[0012] "Health checkup results" refer to data about one's health status obtained through examinations conducted at a medical institution.

[0013] "Dietary improvement suggestions" refer to information that suggests a balanced diet based on the user's health condition. [Brief explanation of the drawing]

[0014] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2]It is a conceptual diagram showing an example of the main functions of a data processing device and a smart device according to the first embodiment. [Figure 3] It is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] It is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which multiple emotions are mapped. [Figure 10] It shows an emotion map to which multiple emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when an emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when an emotion engine is combined.

Mode for Carrying Out the Invention

[0015] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described according to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, the labeled processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0018] In the following embodiments, the labeled RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0019] In the following embodiments, the labeled storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, and the like.

[0020] In the following embodiments, the labeled communication I / F (Interface) is an interface including a communication processor and an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark), and the like.

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0022] [First Embodiment]

[0023] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0024] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0027] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0030] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0034] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0035] This invention is a system in which the user inputs data on the ingredients stocked in their refrigerator, and based on that data, it suggests appropriate menus and provides cooking instructions via voice and video. Furthermore, it is a system that supports health promotion by inputting the user's health checkup results and suggesting a balanced diet based on those results. Specific embodiments for carrying out this invention are described below.

[0036] Overall system configuration

[0037] The system consists of user terminals, a central server, and a database.

[0038] User terminal: Refers to devices such as smartphones and PCs, which users use to input data or receive suggested information.

[0039] Central server: This is the central component of a system that processes and manages data and provides users with the information they need.

[0040] Database: Stores data such as ingredient information, recipe information, and health checkup results.

[0041] Program flow

[0042] 1. The user enters the ingredient information.

[0043] Users access and log in to the system using their smartphones or PCs. After logging in, they enter the types and quantities of food items in their refrigerator. For example, they might enter "6 eggs, 2 tomatoes, 500ml milk." The entered information is sent from the device to the server and stored in the database.

[0044] 2. The server suggests a menu.

[0045] The server uses the received ingredient information to query the ingredient database and recipe database to find appropriate menus. For example, it might select "omelet" and "tomato soup" using eggs, tomatoes, and milk. The selected menus are sent to the terminal and displayed to the user.

[0046] 3. Audio and video support for cooking instructions

[0047] The server generates audio guides and video links for cooking instructions corresponding to the suggested menu. These are sent to the terminal for the user to use while cooking. For example, the video link might show steps such as "how to whisk eggs."

[0048] 4. The user enters their health checkup results.

[0049] Users input their health checkup results (e.g., LDL cholesterol levels and blood sugar levels) into the system. This information is sent from the terminal to the server and stored in a database.

[0050] 5. Dietary suggestions based on health checkup results

[0051] The server analyzes the stored health check results and generates suggestions for improving the user's diet based on their health status. For example, if the user has high LDL cholesterol levels, menus using cholesterol-lowering ingredients will be suggested. These suggestions are also sent to the device and displayed to the user.

[0052] 6. Users create meals and promote health.

[0053] Users create and consume appropriate meals according to the server's suggestions, thereby promoting their health. This allows them to maintain a balanced diet.

[0054] Specific example

[0055] For example, if a user enters "6 eggs, 2 tomatoes, 500ml milk" as the ingredients in their refrigerator, the server will suggest recipes for "omelet" and "tomato soup" using these ingredients. The device will also provide a video and audio guide demonstrating the cooking process, making it easy for the user to prepare the dishes. Furthermore, if the user's health check results indicate "high LDL cholesterol levels," the server will advise using ingredients rich in cholesterol-lowering components.

[0056] This invention is a system that suggests appropriate menus based on the ingredients in the user's refrigerator and their health checkup results, supporting a balanced diet and contributing to the user's improved health.

[0057] The following describes the processing flow.

[0058] Step 1:

[0059] Users access the system using their smartphones or PCs and enter their login information. Upon successful login, the main screen is displayed.

[0060] Step 2:

[0061] The user enters information about the food items stocked in their refrigerator. For example, they might enter the type and quantity of food items such as "6 eggs, 2 tomatoes, 500ml milk."

[0062] Step 3:

[0063] The terminal sends the entered ingredient information to the server via the interface. This transmission is performed using methods such as HTTP requests or API calls.

[0064] Step 4:

[0065] The server saves the received ingredient information to the database. During the saving process, the type and quantity of ingredients are registered in the database as key-value pairs.

[0066] Step 5:

[0067] The server queries the ingredient database and recipe database based on the stored ingredient information. It then uses database queries to search for suitable recipes.

[0068] Step 6:

[0069] The server generates suggested menus from the recipes it finds. For example, it might select "Omelet" and "Tomato Soup" using eggs, tomatoes, and milk.

[0070] Step 7:

[0071] The server sends the generated suggested menu to the terminal. This is also done via HTTP requests or API responses.

[0072] Step 8:

[0073] The terminal displays suggested menus to the user. The user interface shows detailed recipes for "Omelet" and "Tomato Soup."

[0074] Step 9:

[0075] The server generates an audio guide and video link for the cooking procedure and sends it to the terminal. This process provides the audio file and the URL of the video.

[0076] Step 10:

[0077] The device displays the received audio guide and video link to the user. The user can then play the audio guide or watch the video.

[0078] Step 11:

[0079] The user obtains their own health checkup results and inputs them into the system. For example, they might input data such as "LDL cholesterol 150 mg / dL, blood sugar level 110 mg / dL".

[0080] Step 12:

[0081] The device sends the entered health check results to the server. This is also done via HTTP requests or API calls.

[0082] Step 13:

[0083] The server stores the received health check results in a database and evaluates the user's health status. For example, if the LDL cholesterol level is high, the system may advise avoiding certain foods.

[0084] Step 14:

[0085] The server generates dietary improvement suggestions based on the user's health status. For example, it advises users with high LDL cholesterol levels to use more tomatoes and other low-cholesterol foods.

[0086] Step 15:

[0087] The server sends generated dietary improvement suggestions to the terminal and displays them to the user. These include health-conscious recipes and advice on selecting ingredients.

[0088] Step 16:

[0089] Users prepare meals according to the server's suggestions, aiming to improve their health. Users practice cooking while referring to the provided recipes, audio guides, and videos.

[0090] (Example 1)

[0091] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0092] Current cooking support systems have limited functionality in providing appropriate menus and cooking procedures based on user input of ingredients in their refrigerators. Furthermore, few systems offer personalized dietary suggestions based on health checkup results, making them insufficient for users aiming to improve their health. As a result, users are forced to go through the trouble of searching for information themselves and devising menus that suit their health condition.

[0093] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0094] In this invention, the server includes means for the user to input ingredient information, means for generating suggested menus based on the input ingredient information, means for providing cooking procedures corresponding to the generated menus in audio and video, means for the user to input health checkup results, means for generating suggestions for improving dietary habits based on the input health checkup results, means for a central processing unit to store ingredient information and health checkup results in a database and generate menus and health suggestions based on the stored data, and means for displaying the suggested menus and dietary improvement suggestions on a terminal and providing audio guidance and video. As a result, the user can easily and effectively receive suggestions for appropriate menus based on the ingredients in their refrigerator and their health checkup results, and can cook while confirming the cooking procedures in audio and video. Furthermore, by receiving suggestions for improving dietary habits tailored to their individual health condition, health improvement is promoted.

[0095] A "user" refers to an end user who accesses the system, inputs information, and receives suggested menus and cooking instructions.

[0096] "Ingredient information" refers to information entered by the user regarding the types and quantities of ingredients in the refrigerator.

[0097] "Menu" refers to the recipes and types of dishes suggested based on the ingredient information entered by the user.

[0098] "Cooking instructions" refer to the specific methods for preparing the dishes corresponding to the suggested menu, and these are provided in audio and video formats.

[0099] "Audio guide" refers to audio instructions or guidance provided for users to refer to while cooking.

[0100] A "video link" refers to a link to a video provided to help users visually understand the cooking procedure.

[0101] "Health checkup results" refer to various diagnostic data indicating the user's health status, including blood test results, cholesterol levels, blood sugar levels, and so on.

[0102] A "central processing unit" refers to a server that receives input data from users, performs appropriate processing and data storage based on that data, and generates and provides necessary information.

[0103] A "database" refers to a collection of information used to store and manage information such as food ingredient information, recipe information, and health checkup results.

[0104] "Suggestions" refer to menus and health management methods for improving dietary habits that are generated based on user input data.

[0105] A "terminal" is a device used by a user to access a system, and specifically refers to smartphones and personal computers.

[0106] This invention is a system in which the user inputs data on the ingredients stocked in their refrigerator, and based on that data, it suggests appropriate menus and provides cooking instructions via voice and video. Furthermore, it is a system that supports health promotion by inputting the user's health checkup results and suggesting a balanced diet based on those results. Specific embodiments for carrying out this invention are described below.

[0107] Overall system configuration

[0108] The system consists of user terminals, a central processing unit, and a database.

[0109] User terminal: This refers to devices such as smartphones and personal computers that users use to input data or receive suggested information.

[0110] Central Processing Unit: This is the central component of a system that processes and manages data and provides users with the information they need.

[0111] Database: Stores data such as ingredient information, recipe information, and health checkup results.

[0112] Program processing

[0113] The user enters ingredient information.

[0114] Users access and log in to the system using their smartphones or personal computers. After logging in, they enter the types and quantities of food items in their refrigerator. For example, a user might enter: "6 eggs, 2 tomatoes, 500ml milk." This information is sent from the user's device to the server and stored in the database.

[0115] The server receives ingredient information and stores it in the database.

[0116] The server receives ingredient information sent from the user terminal and performs analysis. The received data is then stored in the database using appropriate SQL queries. In this process, the server specifies the table and field where the data will be stored and places it there.

[0117] The server suggests a menu.

[0118] The server queries the ingredient database and recipe database based on the received ingredient information to find a suitable menu. For example, it might suggest "Omelet with eggs, tomatoes, and milk, and tomato soup." The selected menu is sent to the user's terminal in JSON format and displayed to the user.

[0119] The server prepares and provides audio and video instructions for cooking.

[0120] The server prepares audio guides and video links for cooking instructions corresponding to the proposed menu. These resources are retrieved using existing databases or external APIs. Specifically, it generates video links such as "How to whisk eggs" and sends them to the user's terminal. The user then uses these as a reference to cook.

[0121] The user enters their health checkup results.

[0122] Users enter their health checkup results (e.g., LDL cholesterol levels and blood sugar levels) into a dedicated form. This information is also sent from the user's terminal to the server and stored in the database.

[0123] The server receives the health check results and saves them to the database.

[0124] The server receives health checkup results sent from user terminals, analyzes them, and stores them in a database. Specifically, it adds the health checkup data to the appropriate table and links it to user-specific data.

[0125] The server generates dietary suggestions based on health checkup results.

[0126] The server analyzes the user's health status based on their health checkup results and generates suggestions for appropriate dietary improvements. For example, if the user has high LDL cholesterol levels, the server will suggest menus using ingredients that help lower cholesterol. These suggestions are also sent to the user's terminal in JSON format and displayed to the user.

[0127] Users create meals according to generated suggestions, aiming to improve their health.

[0128] Users prepare dishes by following the menu and cooking instructions received from the server. Specifically, they play the provided audio guide and watch videos while cooking. Through this process, users can maintain a balanced diet and improve their health.

[0129] Specific example

[0130] For example, if a user enters "6 eggs, 2 tomatoes, 500ml milk" as the ingredients in their refrigerator, the server analyzes these ingredients and suggests recipes for "omelet" and "tomato soup." A video link and audio guide demonstrating the cooking process are provided to the user's device, allowing them to easily prepare the dishes. Furthermore, if the user's health check results indicate "high LDL cholesterol levels," the server advises using ingredients rich in cholesterol-lowering components.

[0131] Example of a prompt

[0132] "I have 6 eggs, 2 tomatoes, and 500ml of milk in my refrigerator. Please suggest some recipes using these ingredients. Also, please provide suggestions for what to do if I have high LDL cholesterol levels."

[0133] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0134] Step 1:

[0135] The user enters the ingredient information.

[0136] Operation: Users access the system and log in using a smartphone or personal computer. After logging in, they enter the types and quantities of food items in their refrigerator into a form. For example, "6 eggs, 2 tomatoes, 500ml milk."

[0137] Input: Information about the ingredients in the refrigerator.

[0138] Output: The entered ingredient information is sent from the terminal to the server.

[0139] Step 2:

[0140] The server receives ingredient information and stores it in the database.

[0141] Operation: The server receives ingredient information sent from the terminal, creates an SQL query, and saves it to the database.

[0142] Input: Ingredient information sent from the device.

[0143] Output: Ingredient information stored in the database.

[0144] Step 3:

[0145] The server suggests a menu.

[0146] Operation: The server queries the ingredient database and recipe database based on the stored ingredient information to find a suitable menu. For example, it might select recipes for "omelet" and "tomato soup" using "eggs, tomatoes, and milk."

[0147] Input: Food ingredient information stored in the database.

[0148] Output: The suggested menu is sent to the terminal.

[0149] Step 4:

[0150] The server prepares and provides audio and video instructions for the cooking process.

[0151] Operation: The server generates audio guides and video links for cooking instructions corresponding to the proposed menu and sends them to the terminal. For example, it provides audio guides and video links for "how to whisk eggs."

[0152] Input: Information about the suggested menu.

[0153] Output: An audio guide and video link for the cooking procedure are sent to the device.

[0154] Step 5:

[0155] The user enters their health checkup results.

[0156] Operation: The user enters their health check results into a dedicated form. Examples include LDL cholesterol levels and blood sugar levels. The entered data is sent from the terminal to the server.

[0157] Input: Health checkup results.

[0158] Output: The entered health check results are sent from the terminal to the server.

[0159] Step 6:

[0160] The server receives the health check results and saves them to the database.

[0161] Operation: The server receives health check results sent from the terminal, analyzes them, and saves them to the database. The server creates an SQL query and adds the data to the appropriate table.

[0162] Input: Health checkup results sent from the device.

[0163] Output: Health checkup results stored in the database.

[0164] Step 7:

[0165] The server generates dietary recommendations based on health checkup results.

[0166] Operation: The server analyzes stored health check results and generates dietary improvement suggestions based on the user's health status. For example, if the LDL cholesterol level is high, it will suggest menus using ingredients that contain components that lower cholesterol.

[0167] Input: Health checkup results stored in the database.

[0168] Output: The generated dietary suggestions are sent to the terminal.

[0169] Step 8:

[0170] The user prepares meals according to generated suggestions, aiming to improve their health.

[0171] Operation: The user reviews the suggested menu and cooking instructions, then prepares the dish. They listen to audio guides and watch videos while cooking. This process helps maintain a balanced diet and promotes health.

[0172] Input: Suggested menu and cooking instructions.

[0173] Output: Finished dish and improved health.

[0174] (Application Example 1)

[0175] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0176] Traditional recipe suggestion systems focused solely on providing menus that efficiently utilized the ingredients a user already possessed. However, they lacked features to improve dietary habits based on health checkup results and suggest menus containing the most suitable ingredients based on those results. Furthermore, the lack of a means to quickly obtain the suggested menus meant that user convenience was not adequately ensured. To address these issues, a system is needed that includes optimal menu suggestions based on ingredient information and health checkup results, as well as a delivery ordering function.

[0177] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0178] In this invention, the server includes means for the user to input ingredient information, means for generating suggested menus based on the input ingredient information, means for providing cooking procedures corresponding to the generated menus in audio and video, means for the user to input health checkup results, means for generating suggestions for improving dietary habits based on the input health checkup results, and means for suggesting the optimal menu from the ingredient information and health checkup results entered by the user and ordering it for delivery. As a result, the user can easily obtain a menu that allows for proper nutritional intake and can quickly obtain the necessary ingredients and dishes by using a delivery service.

[0179] "Means for users to input food information" refers to an interface for users to input information about the food items in their refrigerator into the system, and is an application used on a smartphone or computer.

[0180] "Means for generating menu suggestions based on input ingredient information" refers to algorithms and databases that search for and suggest available recipes based on ingredient information entered by the user.

[0181] "Means of providing cooking instructions corresponding to the generated menu in audio and video" refers to a function that provides cooking instructions based on the proposed recipe in an easily understandable visual format as audio or video.

[0182] "A means for users to input their health checkup results" refers to an interface for users to input their health checkup results into a system, and this is also an application used on smartphones and computers.

[0183] "Means for generating suggestions to improve dietary habits based on input health check results" refers to algorithms and databases that generate and propose dietary plans appropriate to the user's health condition based on their health check data.

[0184] "A means of suggesting the optimal menu based on the food information and health check results entered by the user, and ordering it for delivery" refers to a function that suggests the optimal menu based on the user's food information and health check results, and allows users to quickly order that menu for delivery.

[0185] Modes for carrying out the invention

[0186] One embodiment of the present invention is a system in which the user inputs information about the ingredients in their refrigerator and their health check results, and based on that information, the system suggests appropriate menus and even supports them in placing delivery orders. The system consists of a user terminal, a central server, and a database.

[0187] User terminal and input means

[0188] A user terminal refers to a device such as a smartphone or computer, which the user uses to input data and receive suggested information. A dedicated application is installed on this terminal, allowing the user to input information about the food items in their refrigerator. For example, if the user inputs "6 eggs, 2 tomatoes, 500ml milk," this data is sent to the server via the interface and stored there.

[0189] Data processing and menu suggestion means

[0190] The central server queries a database based on the received ingredient information and generates an appropriate menu. Specifically, the server refers to the ingredient database and the recipe database to search for recipes that can be made with the ingredients the user has. For example, it can suggest "omelet" or "tomato soup" using "eggs" and "tomatoes." This suggested menu is sent to the user's terminal and displayed to the user.

[0191] Means of providing cooking instructions

[0192] The cooking instructions corresponding to the suggested menu are provided as audio guides and video links. The server generates audio data and video links based on the cooking instructions for the relevant recipe and sends them to the user's terminal. The user can then easily cook by referring to these.

[0193] Input and analysis methods for health checkup results

[0194] Users input their health checkup results, and this data is sent to the server and stored in a database. The server analyzes this health checkup data and generates suggestions for improving the user's diet based on their health status. For example, a user with high LDL cholesterol levels will be suggested a menu that contains many ingredients that lower cholesterol.

[0195] Delivery ordering methods

[0196] Based on the ingredient information and health check results entered by the user, the system not only suggests the most suitable menu but also allows for quick delivery orders. The server displays the suggested menu on the user's terminal, and the user can place a delivery order directly if they wish.

[0197] Hardware and software to be used

[0198] Hardware: Smartphones, computers

[0199] Software: Flask (Python framework), JSON

[0200] Database: Food ingredient information, recipe information, health checkup results

[0201] Specific examples and prompt statements

[0202] For example, if a user enters their health checkup results and provides information such as "high LDL cholesterol levels," the server will suggest a menu based on that information. The suggested menu may include healthy recipes such as "vegetable soup" and "low-fat cooking."

[0203] Examples of prompts to input into a generative AI model:

[0204] text

[0205] The user's health check results indicate high LDL cholesterol. Based on the ingredients in the user's refrigerator, please suggest recipes that include cholesterol-lowering ingredients. The following is a list of the ingredients in the refrigerator.

[0206] Ingredients: 6 eggs, 2 tomatoes, 500ml milk, 1 cabbage

[0207] Thus, the system of the present invention makes it possible to suggest an optimal menu based on the user's food information and health check results, and to order it for delivery.

[0208] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0209] Step 1:

[0210] The user inputs food information. The user uses a smartphone or computer application to input information about the food in their refrigerator (for example, "6 eggs, 2 tomatoes, 500ml milk"). The entered data is sent from the terminal to the server. In this step, the input is the food information entered by the user, and the output is the food information stored on the server.

[0211] Step 2:

[0212] The server receives ingredient information and stores it in the database. The server stores ingredient information received from the terminal in the database. Specifically, each item of ingredient information (ingredient name, quantity, etc.) is stored in the appropriate field in the database. The input for this step is ingredient information sent by the user, and the output is ingredient information stored in the database.

[0213] Step 3:

[0214] The server queries the database to suggest a menu. Based on the stored ingredient information, the server searches the recipe database and retrieves available recipes. Specifically, it selects recipes that contain all the necessary ingredients. The input for this step is ingredient information, and the output is a list of suggested recipes.

[0215] Step 4:

[0216] The server sends the suggested menu to the user's terminal. The server sends the selected recipe as JSON data to the user's terminal. The user can view the suggested menu list on their terminal. The input for this step is the suggested recipe list, and the output is the menu information displayed on the user's terminal.

[0217] Step 5:

[0218] The server generates audio guides and video links for the cooking procedure. Based on the cooking procedure corresponding to the proposed menu, the server generates audio data and links to related videos and sends them to the user's terminal. Specifically, it generates and sends audio files explaining each step and URLs to the corresponding videos. The input for this step is the cooking procedure information for the proposed recipe, and the output is audio data and video links.

[0219] Step 6:

[0220] The user enters their health checkup results. The user enters their health checkup results (e.g., "LDL cholesterol level 120") into the application, and this data is sent to the server. In this step, the input is the health checkup results entered by the user, and the output is the health checkup results stored on the server.

[0221] Step 7:

[0222] The server receives the health check results and saves them to the database. The server saves the health check results received from the terminal to the database. Specifically, it stores each health data item (and its reference value, if any) in the appropriate field in the database. The input for this step is the health check results submitted by the user, and the output is the health data saved in the database.

[0223] Step 8:

[0224] The server generates suggestions for improving dietary habits based on health checkup results. The server analyzes the stored health checkup results and suggests menus and ingredients suitable for the user's health condition. For example, if the LDL cholesterol level is high, recipes using cholesterol-lowering ingredients will be suggested. The input for this step is the health checkup results, and the output is improvement suggestions.

[0225] Step 9:

[0226] The server sends dietary improvement suggestions to the user. The server sends the generated dietary improvement suggestions to the user's terminal. The user can view these suggestions on their terminal. The input for this step is the dietary improvement suggestions, and the output is the suggestion information displayed on the user's terminal.

[0227] Step 10:

[0228] The user places a delivery order. The user reviews the suggested menu and places a delivery order for the desired items. The server receives the user's order information and sends the order details to the partner delivery service. The input for this step is the user's order information, and the output is the order data sent to the delivery service.

[0229] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0230] This invention is a system in which the user inputs data on the ingredients stocked in their refrigerator, and based on that data, it suggests appropriate menus and provides cooking instructions in both audio and video. Furthermore, it is a system that supports health promotion by inputting the user's health checkup results and suggesting a balanced diet based on those results. Moreover, this invention is characterized by its integration of an emotion engine that recognizes the user's emotions, and by its ability to customize suggestions and support based on the user's emotions. Specific embodiments for carrying out this invention are described below.

[0231] Overall system configuration

[0232] The system consists of user terminals, a central server, a database, and an emotion engine.

[0233] User terminal: Refers to devices such as smartphones and PCs, which users use to input data or receive suggested information.

[0234] Central server: This is the central component of a system that processes and manages data and provides users with the information they need.

[0235] Database: Stores data such as ingredient information, recipe information, and health checkup results.

[0236] Emotion Engine: Analyzes the user's facial expressions and voice to recognize emotions and provides that information to the server.

[0237] Program flow

[0238] 1. The user enters the ingredient information.

[0239] Users access the system and log in using their smartphones or PCs. After logging in, they enter the types and quantities of food items in their refrigerator. For example, they might enter "6 eggs, 2 tomatoes, 500ml milk." The entered information is sent from the device to the server and stored in the database.

[0240] 2. The server suggests a menu.

[0241] The server uses the received ingredient information to query the ingredient database and recipe database to find appropriate menus. For example, it might select "omelet" and "tomato soup" using eggs, tomatoes, and milk. The selected menus are sent to the terminal and displayed to the user.

[0242] 3. Audio and video support for cooking instructions

[0243] The server generates audio guides and video links for cooking instructions corresponding to the suggested menu. These are sent to the terminal for the user to use while cooking. For example, the video link might show steps such as "how to whisk eggs."

[0244] 4. The user enters their health checkup results.

[0245] Users input their health checkup results (e.g., LDL cholesterol levels and blood sugar levels) into the system. This information is sent from the terminal to the server and stored in a database.

[0246] 5. Dietary suggestions based on health checkup results

[0247] The server analyzes the stored health check results and generates suggestions for improving the user's diet based on their health status. For example, if the user has high LDL cholesterol levels, menus using cholesterol-lowering ingredients will be suggested. These suggestions are also sent to the device and displayed to the user.

[0248] 6. Recognition of user emotions by an emotion engine

[0249] While the user is using the system, the emotion engine analyzes the user's facial expressions and voice to recognize their emotions. This information is sent to the server in real time.

[0250] 7. Customizing suggestions based on emotions

[0251] The server customizes suggested menus, cooking instructions, audio guides, and video content based on the user's perceived emotions. For example, if the server detects that the user is stressed, it prioritizes suggesting easy-to-make recipes.

[0252] 8. Users create meals and promote health.

[0253] Users aim to improve their health by preparing and consuming appropriate dishes according to the server's suggestions. Users practice cooking while referring to the provided recipes, audio guides, and videos.

[0254] Specific example

[0255] For example, if a user enters "6 eggs, 2 tomatoes, 500ml milk" as the ingredients in their refrigerator, the server will suggest recipes for "omelet" and "tomato soup" using these ingredients. The device will also provide a video and audio guide demonstrating the cooking process. Furthermore, if the user's health check results indicate "high LDL cholesterol levels," the server will advise using ingredients rich in cholesterol-lowering components.

[0256] Furthermore, if the emotion engine detects from the user's facial expressions that they are "tired," the server will suggest a simple and quick recipe. In this way, it is possible to provide support tailored to the user's situation and condition, promoting health management.

[0257] This invention is a system that supports a balanced diet by suggesting appropriate menus based on the ingredients in the user's refrigerator, health check results, and emotions, thereby contributing to the improvement of the user's health.

[0258] The following describes the processing flow.

[0259] Step 1:

[0260] Users access the system using their smartphones or PCs and enter their login information. Upon successful login, the main screen is displayed.

[0261] Step 2:

[0262] The user enters information about the food items stocked in their refrigerator. For example, they might enter the type and quantity of food items such as "6 eggs, 2 tomatoes, 500ml milk."

[0263] Step 3:

[0264] The terminal sends the entered ingredient information to the server via the interface. This transmission is performed using methods such as HTTP requests or API calls.

[0265] Step 4:

[0266] The server saves the received ingredient information to the database. During the saving process, the type and quantity of ingredients are registered in the database as key-value pairs.

[0267] Step 5:

[0268] The server queries the ingredient database and recipe database based on the stored ingredient information. It then uses database queries to search for suitable recipes.

[0269] Step 6:

[0270] The server generates suggested menus from the recipes it finds. For example, it might select "Omelet" and "Tomato Soup" using eggs, tomatoes, and milk.

[0271] Step 7:

[0272] The server sends the generated suggested menu to the terminal. This is also done via HTTP requests or API responses.

[0273] Step 8:

[0274] The terminal displays suggested menus to the user. The user interface shows detailed recipes for "Omelet" and "Tomato Soup."

[0275] Step 9:

[0276] The server generates an audio guide and video link for the cooking procedure and sends it to the terminal. This process provides the audio file and the URL of the video.

[0277] Step 10:

[0278] The device displays the received audio guide and video link to the user. The user can then play the audio guide or watch the video.

[0279] Step 11:

[0280] The user obtains their own health checkup results and inputs them into the system. For example, they might input data such as "LDL cholesterol 150 mg / dL, blood sugar level 110 mg / dL".

[0281] Step 12:

[0282] The terminal sends the input health examination results to the server. This is also done via an HTTP request or an API call.

[0283] Step 13:

[0284] The server saves the received health examination results in the database and evaluates the user's health status. For example, if the LDL cholesterol value is high, an evaluation is made that specific foods should be avoided.

[0285] Step 14:

[0286] The server generates a proposal for improving the diet based on the health status. For users with a high LDL cholesterol value, advice is given to use more tomatoes and other low-cholesterol foods.

[0287] Step 15:

[0288] The server sends the generated proposal for improving the diet to the terminal and displays it to the user. This includes recipes and advice on ingredient selection that take health into consideration.

[0289] Step 16:

[0290] While the user is using the system, the emotion engine analyzes the user's facial expressions and voice to recognize emotions. This information is sent to the server in real time.

[0291] Step 17:

[0292] Based on the user's emotion information recognized by the server, the content of the proposal is customized. For example, if the user is recognized as feeling stressed, recipes that can be easily made are prioritized in the proposal.

[0293] Step 18:

[0294] The server sends emotionally-based, customized suggestions to the device and displays them to the user. This provides information that is tailored to the user's emotions and state of mind.

[0295] Step 19:

[0296] The user creates a dish according to customized suggestions. For example, when the user is tired, they can select a simple recipe to reduce cooking time.

[0297] Step 20:

[0298] Users consume meals they have created to improve their health. The system provides balanced dietary suggestions and emotionally-based customization, enabling effective health management.

[0299] (Example 2)

[0300] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0301] Maintaining an efficient and healthy diet is difficult in today's busy lifestyle. In particular, there is a need for systems that offer menu suggestions utilizing ingredients already in the refrigerator, and suggestions for dietary improvements based on individual health checkup results. Customized suggestions that take user emotions into account are also necessary, but currently, no system exists that can effectively achieve this.

[0302] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0303] In this invention, the server includes means for the user to input food ingredient information, means for generating a menu proposed based on the input food ingredient information, means for providing cooking procedures corresponding to the generated menu in voice and video, means for the user to input a health diagnosis result, means for generating a proposal for improving the diet based on the input health diagnosis result, means for recognizing the user's emotion, and means for customizing the proposed content based on the recognized emotion. As a result, the user can effectively utilize the food ingredients in the refrigerator, consume an appropriate diet based on the health diagnosis result, and furthermore, can receive proposed content according to the emotional state of the day.

[0304] The "user" refers to a person who inputs food ingredient information and health diagnosis results using the system and utilizes the proposed menu and information.

[0305] The "food ingredient information" refers to data on the types and quantities of food ingredients in the refrigerator that the user inputs into the system.

[0306] The "means for generating a menu" refers to an algorithm or function for presenting appropriate recipes and cooking ideas based on the input food ingredient information.

[0307] The "means for providing cooking procedures in voice and video" refers to a function for notifying the user of the cooking method corresponding to the proposed menu through voice guidance and video.

[0308] The "health diagnosis result" refers to information on the user's blood test results and other health conditions that the user inputs into the system.

[0309] The "means for generating a proposal for improving the diet" refers to a function for proposing appropriate diet contents and menus considering the user's health condition based on the input health diagnosis result.

[0310] The "means for recognizing emotion" refers to a technology for analyzing the user's expression and tone of voice and judging the emotional state at that time.

[0311] "Means for customizing suggestions" refers to a function that appropriately changes the content of suggested menus and information according to the recognized emotional state of the user.

[0312] A "database" refers to an information storage system used to accumulate and manage various types of data, such as ingredient information, recipe information, and health checkup results.

[0313] A "terminal" refers to a device that a user uses to input information or receive system suggestions.

[0314] This invention is a system that proposes an efficient and healthy diet to users. Based on ingredient information and health checkup results, the system proposes the optimal menu to the user and provides cooking instructions via voice and video. It is also distinctive in that it recognizes the user's emotions and provides customized suggestions according to the situation.

[0315] The entire system consists of user terminals, a central server, a database, and an emotion engine.

[0316] User terminal

[0317] A user terminal refers to a device such as a smartphone or PC, which the user uses to input food information and health checkup results, and to receive suggested information. This terminal accesses the system using a dedicated application or web browser.

[0318] Central server

[0319] The central server is the core of the system, processing and managing data and providing users with the information they need. The server has the following functions:

[0320] Data reception and storage: Receive food ingredient information and health checkup results submitted by users and store them in the database.

[0321] Menu Suggestion: Based on saved ingredient information, the system cross-references the ingredient database and recipe database to generate appropriate menus.

[0322] Cooking Instruction Generation: Generate links to provide audio and video instructions for cooking the proposed menu.

[0323] Suggestions based on health checkup results: Analyze saved health checkup results and generate appropriate dietary improvement suggestions.

[0324] Emotion recognition data processing: Receive data sent from the emotion engine and customize the suggested content.

[0325] database

[0326] The database stores various data, such as ingredient information, recipe information, and health checkup results. This allows the central server to quickly access the data when needed and provide optimal suggestions to users.

[0327] Emotional Engine

[0328] The emotion engine analyzes facial expressions and voice through the user's device's camera and microphone to recognize the user's emotions. The recognized emotion information is sent to a central server and used to customize the suggested content.

[0329] Specific example

[0330] For example, suppose a user enters "6 eggs, 2 tomatoes, 500ml milk" as the ingredients in their refrigerator. Based on this information, the central server suggests "omelet" and "tomato soup." The suggestions also include audio guides and video links showing the cooking procedure. Furthermore, if the user's health check results indicate "high LDL cholesterol levels," the central server will suggest menus that contain many cholesterol-lowering ingredients. Additionally, if the emotion engine recognizes from the user's facial expression that they are "tired," the central server will prioritize suggesting recipes that are easy to prepare and can be made quickly.

[0331] Example of a prompt

[0332] "Please enter the ingredients in your refrigerator (e.g., 6 eggs, 2 tomatoes, 500ml milk)."

[0333] "Please enter your health checkup results (e.g., LDL cholesterol level, blood sugar level)."

[0334] "Please suggest a dish. The emotion engine recognizes the user's emotions."

[0335] This allows users to make effective use of the ingredients in their refrigerator according to the system's suggestions and eat meals based on their health condition. Furthermore, they can receive optimal suggestions tailored to their emotional state at the time.

[0336] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0337] Step 1:

[0338] The user enters ingredient information.

[0339] Users access the system using their smartphones or PCs and log in by entering their authentication information on the login screen. Upon successful login, a screen appears where the user can enter the types and quantities of food items in their refrigerator. For example, "6 eggs, 2 tomatoes, 500ml milk." The device serializes this food information into JSON format and sends it to the server as an HTTP POST request.

[0340] Input: Information about the contents of the refrigerator (6 eggs, 2 tomatoes, 500ml milk)

[0341] Output: Ingredient information in JSON format sent to the server

[0342] Specific operation: The terminal collects the entered ingredient information, converts it to JSON format, and sends it to the server using the HTTPS protocol.

[0343] Step 2:

[0344] The server suggests a menu.

[0345] Based on the received ingredient information, the server cross-references the ingredient database and recipe database to find the most suitable menu. For example, using "eggs, tomatoes, and milk," it might select "omelet" and "tomato soup." The server then sorts these menus appropriately and sends them.

[0346] Input: Received ingredient information

[0347] Output: Suggestion menu information sent to the user terminal

[0348] Specific operation: The server executes an SQL query to retrieve recipes that match the ingredients, sorts the best recipes using an algorithm, and sends them to the terminal in JSON format.

[0349] Step 3:

[0350] Audio and video support for cooking instructions

[0351] The server generates cooking instructions corresponding to the proposed menu as audio guides and video links. The audio guides are generated using text-to-speech technology, and the video links are URLs to YouTube® or internal storage showing the corresponding cooking instructions. This information is sent to the terminal so that the user can view it.

[0352] Input: Suggested menu information

[0353] Output: Audio guide and video link for cooking instructions sent to the user's device.

[0354] Specific operation: The server calls an API to convert text to speech, selects a video link, and sends it to the terminal along with the audio file.

[0355] Step 4:

[0356] The user enters their health checkup results.

[0357] Users input their health checkup results (e.g., LDL cholesterol levels and blood sugar levels) into the system. The entered data is securely transmitted from the terminal to the server and stored in a database.

[0358] Input: Health checkup results

[0359] Output: Health checkup results sent to the server and stored in the database.

[0360] Specific operation: The terminal serializes the health check results in JSON format and sends them to the server using the HTTPS protocol. The server receives them and stores them in a database.

[0361] Step 5:

[0362] Dietary suggestions based on health checkup results

[0363] The server analyzes stored health check results and generates dietary suggestions based on the user's health status. For example, if the user has high LDL cholesterol levels, it will suggest menus containing many cholesterol-lowering ingredients. These suggestions are also sent to the terminal and displayed to the user.

[0364] Input: Saved health check results

[0365] Output: Dietary improvement suggestions sent to the user's terminal.

[0366] Specific operation: The server executes an algorithm to analyze the health check results, generates suggestions, and sends them to the terminal in JSON format. The terminal displays the suggestions.

[0367] Step 6:

[0368] Recognition of user emotions by an emotion engine

[0369] The device uses its camera and microphone to transmit the user's facial expressions and voice in real time to the emotion engine while the system is in use. The emotion engine analyzes this data to understand the user's current emotional state. This information is then sent to the server.

[0370] Input: Real-time facial and audio data

[0371] Output: Emotion recognition data sent to the server

[0372] Specific operation: The device sends images captured by the camera and audio recorded by the microphone to the emotion engine, and then sends the results of the emotion analysis to the server.

[0373] Step 7:

[0374] Emotion-based customization of suggestions

[0375] The server customizes suggested menus, cooking instructions, audio guides, and video content based on the user's perceived emotions. For example, if the server detects that the user is stressed, it will suggest simple recipes that reduce the effort involved in cooking.

[0376] Input: Emotion recognition data

[0377] Output: Customized suggestions sent to the user terminal

[0378] Specific operation: The server analyzes emotional data, generates optimal suggestions based on the user's situation, and sends them to the terminal. The terminal then displays this information.

[0379] Step 8:

[0380] Users create meals and aim to improve their health.

[0381] Users aim to improve their health by preparing and consuming appropriate dishes according to the server's suggestions. Users practice cooking while referring to the provided recipes, audio guides, and videos.

[0382] Input: Recipe information, audio guide, and video links provided by the server.

[0383] Output: Health improvement obtained by consuming a healthy diet.

[0384] Specific actions: The user cooks and consumes the food based on the guide displayed on their device.

[0385] (Application Example 2)

[0386] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0387] Conventional systems often result in food waste because users are unable to effectively utilize the ingredients in their refrigerators. Furthermore, dietary suggestions aimed at health management are not tailored to the user's emotional needs, sometimes making them burdensome. This leads to food waste and a lack of consistent meal preparation as part of health management. To address these challenges, a system is needed that effectively utilizes ingredients and offers healthy dietary suggestions that resonate with the user's emotional needs.

[0388] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0389] In this invention, the server includes means for the user to input ingredient information, means for generating suggested menus based on the input ingredient information, means for providing cooking procedures corresponding to the generated menus in audio and video, means for the user to input health checkup results, means for generating suggestions for improving dietary habits based on the input health checkup results, and means for recognizing the user's emotions and customizing the suggested content based on those emotions. This enables the server to suggest the optimal menu based on the ingredient information input by the user, as well as to provide customized suggestions according to the user's health condition and emotions.

[0390] "User" refers to a person who uses a service or system.

[0391] "Food ingredient information" refers to data about the types and quantities of food and ingredients owned by the user.

[0392] "Menu" refers to a list of suggested dishes.

[0393] "Cooking instructions" refer to a step-by-step guide that shows the process of preparing a dish.

[0394] "Audio and video" refers to multimedia content that provides users with visual and auditory instructions.

[0395] "Health checkup results" refers to medical data that indicates the user's health status.

[0396] "Dietary habits" refers to the user's daily eating habits and the types of food they consume.

[0397] "Suggestions" refer to recommendations or advice provided to users.

[0398] A "system" refers to a set of devices and software that operate by integrating the above-mentioned means.

[0399] A "database" refers to digital storage that stores information on food ingredients and health check results, and allows them to be retrieved as needed.

[0400] A "terminal" refers to a digital device that a user uses to interact with a system.

[0401] "Emotions" refers to the psychological and sensory states that a user exhibits while using the system.

[0402] "Recognizing emotions" refers to the process of determining a user's psychological state from their facial expressions and voice.

[0403] "Customizing the proposal" refers to adjusting and modifying the content of the proposal based on the user's emotional state.

[0404] Overall system configuration

[0405] This invention is a system in which a user inputs information about the ingredients in their refrigerator and their health check results, and based on that data, it suggests appropriate menus and provides cooking instructions. The system consists of a user terminal, a central server, a database, and an emotion engine.

[0406] User terminal

[0407] User terminals refer to devices such as smartphones, tablets, and PCs. Users use these terminals to input information about the ingredients in their refrigerator and their health check results. The terminals are also used to display suggested menus and cooking instructions (audio and video).

[0408] Central server

[0409] The central server is the core component of the system, receiving and processing food ingredient information and health checkup results submitted by users. Based on the food ingredient information, the server generates optimal menus and provides cooking instructions via audio and video. It also generates suggestions for improving dietary habits based on health checkup results.

[0410] database

[0411] The database is a system that stores user-entered information on ingredients, recipes, and health checkup results. The database contains a large amount of pre-stored recipes and health information, and the server accesses this data to suggest menus and dietary recommendations.

[0412] Emotional Engine

[0413] The emotion engine is a component that recognizes the user's emotions by analyzing their facial expressions and voice. The emotion engine sends the user's emotional information to a central server in real time, and the server customizes the suggested content based on this information.

[0414] Program flow

[0415] 1. The user enters the ingredient information.

[0416] Users access the system and log in using their smartphones or PCs. After logging in, they enter information about the ingredients in their refrigerator. For example, if they enter "500g chicken, 2 onions, 3 tomatoes," that information is sent from their device to the server and stored in the database.

[0417] 2. The server suggests a menu.

[0418] The server uses the received ingredient information to query the ingredient database and recipe database to find appropriate menus. For example, it might select "Chicken Curry" and "Tomato Salad," both using chicken, onions, and tomatoes. The selected menus are sent to the terminal and displayed to the user.

[0419] 3. Audio and video support for cooking instructions

[0420] The server generates audio guides and video links for cooking instructions corresponding to the suggested menu. These are sent to the terminal for the user to use while cooking. For example, the video link might show steps such as "how to finely chop an onion."

[0421] 4. The user enters their health checkup results.

[0422] Users input their health checkup results (e.g., blood pressure and blood sugar levels) into the system. This information is sent from the terminal to the server and stored in a database.

[0423] 5. Dietary suggestions based on health checkup results

[0424] The server analyzes the stored health check results and generates suggestions for improving the user's diet based on their health status. For example, if the user has high blood sugar levels, a menu using low-carbohydrate ingredients will be suggested. This suggestion is also sent to the terminal and displayed to the user.

[0425] 6. Recognition of user emotions by an emotion engine

[0426] While the user is using the system, the emotion engine analyzes the user's facial expressions and voice to recognize their emotions. This information is sent to the server in real time.

[0427] 7. Customizing suggestions based on emotions

[0428] The server customizes suggested menus, cooking instructions, audio guides, and video content based on the user's perceived emotions. For example, if the server detects that the user is tired, it prioritizes suggesting easy-to-make recipes.

[0429] Specific example

[0430] For example, if a user enters "500g chicken, 2 onions, 3 tomatoes" as ingredients in their refrigerator, the server will suggest recipes for "chicken curry" and "tomato salad" using these ingredients. The device will also provide a video and audio guide demonstrating the cooking process. Furthermore, if the user's health check results indicate "high blood sugar," the server will advise using more low-carbohydrate ingredients. Additionally, if the emotion engine detects the user's fatigue from their facial expression, the server will suggest easy and quick recipes. In this way, the system can provide support tailored to the user's situation and condition, promoting better health management.

[0431] Example of a prompt

[0432] For example, the prompt text to be input to the generative AI model would be as follows:

[0433] "If a user's health check results show 'high blood sugar' and their refrigerator contains '500g chicken, 2 onions, and 3 tomatoes,' please list some simple low-carb recipes that can be provided."

[0434] This invention is a system that supports a balanced diet by suggesting appropriate menus based on the ingredients in the user's refrigerator, health check results, and emotions, thereby contributing to the improvement of the user's health.

[0435] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0436] Step 1:

[0437] Input: Users log in to the system using a smartphone or PC and enter information about the ingredients in their refrigerator. For example, they might enter "500g chicken, 2 onions, 3 tomatoes."

[0438] Operation: The ingredient information entered by the user is sent from the terminal to the server.

[0439] Output: The entered ingredient information is saved in the database.

[0440] Step 2:

[0441] Input: The server retrieves the stored ingredient information.

[0442] Operation: The server queries the ingredient database and recipe database to find a suitable menu based on the entered ingredients.

[0443] Output: The searched menu information is sent to the terminal.

[0444] Step 3:

[0445] Input: The terminal displays menu information received from the server.

[0446] Action: The user reviews the suggested menu. For example, recipes for "Chicken Curry" and "Tomato Salad" are displayed.

[0447] Output: Menu information displayed to the user.

[0448] Step 4:

[0449] Input: Menu information selected by the user.

[0450] Operation: The server generates cooking instructions corresponding to the selected menu item and creates audio guides and video links.

[0451] Output: The generated cooking instructions, along with a video link, will be sent to the device.

[0452] Step 5:

[0453] Input: The terminal displays the audio guide and video link received from the server.

[0454] Operation: The user refers to audio guides and video links while cooking. For example, a video on "how to finely chop an onion" is displayed.

[0455] Output: Support for cooking instructions provided to the user.

[0456] Step 6:

[0457] Input: The user enters their health checkup results. For example, they might enter "high blood sugar."

[0458] Operation: The entered health check results are sent from the terminal to the server and stored in the database.

[0459] Output: The entered health check results are saved in the database.

[0460] Step 7:

[0461] Input: The server retrieves the saved health check results.

[0462] Operation: The server analyzes health check results and generates suggestions for improving the user's diet based on their health status. For example, it might advise using low-carbohydrate foods for those with high blood sugar levels.

[0463] Output: The generated dietary improvement suggestions are sent to the terminal.

[0464] Step 8:

[0465] Input: The terminal displays dietary improvement suggestions received from the server.

[0466] Action: The user reviews the suggested diet.

[0467] Output: Suggestions for improving dietary habits provided to the user.

[0468] Step 9:

[0469] Input: User's facial expressions and voice data.

[0470] Operation: The emotion engine analyzes the user's facial expressions and voice to recognize their emotions. Emotional information is sent to the server in real time.

[0471] Output: Recognized emotion information is sent to the server.

[0472] Step 10:

[0473] Input: The server customizes suggestions based on emotional information.

[0474] Operation: If the recognized emotion is determined to be "tired," the server will prioritize suggesting menus that are easy to prepare.

[0475] Output: Customized menu information is sent to the terminal.

[0476] The above is the specific processing flow of this system.

[0477] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0478] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0479] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0480] [Second Embodiment]

[0481] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0482] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0483] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0484] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0485] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0486] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0487] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0488] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0489] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0490] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0491] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0492] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0493] This invention is a system in which the user inputs data on the ingredients stocked in their refrigerator, and based on that data, it suggests appropriate menus and provides cooking instructions via voice and video. Furthermore, it is a system that supports health promotion by inputting the user's health checkup results and suggesting a balanced diet based on those results. Specific embodiments for carrying out this invention are described below.

[0494] Overall system configuration

[0495] The system consists of user terminals, a central server, and a database.

[0496] User terminal: Refers to devices such as smartphones and PCs, which users use to input data or receive suggested information.

[0497] Central server: This is the central component of a system that processes and manages data and provides users with the information they need.

[0498] Database: Stores data such as ingredient information, recipe information, and health checkup results.

[0499] Program flow

[0500] 1. The user enters the ingredient information.

[0501] Users access and log in to the system using their smartphones or PCs. After logging in, they enter the types and quantities of food items in their refrigerator. For example, they might enter "6 eggs, 2 tomatoes, 500ml milk." The entered information is sent from the device to the server and stored in the database.

[0502] 2. The server suggests a menu.

[0503] The server uses the received ingredient information to query the ingredient database and recipe database to find appropriate menus. For example, it might select "omelet" and "tomato soup" using eggs, tomatoes, and milk. The selected menus are sent to the terminal and displayed to the user.

[0504] 3. Audio and video support for cooking instructions

[0505] The server generates audio guides and video links for cooking instructions corresponding to the suggested menu. These are sent to the terminal for the user to use while cooking. For example, the video link might show steps such as "how to whisk eggs."

[0506] 4. The user enters their health checkup results.

[0507] Users input their health checkup results (e.g., LDL cholesterol levels and blood sugar levels) into the system. This information is sent from the terminal to the server and stored in a database.

[0508] 5. Dietary suggestions based on health checkup results

[0509] The server analyzes the stored health check results and generates suggestions for improving the user's diet based on their health status. For example, if the user has high LDL cholesterol levels, menus using cholesterol-lowering ingredients will be suggested. These suggestions are also sent to the device and displayed to the user.

[0510] 6. Users create meals and promote health.

[0511] Users create and consume appropriate meals according to the server's suggestions, thereby promoting their health. This allows them to maintain a balanced diet.

[0512] Specific example

[0513] For example, if a user enters "6 eggs, 2 tomatoes, 500ml milk" as the ingredients in their refrigerator, the server will suggest recipes for "omelet" and "tomato soup" using these ingredients. The device will also provide a video and audio guide demonstrating the cooking process, making it easy for the user to prepare the dishes. Furthermore, if the user's health check results indicate "high LDL cholesterol levels," the server will advise using ingredients rich in cholesterol-lowering components.

[0514] This invention is a system that suggests appropriate menus based on the ingredients in the user's refrigerator and their health checkup results, supporting a balanced diet and contributing to the user's improved health.

[0515] The following describes the processing flow.

[0516] Step 1:

[0517] Users access the system using their smartphones or PCs and enter their login information. Upon successful login, the main screen is displayed.

[0518] Step 2:

[0519] The user enters information about the food items stocked in their refrigerator. For example, they might enter the type and quantity of food items such as "6 eggs, 2 tomatoes, 500ml milk."

[0520] Step 3:

[0521] The terminal sends the entered ingredient information to the server via the interface. This transmission is performed using methods such as HTTP requests or API calls.

[0522] Step 4:

[0523] The server saves the received ingredient information to the database. During the saving process, the type and quantity of ingredients are registered in the database as key-value pairs.

[0524] Step 5:

[0525] The server queries the ingredient database and recipe database based on the stored ingredient information. It then uses database queries to search for suitable recipes.

[0526] Step 6:

[0527] The server generates suggested menus from the recipes it finds. For example, it might select "Omelet" and "Tomato Soup" using eggs, tomatoes, and milk.

[0528] Step 7:

[0529] The server sends the generated suggested menu to the terminal. This is also done via HTTP requests or API responses.

[0530] Step 8:

[0531] The terminal displays suggested menus to the user. The user interface shows detailed recipes for "Omelet" and "Tomato Soup."

[0532] Step 9:

[0533] The server generates an audio guide and video link for the cooking procedure and sends it to the terminal. This process provides the audio file and the URL of the video.

[0534] Step 10:

[0535] The device displays the received audio guide and video link to the user. The user can then play the audio guide or watch the video.

[0536] Step 11:

[0537] The user obtains their own health checkup results and inputs them into the system. For example, they might input data such as "LDL cholesterol 150 mg / dL, blood sugar level 110 mg / dL".

[0538] Step 12:

[0539] The device sends the entered health check results to the server. This is also done via HTTP requests or API calls.

[0540] Step 13:

[0541] The server stores the received health check results in a database and evaluates the user's health status. For example, if the LDL cholesterol level is high, the system may advise avoiding certain foods.

[0542] Step 14:

[0543] The server generates dietary improvement suggestions based on the user's health status. For example, it advises users with high LDL cholesterol levels to use more tomatoes and other low-cholesterol foods.

[0544] Step 15:

[0545] The server sends generated dietary improvement suggestions to the terminal and displays them to the user. These include health-conscious recipes and advice on selecting ingredients.

[0546] Step 16:

[0547] Users prepare meals according to the server's suggestions, aiming to improve their health. Users practice cooking while referring to the provided recipes, audio guides, and videos.

[0548] (Example 1)

[0549] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0550] Current cooking support systems have limited functionality in providing appropriate menus and cooking procedures based on user input of ingredients in their refrigerators. Furthermore, few systems offer personalized dietary suggestions based on health checkup results, making them insufficient for users aiming to improve their health. As a result, users are forced to go through the trouble of searching for information themselves and devising menus that suit their health condition.

[0551] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0552] In this invention, the server includes means for the user to input ingredient information, means for generating suggested menus based on the input ingredient information, means for providing cooking procedures corresponding to the generated menus in audio and video, means for the user to input health checkup results, means for generating suggestions for improving dietary habits based on the input health checkup results, means for a central processing unit to store ingredient information and health checkup results in a database and generate menus and health suggestions based on the stored data, and means for displaying the suggested menus and dietary improvement suggestions on a terminal and providing audio guidance and video. As a result, the user can easily and effectively receive suggestions for appropriate menus based on the ingredients in their refrigerator and their health checkup results, and can cook while confirming the cooking procedures in audio and video. Furthermore, by receiving suggestions for improving dietary habits tailored to their individual health condition, health improvement is promoted.

[0553] A "user" refers to an end user who accesses the system, inputs information, and receives suggested menus and cooking instructions.

[0554] "Ingredient information" refers to information entered by the user regarding the types and quantities of ingredients in the refrigerator.

[0555] "Menu" refers to the recipes and types of dishes suggested based on the ingredient information entered by the user.

[0556] "Cooking instructions" refer to the specific methods for preparing the dishes corresponding to the suggested menu, and these are provided in audio and video formats.

[0557] "Audio guide" refers to audio instructions or guidance provided for users to refer to while cooking.

[0558] A "video link" refers to a link to a video provided to help users visually understand the cooking procedure.

[0559] "Health checkup results" refer to various diagnostic data indicating the user's health status, including blood test results, cholesterol levels, blood sugar levels, and so on.

[0560] A "central processing unit" refers to a server that receives input data from users, performs appropriate processing and data storage based on that data, and generates and provides necessary information.

[0561] A "database" refers to a collection of information used to store and manage information such as food ingredient information, recipe information, and health checkup results.

[0562] "Suggestions" refer to menus and health management methods for improving dietary habits that are generated based on user input data.

[0563] A "terminal" is a device used by a user to access a system, and specifically refers to smartphones and personal computers.

[0564] This invention is a system in which the user inputs data on the ingredients stocked in their refrigerator, and based on that data, it suggests appropriate menus and provides cooking instructions via voice and video. Furthermore, it is a system that supports health promotion by inputting the user's health checkup results and suggesting a balanced diet based on those results. Specific embodiments for carrying out this invention are described below.

[0565] Overall system configuration

[0566] The system consists of user terminals, a central processing unit, and a database.

[0567] User terminal: This refers to devices such as smartphones and personal computers that users use to input data or receive suggested information.

[0568] Central Processing Unit: This is the central component of a system that processes and manages data and provides users with the information they need.

[0569] Database: Stores data such as ingredient information, recipe information, and health checkup results.

[0570] Program processing

[0571] The user enters ingredient information.

[0572] Users access and log in to the system using their smartphones or personal computers. After logging in, they enter the types and quantities of food items in their refrigerator. For example, a user might enter: "6 eggs, 2 tomatoes, 500ml milk." This information is sent from the user's device to the server and stored in the database.

[0573] The server receives ingredient information and stores it in the database.

[0574] The server receives ingredient information sent from the user terminal and performs analysis. The received data is then stored in the database using appropriate SQL queries. In this process, the server specifies the table and field where the data will be stored and places it there.

[0575] The server suggests a menu.

[0576] The server queries the ingredient database and recipe database based on the received ingredient information to find a suitable menu. For example, it might suggest "Omelet with eggs, tomatoes, and milk, and tomato soup." The selected menu is sent to the user's terminal in JSON format and displayed to the user.

[0577] The server prepares and provides audio and video instructions for cooking.

[0578] The server prepares audio guides and video links for cooking instructions corresponding to the proposed menu. These resources are retrieved using existing databases or external APIs. Specifically, it generates video links such as "How to whisk eggs" and sends them to the user's terminal. The user then uses these as a reference to cook.

[0579] The user enters their health checkup results.

[0580] Users enter their health checkup results (e.g., LDL cholesterol levels and blood sugar levels) into a dedicated form. This information is also sent from the user's terminal to the server and stored in the database.

[0581] The server receives the health check results and saves them to the database.

[0582] The server receives health checkup results sent from user terminals, analyzes them, and stores them in a database. Specifically, it adds the health checkup data to the appropriate table and links it to user-specific data.

[0583] The server generates dietary suggestions based on health checkup results.

[0584] The server analyzes the user's health status based on their health checkup results and generates suggestions for appropriate dietary improvements. For example, if the user has high LDL cholesterol levels, the server will suggest menus using ingredients that help lower cholesterol. These suggestions are also sent to the user's terminal in JSON format and displayed to the user.

[0585] Users create meals according to generated suggestions, aiming to improve their health.

[0586] Users prepare dishes by following the menu and cooking instructions received from the server. Specifically, they play the provided audio guide and watch videos while cooking. Through this process, users can maintain a balanced diet and improve their health.

[0587] Specific example

[0588] For example, if a user enters "6 eggs, 2 tomatoes, 500ml milk" as the ingredients in their refrigerator, the server analyzes these ingredients and suggests recipes for "omelet" and "tomato soup." A video link and audio guide demonstrating the cooking process are provided to the user's device, allowing them to easily prepare the dishes. Furthermore, if the user's health check results indicate "high LDL cholesterol levels," the server advises using ingredients rich in cholesterol-lowering components.

[0589] Example of a prompt

[0590] "I have 6 eggs, 2 tomatoes, and 500ml of milk in my refrigerator. Please suggest some recipes using these ingredients. Also, please provide suggestions for what to do if I have high LDL cholesterol levels."

[0591] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0592] Step 1:

[0593] The user enters the ingredient information.

[0594] Operation: Users access the system and log in using a smartphone or personal computer. After logging in, they enter the types and quantities of food items in their refrigerator into a form. For example, "6 eggs, 2 tomatoes, 500ml milk."

[0595] Input: Information about the ingredients in the refrigerator.

[0596] Output: The entered ingredient information is sent from the terminal to the server.

[0597] Step 2:

[0598] The server receives ingredient information and stores it in the database.

[0599] Operation: The server receives ingredient information sent from the terminal, creates an SQL query, and saves it to the database.

[0600] Input: Ingredient information sent from the device.

[0601] Output: Ingredient information stored in the database.

[0602] Step 3:

[0603] The server suggests a menu.

[0604] Operation: The server queries the ingredient database and recipe database based on the stored ingredient information to find a suitable menu. For example, it might select recipes for "omelet" and "tomato soup" using "eggs, tomatoes, and milk."

[0605] Input: Food ingredient information stored in the database.

[0606] Output: The suggested menu is sent to the terminal.

[0607] Step 4:

[0608] The server prepares and provides audio and video instructions for the cooking process.

[0609] Operation: The server generates audio guides and video links for cooking instructions corresponding to the proposed menu and sends them to the terminal. For example, it provides audio guides and video links for "how to whisk eggs."

[0610] Input: Information about the suggested menu.

[0611] Output: An audio guide and video link for the cooking procedure are sent to the device.

[0612] Step 5:

[0613] The user enters their health checkup results.

[0614] Operation: The user enters their health check results into a dedicated form. Examples include LDL cholesterol levels and blood sugar levels. The entered data is sent from the terminal to the server.

[0615] Input: Health checkup results.

[0616] Output: The entered health check results are sent from the terminal to the server.

[0617] Step 6:

[0618] The server receives the health check results and saves them to the database.

[0619] Operation: The server receives health check results sent from the terminal, analyzes them, and saves them to the database. The server creates an SQL query and adds the data to the appropriate table.

[0620] Input: Health checkup results sent from the device.

[0621] Output: Health checkup results stored in the database.

[0622] Step 7:

[0623] The server generates dietary recommendations based on health checkup results.

[0624] Operation: The server analyzes stored health check results and generates dietary improvement suggestions based on the user's health status. For example, if the LDL cholesterol level is high, it will suggest menus using ingredients that contain components that lower cholesterol.

[0625] Input: Health checkup results stored in the database.

[0626] Output: The generated dietary suggestions are sent to the terminal.

[0627] Step 8:

[0628] The user prepares meals according to generated suggestions, aiming to improve their health.

[0629] Operation: The user reviews the suggested menu and cooking instructions, then prepares the dish. They listen to audio guides and watch videos while cooking. This process helps maintain a balanced diet and promotes health.

[0630] Input: Suggested menu and cooking instructions.

[0631] Output: Finished dish and improved health.

[0632] (Application Example 1)

[0633] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0634] Traditional recipe suggestion systems focused solely on providing menus that efficiently utilized the ingredients a user already possessed. However, they lacked features to improve dietary habits based on health checkup results and suggest menus containing the most suitable ingredients based on those results. Furthermore, the lack of a means to quickly obtain the suggested menus meant that user convenience was not adequately ensured. To address these issues, a system is needed that includes optimal menu suggestions based on ingredient information and health checkup results, as well as a delivery ordering function.

[0635] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0636] In this invention, the server includes means for the user to input ingredient information, means for generating suggested menus based on the input ingredient information, means for providing cooking procedures corresponding to the generated menus in audio and video, means for the user to input health checkup results, means for generating suggestions for improving dietary habits based on the input health checkup results, and means for suggesting the optimal menu from the ingredient information and health checkup results entered by the user and ordering it for delivery. As a result, the user can easily obtain a menu that allows for proper nutritional intake and can quickly obtain the necessary ingredients and dishes by using a delivery service.

[0637] "Means for users to input food information" refers to an interface for users to input information about the food items in their refrigerator into the system, and is an application used on a smartphone or computer.

[0638] "Means for generating menu suggestions based on input ingredient information" refers to algorithms and databases that search for and suggest available recipes based on ingredient information entered by the user.

[0639] "Means of providing cooking instructions corresponding to the generated menu in audio and video" refers to a function that provides cooking instructions based on the proposed recipe in an easily understandable visual format as audio or video.

[0640] "A means for users to input their health checkup results" refers to an interface for users to input their health checkup results into a system, and this is also an application used on smartphones and computers.

[0641] "Means for generating suggestions to improve dietary habits based on input health check results" refers to algorithms and databases that generate and propose dietary plans appropriate to the user's health condition based on their health check data.

[0642] "A means of suggesting the optimal menu based on the food information and health check results entered by the user, and ordering it for delivery" refers to a function that suggests the optimal menu based on the user's food information and health check results, and allows users to quickly order that menu for delivery.

[0643] Modes for carrying out the invention

[0644] One embodiment of the present invention is a system in which the user inputs information about the ingredients in their refrigerator and their health check results, and based on that information, the system suggests appropriate menus and even supports them in placing delivery orders. The system consists of a user terminal, a central server, and a database.

[0645] User terminal and input means

[0646] A user terminal refers to a device such as a smartphone or computer, which the user uses to input data and receive suggested information. A dedicated application is installed on this terminal, allowing the user to input information about the food items in their refrigerator. For example, if the user inputs "6 eggs, 2 tomatoes, 500ml milk," this data is sent to the server via the interface and stored there.

[0647] Data processing and menu suggestion means

[0648] The central server queries a database based on the received ingredient information and generates an appropriate menu. Specifically, the server refers to the ingredient database and the recipe database to search for recipes that can be made with the ingredients the user has. For example, it can suggest "omelet" or "tomato soup" using "eggs" and "tomatoes." This suggested menu is sent to the user's terminal and displayed to the user.

[0649] Means of providing cooking instructions

[0650] The cooking instructions corresponding to the suggested menu are provided as audio guides and video links. The server generates audio data and video links based on the cooking instructions for the relevant recipe and sends them to the user's terminal. The user can then easily cook by referring to these.

[0651] Input and analysis methods for health checkup results

[0652] Users input their health checkup results, and this data is sent to the server and stored in a database. The server analyzes this health checkup data and generates suggestions for improving the user's diet based on their health status. For example, a user with high LDL cholesterol levels will be suggested a menu that contains many ingredients that lower cholesterol.

[0653] Delivery ordering methods

[0654] Based on the ingredient information and health check results entered by the user, the system not only suggests the most suitable menu but also allows for quick delivery orders. The server displays the suggested menu on the user's terminal, and the user can place a delivery order directly if they wish.

[0655] Hardware and software to be used

[0656] Hardware: Smartphones, computers

[0657] Software: Flask (Python framework), JSON

[0658] Database: Food ingredient information, recipe information, health checkup results

[0659] Specific examples and prompt statements

[0660] For example, if a user enters their health checkup results and provides information such as "high LDL cholesterol levels," the server will suggest a menu based on that information. The suggested menu may include healthy recipes such as "vegetable soup" and "low-fat cooking."

[0661] Examples of prompts to input into a generative AI model:

[0662] text

[0663] The user's health check results indicate high LDL cholesterol. Based on the ingredients in the user's refrigerator, please suggest recipes that include cholesterol-lowering ingredients. The following is a list of the ingredients in the refrigerator.

[0664] Ingredients: 6 eggs, 2 tomatoes, 500ml milk, 1 cabbage

[0665] Thus, the system of the present invention makes it possible to suggest an optimal menu based on the user's food information and health check results, and to order it for delivery.

[0666] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0667] Step 1:

[0668] The user inputs food information. The user uses a smartphone or computer application to input information about the food in their refrigerator (for example, "6 eggs, 2 tomatoes, 500ml milk"). The entered data is sent from the terminal to the server. In this step, the input is the food information entered by the user, and the output is the food information stored on the server.

[0669] Step 2:

[0670] The server receives ingredient information and stores it in the database. The server stores ingredient information received from the terminal in the database. Specifically, each item of ingredient information (ingredient name, quantity, etc.) is stored in the appropriate field in the database. The input for this step is ingredient information sent by the user, and the output is ingredient information stored in the database.

[0671] Step 3:

[0672] The server queries the database to suggest a menu. Based on the stored ingredient information, the server searches the recipe database and retrieves available recipes. Specifically, it selects recipes that contain all the necessary ingredients. The input for this step is ingredient information, and the output is a list of suggested recipes.

[0673] Step 4:

[0674] The server sends the suggested menu to the user's terminal. The server sends the selected recipe as JSON data to the user's terminal. The user can view the suggested menu list on their terminal. The input for this step is the suggested recipe list, and the output is the menu information displayed on the user's terminal.

[0675] Step 5:

[0676] The server generates audio guides and video links for the cooking procedure. Based on the cooking procedure corresponding to the proposed menu, the server generates audio data and links to related videos and sends them to the user's terminal. Specifically, it generates and sends audio files explaining each step and URLs to the corresponding videos. The input for this step is the cooking procedure information for the proposed recipe, and the output is audio data and video links.

[0677] Step 6:

[0678] The user enters their health checkup results. The user enters their health checkup results (e.g., "LDL cholesterol level 120") into the application, and this data is sent to the server. In this step, the input is the health checkup results entered by the user, and the output is the health checkup results stored on the server.

[0679] Step 7:

[0680] The server receives the health check results and saves them to the database. The server saves the health check results received from the terminal to the database. Specifically, it stores each health data item (and its reference value, if any) in the appropriate field in the database. The input for this step is the health check results submitted by the user, and the output is the health data saved in the database.

[0681] Step 8:

[0682] The server generates suggestions for improving dietary habits based on health checkup results. The server analyzes the stored health checkup results and suggests menus and ingredients suitable for the user's health condition. For example, if the LDL cholesterol level is high, recipes using cholesterol-lowering ingredients will be suggested. The input for this step is the health checkup results, and the output is improvement suggestions.

[0683] Step 9:

[0684] The server sends dietary improvement suggestions to the user. The server sends the generated dietary improvement suggestions to the user's terminal. The user can view these suggestions on their terminal. The input for this step is the dietary improvement suggestions, and the output is the suggestion information displayed on the user's terminal.

[0685] Step 10:

[0686] The user places a delivery order. The user reviews the suggested menu and places a delivery order for the desired items. The server receives the user's order information and sends the order details to the partner delivery service. The input for this step is the user's order information, and the output is the order data sent to the delivery service.

[0687] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0688] This invention is a system in which the user inputs data on the ingredients stocked in their refrigerator, and based on that data, it suggests appropriate menus and provides cooking instructions in both audio and video. Furthermore, it is a system that supports health promotion by inputting the user's health checkup results and suggesting a balanced diet based on those results. Moreover, this invention is characterized by its integration of an emotion engine that recognizes the user's emotions, and by its ability to customize suggestions and support based on the user's emotions. Specific embodiments for carrying out this invention are described below.

[0689] Overall system configuration

[0690] The system consists of user terminals, a central server, a database, and an emotion engine.

[0691] User terminal: Refers to devices such as smartphones and PCs, which users use to input data or receive suggested information.

[0692] Central server: This is the central component of a system that processes and manages data and provides users with the information they need.

[0693] Database: Stores data such as ingredient information, recipe information, and health checkup results.

[0694] Emotion Engine: Analyzes the user's facial expressions and voice to recognize emotions and provides that information to the server.

[0695] Program flow

[0696] 1. The user enters the ingredient information.

[0697] Users access the system and log in using their smartphones or PCs. After logging in, they enter the types and quantities of food items in their refrigerator. For example, they might enter "6 eggs, 2 tomatoes, 500ml milk." The entered information is sent from the device to the server and stored in the database.

[0698] 2. The server suggests a menu.

[0699] The server uses the received ingredient information to query the ingredient database and recipe database to find appropriate menus. For example, it might select "omelet" and "tomato soup" using eggs, tomatoes, and milk. The selected menus are sent to the terminal and displayed to the user.

[0700] 3. Audio and video support for cooking instructions

[0701] The server generates audio guides and video links for cooking instructions corresponding to the suggested menu. These are sent to the terminal for the user to use while cooking. For example, the video link might show steps such as "how to whisk eggs."

[0702] 4. The user enters their health checkup results.

[0703] Users input their health checkup results (e.g., LDL cholesterol levels and blood sugar levels) into the system. This information is sent from the terminal to the server and stored in a database.

[0704] 5. Dietary suggestions based on health checkup results

[0705] The server analyzes the stored health check results and generates suggestions for improving the user's diet based on their health status. For example, if the user has high LDL cholesterol levels, menus using cholesterol-lowering ingredients will be suggested. These suggestions are also sent to the device and displayed to the user.

[0706] 6. Recognition of user emotions by an emotion engine

[0707] While the user is using the system, the emotion engine analyzes the user's facial expressions and voice to recognize their emotions. This information is sent to the server in real time.

[0708] 7. Customizing suggestions based on emotions

[0709] The server customizes suggested menus, cooking instructions, audio guides, and video content based on the user's perceived emotions. For example, if the server detects that the user is stressed, it prioritizes suggesting easy-to-make recipes.

[0710] 8. Users create meals and promote health.

[0711] Users aim to improve their health by preparing and consuming appropriate dishes according to the server's suggestions. Users practice cooking while referring to the provided recipes, audio guides, and videos.

[0712] Specific example

[0713] For example, if a user enters "6 eggs, 2 tomatoes, 500ml milk" as the ingredients in their refrigerator, the server will suggest recipes for "omelet" and "tomato soup" using these ingredients. The device will also provide a video and audio guide demonstrating the cooking process. Furthermore, if the user's health check results indicate "high LDL cholesterol levels," the server will advise using ingredients rich in cholesterol-lowering components.

[0714] Furthermore, if the emotion engine detects from the user's facial expressions that they are "tired," the server will suggest a simple and quick recipe. In this way, it is possible to provide support tailored to the user's situation and condition, promoting health management.

[0715] This invention is a system that supports a balanced diet by suggesting appropriate menus based on the ingredients in the user's refrigerator, health check results, and emotions, thereby contributing to the improvement of the user's health.

[0716] The following describes the processing flow.

[0717] Step 1:

[0718] Users access the system using their smartphones or PCs and enter their login information. Upon successful login, the main screen is displayed.

[0719] Step 2:

[0720] The user enters information about the food items stocked in their refrigerator. For example, they might enter the type and quantity of food items such as "6 eggs, 2 tomatoes, 500ml milk."

[0721] Step 3:

[0722] The terminal sends the entered ingredient information to the server via the interface. This transmission is performed using methods such as HTTP requests or API calls.

[0723] Step 4:

[0724] The server saves the received ingredient information to the database. During the saving process, the type and quantity of ingredients are registered in the database as key-value pairs.

[0725] Step 5:

[0726] The server queries the ingredient database and recipe database based on the stored ingredient information. It then uses database queries to search for suitable recipes.

[0727] Step 6:

[0728] The server generates suggested menus from the recipes it finds. For example, it might select "Omelet" and "Tomato Soup" using eggs, tomatoes, and milk.

[0729] Step 7:

[0730] The server sends the generated suggested menu to the terminal. This is also done via HTTP requests or API responses.

[0731] Step 8:

[0732] The terminal displays suggested menus to the user. The user interface shows detailed recipes for "Omelet" and "Tomato Soup."

[0733] Step 9:

[0734] The server generates an audio guide and video link for the cooking procedure and sends it to the terminal. This process provides the audio file and the URL of the video.

[0735] Step 10:

[0736] The device displays the received audio guide and video link to the user. The user can then play the audio guide or watch the video.

[0737] Step 11:

[0738] The user obtains their own health checkup results and inputs them into the system. For example, they might input data such as "LDL cholesterol 150 mg / dL, blood sugar level 110 mg / dL".

[0739] Step 12:

[0740] The device sends the entered health check results to the server. This is also done via HTTP requests or API calls.

[0741] Step 13:

[0742] The server stores the received health check results in a database and evaluates the user's health status. For example, if the LDL cholesterol level is high, the system may advise avoiding certain foods.

[0743] Step 14:

[0744] The server generates dietary improvement suggestions based on the user's health status. For example, it advises users with high LDL cholesterol levels to use more tomatoes and other low-cholesterol foods.

[0745] Step 15:

[0746] The server sends generated dietary improvement suggestions to the terminal and displays them to the user. These include health-conscious recipes and advice on selecting ingredients.

[0747] Step 16:

[0748] While the user is using the system, the emotion engine analyzes the user's facial expressions and voice to recognize their emotions. This information is sent to the server in real time.

[0749] Step 17:

[0750] The server customizes suggestions based on the user's recognized emotional state. For example, if the server detects that the user is stressed, it will prioritize suggesting easy-to-make recipes.

[0751] Step 18:

[0752] The server sends emotionally-based, customized suggestions to the device and displays them to the user. This provides information that is tailored to the user's emotions and state of mind.

[0753] Step 19:

[0754] The user creates a dish according to customized suggestions. For example, when the user is tired, they can select a simple recipe to reduce cooking time.

[0755] Step 20:

[0756] Users consume meals they have created to improve their health. The system provides balanced dietary suggestions and emotionally-based customization, enabling effective health management.

[0757] (Example 2)

[0758] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0759] Maintaining an efficient and healthy diet is difficult in today's busy lifestyle. In particular, there is a need for systems that offer menu suggestions utilizing ingredients already in the refrigerator, and suggestions for dietary improvements based on individual health checkup results. Customized suggestions that take user emotions into account are also necessary, but currently, no system exists that can effectively achieve this.

[0760] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0761] In this invention, the server includes means for the user to input ingredient information, means for generating suggested menus based on the input ingredient information, means for providing cooking procedures corresponding to the generated menus in audio and video, means for the user to input health checkup results, means for generating suggestions to improve dietary habits based on the input health checkup results, means for recognizing the user's emotions, and means for customizing the suggested content based on the recognized emotions. As a result, the user can effectively utilize the ingredients in their refrigerator, eat appropriate meals based on their health checkup results, and receive suggestions tailored to their emotional state on that day.

[0762] A "user" refers to someone who uses the system to input information about ingredients and health check results, and then utilizes the suggested menus and information.

[0763] "Ingredient information" refers to data on the types and quantities of ingredients in the refrigerator that the user enters into the system.

[0764] "Means for generating menus" refers to algorithms or functions that present appropriate recipes or cooking ideas based on input ingredient information.

[0765] "Means of providing cooking instructions via audio and video" refers to a function that informs the user of the cooking method corresponding to the proposed menu through audio guidance and video.

[0766] "Health checkup results" refers to information that users enter into the system, such as their blood test results and other information indicating their health status.

[0767] "Means for generating suggestions to improve eating habits" refers to a function that suggests appropriate meals and menus that take into account the user's health condition, based on the entered health checkup results.

[0768] "Means of recognizing emotions" refers to technologies that analyze a user's facial expressions, tone of voice, etc., to determine their emotional state at that time.

[0769] "Means for customizing suggestions" refers to a function that appropriately changes the content of suggested menus and information according to the recognized emotional state of the user.

[0770] A "database" refers to an information storage system used to accumulate and manage various types of data, such as ingredient information, recipe information, and health checkup results.

[0771] A "terminal" refers to a device that a user uses to input information or receive system suggestions.

[0772] This invention is a system that proposes an efficient and healthy diet to users. Based on ingredient information and health checkup results, the system proposes the optimal menu to the user and provides cooking instructions via voice and video. It is also distinctive in that it recognizes the user's emotions and provides customized suggestions according to the situation.

[0773] The entire system consists of user terminals, a central server, a database, and an emotion engine.

[0774] User terminal

[0775] A user terminal refers to a device such as a smartphone or PC, which the user uses to input food information and health checkup results, and to receive suggested information. This terminal accesses the system using a dedicated application or web browser.

[0776] Central server

[0777] The central server is the core of the system, processing and managing data and providing users with the information they need. The server has the following functions:

[0778] Data reception and storage: Receive food ingredient information and health checkup results submitted by users and store them in the database.

[0779] Menu Suggestion: Based on saved ingredient information, the system cross-references the ingredient database and recipe database to generate appropriate menus.

[0780] Cooking Instruction Generation: Generate links to provide audio and video instructions for cooking the proposed menu.

[0781] Suggestions based on health checkup results: Analyze saved health checkup results and generate appropriate dietary improvement suggestions.

[0782] Emotion recognition data processing: Receive data sent from the emotion engine and customize the suggested content.

[0783] database

[0784] The database stores various data, such as ingredient information, recipe information, and health checkup results. This allows the central server to quickly access the data when needed and provide optimal suggestions to users.

[0785] Emotional Engine

[0786] The emotion engine analyzes facial expressions and voice through the user's device's camera and microphone to recognize the user's emotions. The recognized emotion information is sent to a central server and used to customize the suggested content.

[0787] Specific example

[0788] For example, suppose a user enters "6 eggs, 2 tomatoes, 500ml milk" as the ingredients in their refrigerator. Based on this information, the central server suggests "omelet" and "tomato soup." The suggestions also include audio guides and video links showing the cooking procedure. Furthermore, if the user's health check results indicate "high LDL cholesterol levels," the central server will suggest menus that contain many cholesterol-lowering ingredients. Additionally, if the emotion engine recognizes from the user's facial expression that they are "tired," the central server will prioritize suggesting recipes that are easy to prepare and can be made quickly.

[0789] Example of a prompt

[0790] "Please enter the ingredients in your refrigerator (e.g., 6 eggs, 2 tomatoes, 500ml milk)."

[0791] "Please enter your health checkup results (e.g., LDL cholesterol level, blood sugar level)."

[0792] "Please suggest a dish. The emotion engine recognizes the user's emotions."

[0793] This allows users to make effective use of the ingredients in their refrigerator according to the system's suggestions and eat meals based on their health condition. Furthermore, they can receive optimal suggestions tailored to their emotional state at the time.

[0794] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0795] Step 1:

[0796] The user enters ingredient information.

[0797] Users access the system using their smartphones or PCs and log in by entering their authentication information on the login screen. Upon successful login, a screen appears where the user can enter the types and quantities of food items in their refrigerator. For example, "6 eggs, 2 tomatoes, 500ml milk." The device serializes this food information into JSON format and sends it to the server as an HTTP POST request.

[0798] Input: Information about the contents of the refrigerator (6 eggs, 2 tomatoes, 500ml milk)

[0799] Output: Ingredient information in JSON format sent to the server

[0800] Specific operation: The terminal collects the entered ingredient information, converts it to JSON format, and sends it to the server using the HTTPS protocol.

[0801] Step 2:

[0802] The server suggests a menu.

[0803] Based on the received ingredient information, the server cross-references the ingredient database and recipe database to find the most suitable menu. For example, using "eggs, tomatoes, and milk," it might select "omelet" and "tomato soup." The server then sorts these menus appropriately and sends them.

[0804] Input: Received ingredient information

[0805] Output: Suggestion menu information sent to the user terminal

[0806] Specific operation: The server executes an SQL query to retrieve recipes that match the ingredients, sorts the best recipes using an algorithm, and sends them to the terminal in JSON format.

[0807] Step 3:

[0808] Audio and video support for cooking instructions

[0809] The server generates cooking instructions corresponding to the proposed menu as audio guides and video links. The audio guides are generated using text-to-speech technology, and the video links are URLs to YouTube or internal storage showing the corresponding cooking instructions. This information is sent to the device, allowing the user to view it.

[0810] Input: Suggested menu information

[0811] Output: Audio guide and video link for cooking instructions sent to the user's device.

[0812] Specific operation: The server calls an API to convert text to speech, selects a video link, and sends it to the terminal along with the audio file.

[0813] Step 4:

[0814] The user enters their health checkup results.

[0815] Users input their health checkup results (e.g., LDL cholesterol levels and blood sugar levels) into the system. The entered data is securely transmitted from the terminal to the server and stored in a database.

[0816] Input: Health checkup results

[0817] Output: Health checkup results sent to the server and stored in the database.

[0818] Specific operation: The terminal serializes the health check results in JSON format and sends them to the server using the HTTPS protocol. The server receives them and stores them in a database.

[0819] Step 5:

[0820] Dietary suggestions based on health checkup results

[0821] The server analyzes stored health check results and generates dietary suggestions based on the user's health status. For example, if the user has high LDL cholesterol levels, it will suggest menus containing many cholesterol-lowering ingredients. These suggestions are also sent to the terminal and displayed to the user.

[0822] Input: Saved health check results

[0823] Output: Dietary improvement suggestions sent to the user's terminal.

[0824] Specific operation: The server executes an algorithm to analyze the health check results, generates suggestions, and sends them to the terminal in JSON format. The terminal displays the suggestions.

[0825] Step 6:

[0826] Recognition of user emotions by an emotion engine

[0827] The device uses its camera and microphone to transmit the user's facial expressions and voice in real time to the emotion engine while the system is in use. The emotion engine analyzes this data to understand the user's current emotional state. This information is then sent to the server.

[0828] Input: Real-time facial and audio data

[0829] Output: Emotion recognition data sent to the server

[0830] Specific operation: The device sends images captured by the camera and audio recorded by the microphone to the emotion engine, and then sends the results of the emotion analysis to the server.

[0831] Step 7:

[0832] Emotion-based customization of suggestions

[0833] The server customizes suggested menus, cooking instructions, audio guides, and video content based on the user's perceived emotions. For example, if the server detects that the user is stressed, it will suggest simple recipes that reduce the effort involved in cooking.

[0834] Input: Emotion recognition data

[0835] Output: Customized suggestions sent to the user terminal

[0836] Specific operation: The server analyzes emotional data, generates optimal suggestions based on the user's situation, and sends them to the terminal. The terminal then displays this information.

[0837] Step 8:

[0838] Users create meals and aim to improve their health.

[0839] Users aim to improve their health by preparing and consuming appropriate dishes according to the server's suggestions. Users practice cooking while referring to the provided recipes, audio guides, and videos.

[0840] Input: Recipe information, audio guide, and video links provided by the server.

[0841] Output: Health improvement obtained by consuming a healthy diet.

[0842] Specific actions: The user cooks and consumes the food based on the guide displayed on their device.

[0843] (Application Example 2)

[0844] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0845] Conventional systems often result in food waste because users are unable to effectively utilize the ingredients in their refrigerators. Furthermore, dietary suggestions aimed at health management are not tailored to the user's emotional needs, sometimes making them burdensome. This leads to food waste and a lack of consistent meal preparation as part of health management. To address these challenges, a system is needed that effectively utilizes ingredients and offers healthy dietary suggestions that resonate with the user's emotional needs.

[0846] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0847] In this invention, the server includes means for the user to input ingredient information, means for generating suggested menus based on the input ingredient information, means for providing cooking procedures corresponding to the generated menus in audio and video, means for the user to input health checkup results, means for generating suggestions for improving dietary habits based on the input health checkup results, and means for recognizing the user's emotions and customizing the suggested content based on those emotions. This enables the server to suggest the optimal menu based on the ingredient information input by the user, as well as to provide customized suggestions according to the user's health condition and emotions.

[0848] "User" refers to a person who uses a service or system.

[0849] "Food ingredient information" refers to data about the types and quantities of food and ingredients owned by the user.

[0850] "Menu" refers to a list of suggested dishes.

[0851] "Cooking instructions" refer to a step-by-step guide that shows the process of preparing a dish.

[0852] "Audio and video" refers to multimedia content that provides users with visual and auditory instructions.

[0853] "Health checkup results" refers to medical data that indicates the user's health status.

[0854] "Dietary habits" refers to the user's daily eating habits and the types of food they consume.

[0855] "Suggestions" refer to recommendations or advice provided to users.

[0856] A "system" refers to a set of devices and software that operate by integrating the above-mentioned means.

[0857] A "database" refers to digital storage that stores information on food ingredients and health check results, and allows them to be retrieved as needed.

[0858] A "terminal" refers to a digital device that a user uses to interact with a system.

[0859] "Emotions" refers to the psychological and sensory states that a user exhibits while using the system.

[0860] "Recognizing emotions" refers to the process of determining a user's psychological state from their facial expressions and voice.

[0861] "Customizing the proposal" refers to adjusting and modifying the content of the proposal based on the user's emotional state.

[0862] Overall system configuration

[0863] This invention is a system in which a user inputs information about the ingredients in their refrigerator and their health check results, and based on that data, it suggests appropriate menus and provides cooking instructions. The system consists of a user terminal, a central server, a database, and an emotion engine.

[0864] User terminal

[0865] User terminals refer to devices such as smartphones, tablets, and PCs. Users use these terminals to input information about the ingredients in their refrigerator and their health check results. The terminals are also used to display suggested menus and cooking instructions (audio and video).

[0866] Central server

[0867] The central server is the core component of the system, receiving and processing food ingredient information and health checkup results submitted by users. Based on the food ingredient information, the server generates optimal menus and provides cooking instructions via audio and video. It also generates suggestions for improving dietary habits based on health checkup results.

[0868] database

[0869] The database is a system that stores user-entered information on ingredients, recipes, and health checkup results. The database contains a large amount of pre-stored recipes and health information, and the server accesses this data to suggest menus and dietary recommendations.

[0870] Emotional Engine

[0871] The emotion engine is a component that recognizes the user's emotions by analyzing their facial expressions and voice. The emotion engine sends the user's emotional information to a central server in real time, and the server customizes the suggested content based on this information.

[0872] Program flow

[0873] 1. The user enters the ingredient information.

[0874] Users access the system and log in using their smartphones or PCs. After logging in, they enter information about the ingredients in their refrigerator. For example, if they enter "500g chicken, 2 onions, 3 tomatoes," that information is sent from their device to the server and stored in the database.

[0875] 2. The server suggests a menu.

[0876] The server uses the received ingredient information to query the ingredient database and recipe database to find appropriate menus. For example, it might select "Chicken Curry" and "Tomato Salad," both using chicken, onions, and tomatoes. The selected menus are sent to the terminal and displayed to the user.

[0877] 3. Audio and video support for cooking instructions

[0878] The server generates audio guides and video links for cooking instructions corresponding to the suggested menu. These are sent to the terminal for the user to use while cooking. For example, the video link might show steps such as "how to finely chop an onion."

[0879] 4. The user enters their health checkup results.

[0880] Users input their health checkup results (e.g., blood pressure and blood sugar levels) into the system. This information is sent from the terminal to the server and stored in a database.

[0881] 5. Dietary suggestions based on health checkup results

[0882] The server analyzes the stored health check results and generates suggestions for improving the user's diet based on their health status. For example, if the user has high blood sugar levels, a menu using low-carbohydrate ingredients will be suggested. This suggestion is also sent to the terminal and displayed to the user.

[0883] 6. Recognition of user emotions by an emotion engine

[0884] While the user is using the system, the emotion engine analyzes the user's facial expressions and voice to recognize their emotions. This information is sent to the server in real time.

[0885] 7. Customizing suggestions based on emotions

[0886] The server customizes suggested menus, cooking instructions, audio guides, and video content based on the user's perceived emotions. For example, if the server detects that the user is tired, it prioritizes suggesting easy-to-make recipes.

[0887] Specific example

[0888] For example, if a user enters "500g chicken, 2 onions, 3 tomatoes" as ingredients in their refrigerator, the server will suggest recipes for "chicken curry" and "tomato salad" using these ingredients. The device will also provide a video and audio guide demonstrating the cooking process. Furthermore, if the user's health check results indicate "high blood sugar," the server will advise using more low-carbohydrate ingredients. Additionally, if the emotion engine detects the user's fatigue from their facial expression, the server will suggest easy and quick recipes. In this way, the system can provide support tailored to the user's situation and condition, promoting better health management.

[0889] Example of a prompt

[0890] For example, the prompt text to be input to the generative AI model would be as follows:

[0891] "If a user's health check results show 'high blood sugar' and their refrigerator contains '500g chicken, 2 onions, and 3 tomatoes,' please list some simple low-carb recipes that can be provided."

[0892] This invention is a system that supports a balanced diet by suggesting appropriate menus based on the ingredients in the user's refrigerator, health check results, and emotions, thereby contributing to the improvement of the user's health.

[0893] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0894] Step 1:

[0895] Input: Users log in to the system using a smartphone or PC and enter information about the ingredients in their refrigerator. For example, they might enter "500g chicken, 2 onions, 3 tomatoes."

[0896] Operation: The ingredient information entered by the user is sent from the terminal to the server.

[0897] Output: The entered ingredient information is saved in the database.

[0898] Step 2:

[0899] Input: The server retrieves the stored ingredient information.

[0900] Operation: The server queries the ingredient database and recipe database to find a suitable menu based on the entered ingredients.

[0901] Output: The searched menu information is sent to the terminal.

[0902] Step 3:

[0903] Input: The terminal displays menu information received from the server.

[0904] Action: The user reviews the suggested menu. For example, recipes for "Chicken Curry" and "Tomato Salad" are displayed.

[0905] Output: Menu information displayed to the user.

[0906] Step 4:

[0907] Input: Menu information selected by the user.

[0908] Operation: The server generates cooking instructions corresponding to the selected menu item and creates audio guides and video links.

[0909] Output: The generated cooking instructions, along with a video link, will be sent to the device.

[0910] Step 5:

[0911] Input: The terminal displays the audio guide and video link received from the server.

[0912] Operation: The user refers to audio guides and video links while cooking. For example, a video on "how to finely chop an onion" is displayed.

[0913] Output: Support for cooking instructions provided to the user.

[0914] Step 6:

[0915] Input: The user enters their health checkup results. For example, they might enter "high blood sugar."

[0916] Operation: The entered health check results are sent from the terminal to the server and stored in the database.

[0917] Output: The entered health check results are saved in the database.

[0918] Step 7:

[0919] Input: The server retrieves the saved health check results.

[0920] Operation: The server analyzes health check results and generates suggestions for improving the user's diet based on their health status. For example, it might advise using low-carbohydrate foods for those with high blood sugar levels.

[0921] Output: The generated dietary improvement suggestions are sent to the terminal.

[0922] Step 8:

[0923] Input: The terminal displays dietary improvement suggestions received from the server.

[0924] Action: The user reviews the suggested diet.

[0925] Output: Suggestions for improving dietary habits provided to the user.

[0926] Step 9:

[0927] Input: User's facial expressions and voice data.

[0928] Operation: The emotion engine analyzes the user's facial expressions and voice to recognize their emotions. Emotional information is sent to the server in real time.

[0929] Output: Recognized emotion information is sent to the server.

[0930] Step 10:

[0931] Input: The server customizes suggestions based on emotional information.

[0932] Operation: If the recognized emotion is determined to be "tired," the server will prioritize suggesting menus that are easy to prepare.

[0933] Output: Customized menu information is sent to the terminal.

[0934] The above is the specific processing flow of this system.

[0935] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0936] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0937] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0938] [Third Embodiment]

[0939] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0940] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0941] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0942] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0943] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0944] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0945] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0946] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0947] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0948] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0949] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0950] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0951] This invention is a system in which the user inputs data on the ingredients stocked in their refrigerator, and based on that data, it suggests appropriate menus and provides cooking instructions via voice and video. Furthermore, it is a system that supports health promotion by inputting the user's health checkup results and suggesting a balanced diet based on those results. Specific embodiments for carrying out this invention are described below.

[0952] Overall system configuration

[0953] The system consists of user terminals, a central server, and a database.

[0954] User terminal: Refers to devices such as smartphones and PCs, which users use to input data or receive suggested information.

[0955] Central server: This is the central component of a system that processes and manages data and provides users with the information they need.

[0956] Database: Stores data such as ingredient information, recipe information, and health checkup results.

[0957] Program flow

[0958] 1. The user enters the ingredient information.

[0959] Users access and log in to the system using their smartphones or PCs. After logging in, they enter the types and quantities of food items in their refrigerator. For example, they might enter "6 eggs, 2 tomatoes, 500ml milk." The entered information is sent from the device to the server and stored in the database.

[0960] 2. The server suggests a menu.

[0961] The server uses the received ingredient information to query the ingredient database and recipe database to find appropriate menus. For example, it might select "omelet" and "tomato soup" using eggs, tomatoes, and milk. The selected menus are sent to the terminal and displayed to the user.

[0962] 3. Audio and video support for cooking instructions

[0963] The server generates audio guides and video links for cooking instructions corresponding to the suggested menu. These are sent to the terminal for the user to use while cooking. For example, the video link might show steps such as "how to whisk eggs."

[0964] 4. The user enters their health checkup results.

[0965] Users input their health checkup results (e.g., LDL cholesterol levels and blood sugar levels) into the system. This information is sent from the terminal to the server and stored in a database.

[0966] 5. Dietary suggestions based on health checkup results

[0967] The server analyzes the stored health check results and generates suggestions for improving the user's diet based on their health status. For example, if the user has high LDL cholesterol levels, menus using cholesterol-lowering ingredients will be suggested. These suggestions are also sent to the device and displayed to the user.

[0968] 6. Users create meals and promote health.

[0969] Users create and consume appropriate meals according to the server's suggestions, thereby promoting their health. This allows them to maintain a balanced diet.

[0970] Specific example

[0971] For example, if a user enters "6 eggs, 2 tomatoes, 500ml milk" as the ingredients in their refrigerator, the server will suggest recipes for "omelet" and "tomato soup" using these ingredients. The device will also provide a video and audio guide demonstrating the cooking process, making it easy for the user to prepare the dishes. Furthermore, if the user's health check results indicate "high LDL cholesterol levels," the server will advise using ingredients rich in cholesterol-lowering components.

[0972] This invention is a system that suggests appropriate menus based on the ingredients in the user's refrigerator and their health checkup results, supporting a balanced diet and contributing to the user's improved health.

[0973] The following describes the processing flow.

[0974] Step 1:

[0975] Users access the system using their smartphones or PCs and enter their login information. Upon successful login, the main screen is displayed.

[0976] Step 2:

[0977] The user enters information about the food items stocked in their refrigerator. For example, they might enter the type and quantity of food items such as "6 eggs, 2 tomatoes, 500ml milk."

[0978] Step 3:

[0979] The terminal sends the entered ingredient information to the server via the interface. This transmission is performed using methods such as HTTP requests or API calls.

[0980] Step 4:

[0981] The server saves the received ingredient information to the database. During the saving process, the type and quantity of ingredients are registered in the database as key-value pairs.

[0982] Step 5:

[0983] The server queries the ingredient database and recipe database based on the stored ingredient information. It then uses database queries to search for suitable recipes.

[0984] Step 6:

[0985] The server generates suggested menus from the recipes it finds. For example, it might select "Omelet" and "Tomato Soup" using eggs, tomatoes, and milk.

[0986] Step 7:

[0987] The server sends the generated suggested menu to the terminal. This is also done via HTTP requests or API responses.

[0988] Step 8:

[0989] The terminal displays suggested menus to the user. The user interface shows detailed recipes for "Omelet" and "Tomato Soup."

[0990] Step 9:

[0991] The server generates an audio guide and video link for the cooking procedure and sends it to the terminal. This process provides the audio file and the URL of the video.

[0992] Step 10:

[0993] The device displays the received audio guide and video link to the user. The user can then play the audio guide or watch the video.

[0994] Step 11:

[0995] The user obtains their own health checkup results and inputs them into the system. For example, they might input data such as "LDL cholesterol 150 mg / dL, blood sugar level 110 mg / dL".

[0996] Step 12:

[0997] The device sends the entered health check results to the server. This is also done via HTTP requests or API calls.

[0998] Step 13:

[0999] The server stores the received health check results in a database and evaluates the user's health status. For example, if the LDL cholesterol level is high, the system may advise avoiding certain foods.

[1000] Step 14:

[1001] The server generates dietary improvement suggestions based on the user's health status. For example, it advises users with high LDL cholesterol levels to use more tomatoes and other low-cholesterol foods.

[1002] Step 15:

[1003] The server sends generated dietary improvement suggestions to the terminal and displays them to the user. These include health-conscious recipes and advice on selecting ingredients.

[1004] Step 16:

[1005] Users prepare meals according to the server's suggestions, aiming to improve their health. Users practice cooking while referring to the provided recipes, audio guides, and videos.

[1006] (Example 1)

[1007] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1008] Current cooking support systems have limited functionality in providing appropriate menus and cooking procedures based on user input of ingredients in their refrigerators. Furthermore, few systems offer personalized dietary suggestions based on health checkup results, making them insufficient for users aiming to improve their health. As a result, users are forced to go through the trouble of searching for information themselves and devising menus that suit their health condition.

[1009] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1010] In this invention, the server includes means for the user to input ingredient information, means for generating suggested menus based on the input ingredient information, means for providing cooking procedures corresponding to the generated menus in audio and video, means for the user to input health checkup results, means for generating suggestions for improving dietary habits based on the input health checkup results, means for a central processing unit to store ingredient information and health checkup results in a database and generate menus and health suggestions based on the stored data, and means for displaying the suggested menus and dietary improvement suggestions on a terminal and providing audio guidance and video. As a result, the user can easily and effectively receive suggestions for appropriate menus based on the ingredients in their refrigerator and their health checkup results, and can cook while confirming the cooking procedures in audio and video. Furthermore, by receiving suggestions for improving dietary habits tailored to their individual health condition, health improvement is promoted.

[1011] A "user" refers to an end user who accesses the system, inputs information, and receives suggested menus and cooking instructions.

[1012] "Ingredient information" refers to information entered by the user regarding the types and quantities of ingredients in the refrigerator.

[1013] "Menu" refers to the recipes and types of dishes suggested based on the ingredient information entered by the user.

[1014] "Cooking instructions" refer to the specific methods for preparing the dishes corresponding to the suggested menu, and these are provided in audio and video formats.

[1015] "Audio guide" refers to audio instructions or guidance provided for users to refer to while cooking.

[1016] A "video link" refers to a link to a video provided to help users visually understand the cooking procedure.

[1017] "Health checkup results" refer to various diagnostic data indicating the user's health status, including blood test results, cholesterol levels, blood sugar levels, and so on.

[1018] A "central processing unit" refers to a server that receives input data from users, performs appropriate processing and data storage based on that data, and generates and provides necessary information.

[1019] A "database" refers to a collection of information used to store and manage information such as food ingredient information, recipe information, and health checkup results.

[1020] "Suggestions" refer to menus and health management methods for improving dietary habits that are generated based on user input data.

[1021] A "terminal" is a device used by a user to access a system, and specifically refers to smartphones and personal computers.

[1022] This invention is a system in which the user inputs data on the ingredients stocked in their refrigerator, and based on that data, it suggests appropriate menus and provides cooking instructions via voice and video. Furthermore, it is a system that supports health promotion by inputting the user's health checkup results and suggesting a balanced diet based on those results. Specific embodiments for carrying out this invention are described below.

[1023] Overall system configuration

[1024] The system consists of user terminals, a central processing unit, and a database.

[1025] User terminal: This refers to devices such as smartphones and personal computers that users use to input data or receive suggested information.

[1026] Central Processing Unit: This is the central component of a system that processes and manages data and provides users with the information they need.

[1027] Database: Stores data such as ingredient information, recipe information, and health checkup results.

[1028] Program processing

[1029] The user enters ingredient information.

[1030] Users access and log in to the system using their smartphones or personal computers. After logging in, they enter the types and quantities of food items in their refrigerator. For example, a user might enter: "6 eggs, 2 tomatoes, 500ml milk." This information is sent from the user's device to the server and stored in the database.

[1031] The server receives ingredient information and stores it in the database.

[1032] The server receives ingredient information sent from the user terminal and performs analysis. The received data is then stored in the database using appropriate SQL queries. In this process, the server specifies the table and field where the data will be stored and places it there.

[1033] The server suggests a menu.

[1034] The server queries the ingredient database and recipe database based on the received ingredient information to find a suitable menu. For example, it might suggest "Omelet with eggs, tomatoes, and milk, and tomato soup." The selected menu is sent to the user's terminal in JSON format and displayed to the user.

[1035] The server prepares and provides audio and video instructions for cooking.

[1036] The server prepares audio guides and video links for cooking instructions corresponding to the proposed menu. These resources are retrieved using existing databases or external APIs. Specifically, it generates video links such as "How to whisk eggs" and sends them to the user's terminal. The user then uses these as a reference to cook.

[1037] The user enters their health checkup results.

[1038] Users enter their health checkup results (e.g., LDL cholesterol levels and blood sugar levels) into a dedicated form. This information is also sent from the user's terminal to the server and stored in the database.

[1039] The server receives the health check results and saves them to the database.

[1040] The server receives health checkup results sent from user terminals, analyzes them, and stores them in a database. Specifically, it adds the health checkup data to the appropriate table and links it to user-specific data.

[1041] The server generates dietary suggestions based on health checkup results.

[1042] The server analyzes the user's health status based on their health checkup results and generates suggestions for appropriate dietary improvements. For example, if the user has high LDL cholesterol levels, the server will suggest menus using ingredients that help lower cholesterol. These suggestions are also sent to the user's terminal in JSON format and displayed to the user.

[1043] Users create meals according to generated suggestions, aiming to improve their health.

[1044] Users prepare dishes by following the menu and cooking instructions received from the server. Specifically, they play the provided audio guide and watch videos while cooking. Through this process, users can maintain a balanced diet and improve their health.

[1045] Specific example

[1046] For example, if a user enters "6 eggs, 2 tomatoes, 500ml milk" as the ingredients in their refrigerator, the server analyzes these ingredients and suggests recipes for "omelet" and "tomato soup." A video link and audio guide demonstrating the cooking process are provided to the user's device, allowing them to easily prepare the dishes. Furthermore, if the user's health check results indicate "high LDL cholesterol levels," the server advises using ingredients rich in cholesterol-lowering components.

[1047] Example of a prompt

[1048] "I have 6 eggs, 2 tomatoes, and 500ml of milk in my refrigerator. Please suggest some recipes using these ingredients. Also, please provide suggestions for what to do if I have high LDL cholesterol levels."

[1049] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1050] Step 1:

[1051] The user enters the ingredient information.

[1052] Operation: Users access the system and log in using a smartphone or personal computer. After logging in, they enter the types and quantities of food items in their refrigerator into a form. For example, "6 eggs, 2 tomatoes, 500ml milk."

[1053] Input: Information about the ingredients in the refrigerator.

[1054] Output: The entered ingredient information is sent from the terminal to the server.

[1055] Step 2:

[1056] The server receives ingredient information and stores it in the database.

[1057] Operation: The server receives ingredient information sent from the terminal, creates an SQL query, and saves it to the database.

[1058] Input: Ingredient information sent from the device.

[1059] Output: Ingredient information stored in the database.

[1060] Step 3:

[1061] The server suggests a menu.

[1062] Operation: The server queries the ingredient database and recipe database based on the stored ingredient information to find a suitable menu. For example, it might select recipes for "omelet" and "tomato soup" using "eggs, tomatoes, and milk."

[1063] Input: Food ingredient information stored in the database.

[1064] Output: The suggested menu is sent to the terminal.

[1065] Step 4:

[1066] The server prepares and provides audio and video instructions for the cooking process.

[1067] Operation: The server generates audio guides and video links for cooking instructions corresponding to the proposed menu and sends them to the terminal. For example, it provides audio guides and video links for "how to whisk eggs."

[1068] Input: Information about the suggested menu.

[1069] Output: An audio guide and video link for the cooking procedure are sent to the device.

[1070] Step 5:

[1071] The user enters their health checkup results.

[1072] Operation: The user enters their health check results into a dedicated form. Examples include LDL cholesterol levels and blood sugar levels. The entered data is sent from the terminal to the server.

[1073] Input: Health checkup results.

[1074] Output: The entered health check results are sent from the terminal to the server.

[1075] Step 6:

[1076] The server receives the health check results and saves them to the database.

[1077] Operation: The server receives health check results sent from the terminal, analyzes them, and saves them to the database. The server creates an SQL query and adds the data to the appropriate table.

[1078] Input: Health checkup results sent from the device.

[1079] Output: Health checkup results stored in the database.

[1080] Step 7:

[1081] The server generates dietary recommendations based on health checkup results.

[1082] Operation: The server analyzes stored health check results and generates dietary improvement suggestions based on the user's health status. For example, if the LDL cholesterol level is high, it will suggest menus using ingredients that contain components that lower cholesterol.

[1083] Input: Health checkup results stored in the database.

[1084] Output: The generated dietary suggestions are sent to the terminal.

[1085] Step 8:

[1086] The user prepares meals according to generated suggestions, aiming to improve their health.

[1087] Operation: The user reviews the suggested menu and cooking instructions, then prepares the dish. They listen to audio guides and watch videos while cooking. This process helps maintain a balanced diet and promotes health.

[1088] Input: Suggested menu and cooking instructions.

[1089] Output: Finished dish and improved health.

[1090] (Application Example 1)

[1091] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1092] Traditional recipe suggestion systems focused solely on providing menus that efficiently utilized the ingredients a user already possessed. However, they lacked features to improve dietary habits based on health checkup results and suggest menus containing the most suitable ingredients based on those results. Furthermore, the lack of a means to quickly obtain the suggested menus meant that user convenience was not adequately ensured. To address these issues, a system is needed that includes optimal menu suggestions based on ingredient information and health checkup results, as well as a delivery ordering function.

[1093] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1094] In this invention, the server includes means for the user to input ingredient information, means for generating suggested menus based on the input ingredient information, means for providing cooking procedures corresponding to the generated menus in audio and video, means for the user to input health checkup results, means for generating suggestions for improving dietary habits based on the input health checkup results, and means for suggesting the optimal menu from the ingredient information and health checkup results entered by the user and ordering it for delivery. As a result, the user can easily obtain a menu that allows for proper nutritional intake and can quickly obtain the necessary ingredients and dishes by using a delivery service.

[1095] "Means for users to input food information" refers to an interface for users to input information about the food items in their refrigerator into the system, and is an application used on a smartphone or computer.

[1096] "Means for generating menu suggestions based on input ingredient information" refers to algorithms and databases that search for and suggest available recipes based on ingredient information entered by the user.

[1097] "Means of providing cooking instructions corresponding to the generated menu in audio and video" refers to a function that provides cooking instructions based on the proposed recipe in an easily understandable visual format as audio or video.

[1098] "A means for users to input their health checkup results" refers to an interface for users to input their health checkup results into a system, and this is also an application used on smartphones and computers.

[1099] "Means for generating suggestions to improve dietary habits based on input health check results" refers to algorithms and databases that generate and propose dietary plans appropriate to the user's health condition based on their health check data.

[1100] "A means of suggesting the optimal menu based on the food information and health check results entered by the user, and ordering it for delivery" refers to a function that suggests the optimal menu based on the user's food information and health check results, and allows users to quickly order that menu for delivery.

[1101] Modes for carrying out the invention

[1102] One embodiment of the present invention is a system in which the user inputs information about the ingredients in their refrigerator and their health check results, and based on that information, the system suggests appropriate menus and even supports them in placing delivery orders. The system consists of a user terminal, a central server, and a database.

[1103] User terminal and input means

[1104] A user terminal refers to a device such as a smartphone or computer, which the user uses to input data and receive suggested information. A dedicated application is installed on this terminal, allowing the user to input information about the food items in their refrigerator. For example, if the user inputs "6 eggs, 2 tomatoes, 500ml milk," this data is sent to the server via the interface and stored there.

[1105] Data processing and menu suggestion means

[1106] The central server queries a database based on the received ingredient information and generates an appropriate menu. Specifically, the server refers to the ingredient database and the recipe database to search for recipes that can be made with the ingredients the user has. For example, it can suggest "omelet" or "tomato soup" using "eggs" and "tomatoes." This suggested menu is sent to the user's terminal and displayed to the user.

[1107] Means of providing cooking instructions

[1108] The cooking instructions corresponding to the suggested menu are provided as audio guides and video links. The server generates audio data and video links based on the cooking instructions for the relevant recipe and sends them to the user's terminal. The user can then easily cook by referring to these.

[1109] Input and analysis methods for health checkup results

[1110] Users input their health checkup results, and this data is sent to the server and stored in a database. The server analyzes this health checkup data and generates suggestions for improving the user's diet based on their health status. For example, a user with high LDL cholesterol levels will be suggested a menu that contains many ingredients that lower cholesterol.

[1111] Delivery ordering methods

[1112] Based on the ingredient information and health check results entered by the user, the system not only suggests the most suitable menu but also allows for quick delivery orders. The server displays the suggested menu on the user's terminal, and the user can place a delivery order directly if they wish.

[1113] Hardware and software to be used

[1114] Hardware: Smartphones, computers

[1115] Software: Flask (Python framework), JSON

[1116] Database: Food ingredient information, recipe information, health checkup results

[1117] Specific examples and prompt statements

[1118] For example, if a user enters their health checkup results and provides information such as "high LDL cholesterol levels," the server will suggest a menu based on that information. The suggested menu may include healthy recipes such as "vegetable soup" and "low-fat cooking."

[1119] Examples of prompts to input into a generative AI model:

[1120] text

[1121] The user's health check results indicate high LDL cholesterol. Based on the ingredients in the user's refrigerator, please suggest recipes that include cholesterol-lowering ingredients. The following is a list of the ingredients in the refrigerator.

[1122] Ingredients: 6 eggs, 2 tomatoes, 500ml milk, 1 cabbage

[1123] Thus, the system of the present invention makes it possible to suggest an optimal menu based on the user's food information and health check results, and to order it for delivery.

[1124] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1125] Step 1:

[1126] The user inputs food information. The user uses a smartphone or computer application to input information about the food in their refrigerator (for example, "6 eggs, 2 tomatoes, 500ml milk"). The entered data is sent from the terminal to the server. In this step, the input is the food information entered by the user, and the output is the food information stored on the server.

[1127] Step 2:

[1128] The server receives ingredient information and stores it in the database. The server stores ingredient information received from the terminal in the database. Specifically, each item of ingredient information (ingredient name, quantity, etc.) is stored in the appropriate field in the database. The input for this step is ingredient information sent by the user, and the output is ingredient information stored in the database.

[1129] Step 3:

[1130] The server queries the database to suggest a menu. Based on the stored ingredient information, the server searches the recipe database and retrieves available recipes. Specifically, it selects recipes that contain all the necessary ingredients. The input for this step is ingredient information, and the output is a list of suggested recipes.

[1131] Step 4:

[1132] The server sends the suggested menu to the user's terminal. The server sends the selected recipe as JSON data to the user's terminal. The user can view the suggested menu list on their terminal. The input for this step is the suggested recipe list, and the output is the menu information displayed on the user's terminal.

[1133] Step 5:

[1134] The server generates audio guides and video links for the cooking procedure. Based on the cooking procedure corresponding to the proposed menu, the server generates audio data and links to related videos and sends them to the user's terminal. Specifically, it generates and sends audio files explaining each step and URLs to the corresponding videos. The input for this step is the cooking procedure information for the proposed recipe, and the output is audio data and video links.

[1135] Step 6:

[1136] The user enters their health checkup results. The user enters their health checkup results (e.g., "LDL cholesterol level 120") into the application, and this data is sent to the server. In this step, the input is the health checkup results entered by the user, and the output is the health checkup results stored on the server.

[1137] Step 7:

[1138] The server receives the health check results and saves them to the database. The server saves the health check results received from the terminal to the database. Specifically, it stores each health data item (and its reference value, if any) in the appropriate field in the database. The input for this step is the health check results submitted by the user, and the output is the health data saved in the database.

[1139] Step 8:

[1140] The server generates suggestions for improving dietary habits based on health checkup results. The server analyzes the stored health checkup results and suggests menus and ingredients suitable for the user's health condition. For example, if the LDL cholesterol level is high, recipes using cholesterol-lowering ingredients will be suggested. The input for this step is the health checkup results, and the output is improvement suggestions.

[1141] Step 9:

[1142] The server sends dietary improvement suggestions to the user. The server sends the generated dietary improvement suggestions to the user's terminal. The user can view these suggestions on their terminal. The input for this step is the dietary improvement suggestions, and the output is the suggestion information displayed on the user's terminal.

[1143] Step 10:

[1144] The user places a delivery order. The user reviews the suggested menu and places a delivery order for the desired items. The server receives the user's order information and sends the order details to the partner delivery service. The input for this step is the user's order information, and the output is the order data sent to the delivery service.

[1145] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1146] This invention is a system in which the user inputs data on the ingredients stocked in their refrigerator, and based on that data, it suggests appropriate menus and provides cooking instructions in both audio and video. Furthermore, it is a system that supports health promotion by inputting the user's health checkup results and suggesting a balanced diet based on those results. Moreover, this invention is characterized by its integration of an emotion engine that recognizes the user's emotions, and by its ability to customize suggestions and support based on the user's emotions. Specific embodiments for carrying out this invention are described below.

[1147] Overall system configuration

[1148] The system consists of user terminals, a central server, a database, and an emotion engine.

[1149] User terminal: Refers to devices such as smartphones and PCs, which users use to input data or receive suggested information.

[1150] Central server: This is the central component of a system that processes and manages data and provides users with the information they need.

[1151] Database: Stores data such as ingredient information, recipe information, and health checkup results.

[1152] Emotion Engine: Analyzes the user's facial expressions and voice to recognize emotions and provides that information to the server.

[1153] Program flow

[1154] 1. The user enters the ingredient information.

[1155] Users access the system and log in using their smartphones or PCs. After logging in, they enter the types and quantities of food items in their refrigerator. For example, they might enter "6 eggs, 2 tomatoes, 500ml milk." The entered information is sent from the device to the server and stored in the database.

[1156] 2. The server suggests a menu.

[1157] The server uses the received ingredient information to query the ingredient database and recipe database to find appropriate menus. For example, it might select "omelet" and "tomato soup" using eggs, tomatoes, and milk. The selected menus are sent to the terminal and displayed to the user.

[1158] 3. Audio and video support for cooking instructions

[1159] The server generates audio guides and video links for cooking instructions corresponding to the suggested menu. These are sent to the terminal for the user to use while cooking. For example, the video link might show steps such as "how to whisk eggs."

[1160] 4. The user enters their health checkup results.

[1161] Users input their health checkup results (e.g., LDL cholesterol levels and blood sugar levels) into the system. This information is sent from the terminal to the server and stored in a database.

[1162] 5. Dietary suggestions based on health checkup results

[1163] The server analyzes the stored health check results and generates suggestions for improving the user's diet based on their health status. For example, if the user has high LDL cholesterol levels, menus using cholesterol-lowering ingredients will be suggested. These suggestions are also sent to the device and displayed to the user.

[1164] 6. Recognition of user emotions by an emotion engine

[1165] While the user is using the system, the emotion engine analyzes the user's facial expressions and voice to recognize their emotions. This information is sent to the server in real time.

[1166] 7. Customizing suggestions based on emotions

[1167] The server customizes suggested menus, cooking instructions, audio guides, and video content based on the user's perceived emotions. For example, if the server detects that the user is stressed, it prioritizes suggesting easy-to-make recipes.

[1168] 8. Users create meals and promote health.

[1169] Users aim to improve their health by preparing and consuming appropriate dishes according to the server's suggestions. Users practice cooking while referring to the provided recipes, audio guides, and videos.

[1170] Specific example

[1171] For example, if a user enters "6 eggs, 2 tomatoes, 500ml milk" as the ingredients in their refrigerator, the server will suggest recipes for "omelet" and "tomato soup" using these ingredients. The device will also provide a video and audio guide demonstrating the cooking process. Furthermore, if the user's health check results indicate "high LDL cholesterol levels," the server will advise using ingredients rich in cholesterol-lowering components.

[1172] Furthermore, if the emotion engine detects from the user's facial expressions that they are "tired," the server will suggest a simple and quick recipe. In this way, it is possible to provide support tailored to the user's situation and condition, promoting health management.

[1173] This invention is a system that supports a balanced diet by suggesting appropriate menus based on the ingredients in the user's refrigerator, health check results, and emotions, thereby contributing to the improvement of the user's health.

[1174] The following describes the processing flow.

[1175] Step 1:

[1176] Users access the system using their smartphones or PCs and enter their login information. Upon successful login, the main screen is displayed.

[1177] Step 2:

[1178] The user enters information about the food items stocked in their refrigerator. For example, they might enter the type and quantity of food items such as "6 eggs, 2 tomatoes, 500ml milk."

[1179] Step 3:

[1180] The terminal sends the entered ingredient information to the server via the interface. This transmission is performed using methods such as HTTP requests or API calls.

[1181] Step 4:

[1182] The server saves the received ingredient information to the database. During the saving process, the type and quantity of ingredients are registered in the database as key-value pairs.

[1183] Step 5:

[1184] The server queries the ingredient database and recipe database based on the stored ingredient information. It then uses database queries to search for suitable recipes.

[1185] Step 6:

[1186] The server generates suggested menus from the recipes it finds. For example, it might select "Omelet" and "Tomato Soup" using eggs, tomatoes, and milk.

[1187] Step 7:

[1188] The server sends the generated suggested menu to the terminal. This is also done via HTTP requests or API responses.

[1189] Step 8:

[1190] The terminal displays suggested menus to the user. The user interface shows detailed recipes for "Omelet" and "Tomato Soup."

[1191] Step 9:

[1192] The server generates an audio guide and video link for the cooking procedure and sends it to the terminal. This process provides the audio file and the URL of the video.

[1193] Step 10:

[1194] The device displays the received audio guide and video link to the user. The user can then play the audio guide or watch the video.

[1195] Step 11:

[1196] The user obtains their own health checkup results and inputs them into the system. For example, they might input data such as "LDL cholesterol 150 mg / dL, blood sugar level 110 mg / dL".

[1197] Step 12:

[1198] The device sends the entered health check results to the server. This is also done via HTTP requests or API calls.

[1199] Step 13:

[1200] The server stores the received health check results in a database and evaluates the user's health status. For example, if the LDL cholesterol level is high, the system may advise avoiding certain foods.

[1201] Step 14:

[1202] The server generates dietary improvement suggestions based on the user's health status. For example, it advises users with high LDL cholesterol levels to use more tomatoes and other low-cholesterol foods.

[1203] Step 15:

[1204] The server sends generated dietary improvement suggestions to the terminal and displays them to the user. These include health-conscious recipes and advice on selecting ingredients.

[1205] Step 16:

[1206] While the user is using the system, the emotion engine analyzes the user's facial expressions and voice to recognize their emotions. This information is sent to the server in real time.

[1207] Step 17:

[1208] The server customizes suggestions based on the user's recognized emotional state. For example, if the server detects that the user is stressed, it will prioritize suggesting easy-to-make recipes.

[1209] Step 18:

[1210] The server sends emotionally-based, customized suggestions to the device and displays them to the user. This provides information that is tailored to the user's emotions and state of mind.

[1211] Step 19:

[1212] The user creates a dish according to customized suggestions. For example, when the user is tired, they can select a simple recipe to reduce cooking time.

[1213] Step 20:

[1214] Users consume meals they have created to improve their health. The system provides balanced dietary suggestions and emotionally-based customization, enabling effective health management.

[1215] (Example 2)

[1216] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1217] Maintaining an efficient and healthy diet is difficult in today's busy lifestyle. In particular, there is a need for systems that offer menu suggestions utilizing ingredients already in the refrigerator, and suggestions for dietary improvements based on individual health checkup results. Customized suggestions that take user emotions into account are also necessary, but currently, no system exists that can effectively achieve this.

[1218] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1219] In this invention, the server includes means for the user to input ingredient information, means for generating suggested menus based on the input ingredient information, means for providing cooking procedures corresponding to the generated menus in audio and video, means for the user to input health checkup results, means for generating suggestions to improve dietary habits based on the input health checkup results, means for recognizing the user's emotions, and means for customizing the suggested content based on the recognized emotions. As a result, the user can effectively utilize the ingredients in their refrigerator, eat appropriate meals based on their health checkup results, and receive suggestions tailored to their emotional state on that day.

[1220] A "user" refers to someone who uses the system to input information about ingredients and health check results, and then utilizes the suggested menus and information.

[1221] "Ingredient information" refers to data on the types and quantities of ingredients in the refrigerator that the user enters into the system.

[1222] "Means for generating menus" refers to algorithms or functions that present appropriate recipes or cooking ideas based on input ingredient information.

[1223] "Means of providing cooking instructions via audio and video" refers to a function that informs the user of the cooking method corresponding to the proposed menu through audio guidance and video.

[1224] "Health checkup results" refers to information that users enter into the system, such as their blood test results and other information indicating their health status.

[1225] "Means for generating suggestions to improve eating habits" refers to a function that suggests appropriate meals and menus that take into account the user's health condition, based on the entered health checkup results.

[1226] "Means of recognizing emotions" refers to technologies that analyze a user's facial expressions, tone of voice, etc., to determine their emotional state at that time.

[1227] "Means for customizing suggestions" refers to a function that appropriately changes the content of suggested menus and information according to the recognized emotional state of the user.

[1228] A "database" refers to an information storage system used to accumulate and manage various types of data, such as ingredient information, recipe information, and health checkup results.

[1229] A "terminal" refers to a device that a user uses to input information or receive system suggestions.

[1230] This invention is a system that proposes an efficient and healthy diet to users. Based on ingredient information and health checkup results, the system proposes the optimal menu to the user and provides cooking instructions via voice and video. It is also distinctive in that it recognizes the user's emotions and provides customized suggestions according to the situation.

[1231] The entire system consists of user terminals, a central server, a database, and an emotion engine.

[1232] User terminal

[1233] A user terminal refers to a device such as a smartphone or PC, which the user uses to input food information and health checkup results, and to receive suggested information. This terminal accesses the system using a dedicated application or web browser.

[1234] Central server

[1235] The central server is the core of the system, processing and managing data and providing users with the information they need. The server has the following functions:

[1236] Data reception and storage: Receive food ingredient information and health checkup results submitted by users and store them in the database.

[1237] Menu Suggestion: Based on saved ingredient information, the system cross-references the ingredient database and recipe database to generate appropriate menus.

[1238] Cooking Instruction Generation: Generate links to provide audio and video instructions for cooking the proposed menu.

[1239] Suggestions based on health checkup results: Analyze saved health checkup results and generate appropriate dietary improvement suggestions.

[1240] Emotion recognition data processing: Receive data sent from the emotion engine and customize the suggested content.

[1241] database

[1242] The database stores various data, such as ingredient information, recipe information, and health checkup results. This allows the central server to quickly access the data when needed and provide optimal suggestions to users.

[1243] Emotional Engine

[1244] The emotion engine analyzes facial expressions and voice through the user's device's camera and microphone to recognize the user's emotions. The recognized emotion information is sent to a central server and used to customize the suggested content.

[1245] Specific example

[1246] For example, suppose a user enters "6 eggs, 2 tomatoes, 500ml milk" as the ingredients in their refrigerator. Based on this information, the central server suggests "omelet" and "tomato soup." The suggestions also include audio guides and video links showing the cooking procedure. Furthermore, if the user's health check results indicate "high LDL cholesterol levels," the central server will suggest menus that contain many cholesterol-lowering ingredients. Additionally, if the emotion engine recognizes from the user's facial expression that they are "tired," the central server will prioritize suggesting recipes that are easy to prepare and can be made quickly.

[1247] Example of a prompt

[1248] "Please enter the ingredients in your refrigerator (e.g., 6 eggs, 2 tomatoes, 500ml milk)."

[1249] "Please enter your health checkup results (e.g., LDL cholesterol level, blood sugar level)."

[1250] "Please suggest a dish. The emotion engine recognizes the user's emotions."

[1251] This allows users to make effective use of the ingredients in their refrigerator according to the system's suggestions and eat meals based on their health condition. Furthermore, they can receive optimal suggestions tailored to their emotional state at the time.

[1252] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1253] Step 1:

[1254] The user enters ingredient information.

[1255] Users access the system using their smartphones or PCs and log in by entering their authentication information on the login screen. Upon successful login, a screen appears where the user can enter the types and quantities of food items in their refrigerator. For example, "6 eggs, 2 tomatoes, 500ml milk." The device serializes this food information into JSON format and sends it to the server as an HTTP POST request.

[1256] Input: Information about the contents of the refrigerator (6 eggs, 2 tomatoes, 500ml milk)

[1257] Output: Ingredient information in JSON format sent to the server

[1258] Specific operation: The terminal collects the entered ingredient information, converts it to JSON format, and sends it to the server using the HTTPS protocol.

[1259] Step 2:

[1260] The server suggests a menu.

[1261] Based on the received ingredient information, the server cross-references the ingredient database and recipe database to find the most suitable menu. For example, using "eggs, tomatoes, and milk," it might select "omelet" and "tomato soup." The server then sorts these menus appropriately and sends them.

[1262] Input: Received ingredient information

[1263] Output: Suggestion menu information sent to the user terminal

[1264] Specific operation: The server executes an SQL query to retrieve recipes that match the ingredients, sorts the best recipes using an algorithm, and sends them to the terminal in JSON format.

[1265] Step 3:

[1266] Audio and video support for cooking instructions

[1267] The server generates cooking instructions corresponding to the proposed menu as audio guides and video links. The audio guides are generated using text-to-speech technology, and the video links are URLs to YouTube or internal storage showing the corresponding cooking instructions. This information is sent to the device, allowing the user to view it.

[1268] Input: Suggested menu information

[1269] Output: Audio guide and video link for cooking instructions sent to the user's device.

[1270] Specific operation: The server calls an API to convert text to speech, selects a video link, and sends it to the terminal along with the audio file.

[1271] Step 4:

[1272] The user enters their health checkup results.

[1273] Users input their health checkup results (e.g., LDL cholesterol levels and blood sugar levels) into the system. The entered data is securely transmitted from the terminal to the server and stored in a database.

[1274] Input: Health checkup results

[1275] Output: Health checkup results sent to the server and stored in the database.

[1276] Specific operation: The terminal serializes the health check results in JSON format and sends them to the server using the HTTPS protocol. The server receives them and stores them in a database.

[1277] Step 5:

[1278] Dietary suggestions based on health checkup results

[1279] The server analyzes stored health check results and generates dietary suggestions based on the user's health status. For example, if the user has high LDL cholesterol levels, it will suggest menus containing many cholesterol-lowering ingredients. These suggestions are also sent to the terminal and displayed to the user.

[1280] Input: Saved health check results

[1281] Output: Dietary improvement suggestions sent to the user's terminal.

[1282] Specific operation: The server executes an algorithm to analyze the health check results, generates suggestions, and sends them to the terminal in JSON format. The terminal displays the suggestions.

[1283] Step 6:

[1284] Recognition of user emotions by an emotion engine

[1285] The device uses its camera and microphone to transmit the user's facial expressions and voice in real time to the emotion engine while the system is in use. The emotion engine analyzes this data to understand the user's current emotional state. This information is then sent to the server.

[1286] Input: Real-time facial and audio data

[1287] Output: Emotion recognition data sent to the server

[1288] Specific operation: The device sends images captured by the camera and audio recorded by the microphone to the emotion engine, and then sends the results of the emotion analysis to the server.

[1289] Step 7:

[1290] Emotion-based customization of suggestions

[1291] The server customizes suggested menus, cooking instructions, audio guides, and video content based on the user's perceived emotions. For example, if the server detects that the user is stressed, it will suggest simple recipes that reduce the effort involved in cooking.

[1292] Input: Emotion recognition data

[1293] Output: Customized suggestions sent to the user terminal

[1294] Specific operation: The server analyzes emotional data, generates optimal suggestions based on the user's situation, and sends them to the terminal. The terminal then displays this information.

[1295] Step 8:

[1296] Users create meals and aim to improve their health.

[1297] Users aim to improve their health by preparing and consuming appropriate dishes according to the server's suggestions. Users practice cooking while referring to the provided recipes, audio guides, and videos.

[1298] Input: Recipe information, audio guide, and video links provided by the server.

[1299] Output: Health improvement obtained by consuming a healthy diet.

[1300] Specific actions: The user cooks and consumes the food based on the guide displayed on their device.

[1301] (Application Example 2)

[1302] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1303] Conventional systems often result in food waste because users are unable to effectively utilize the ingredients in their refrigerators. Furthermore, dietary suggestions aimed at health management are not tailored to the user's emotional needs, sometimes making them burdensome. This leads to food waste and a lack of consistent meal preparation as part of health management. To address these challenges, a system is needed that effectively utilizes ingredients and offers healthy dietary suggestions that resonate with the user's emotional needs.

[1304] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[1305] In this invention, the server includes means for the user to input ingredient information, means for generating suggested menus based on the input ingredient information, means for providing cooking procedures corresponding to the generated menus in audio and video, means for the user to input health checkup results, means for generating suggestions for improving dietary habits based on the input health checkup results, and means for recognizing the user's emotions and customizing the suggested content based on those emotions. This enables the server to suggest the optimal menu based on the ingredient information input by the user, as well as to provide customized suggestions according to the user's health condition and emotions.

[1306] "User" refers to a person who uses a service or system.

[1307] "Food ingredient information" refers to data about the types and quantities of food and ingredients owned by the user.

[1308] "Menu" refers to a list of suggested dishes.

[1309] "Cooking instructions" refer to a step-by-step guide that shows the process of preparing a dish.

[1310] "Audio and video" refers to multimedia content that provides users with visual and auditory instructions.

[1311] "Health checkup results" refers to medical data that indicates the user's health status.

[1312] "Dietary habits" refers to the user's daily eating habits and the types of food they consume.

[1313] "Suggestions" refer to recommendations or advice provided to users.

[1314] A "system" refers to a set of devices and software that operate by integrating the above-mentioned means.

[1315] A "database" refers to digital storage that stores information on food ingredients and health check results, and allows them to be retrieved as needed.

[1316] A "terminal" refers to a digital device that a user uses to interact with a system.

[1317] "Emotions" refers to the psychological and sensory states that a user exhibits while using the system.

[1318] "Recognizing emotions" refers to the process of determining a user's psychological state from their facial expressions and voice.

[1319] "Customizing the proposal" refers to adjusting and modifying the content of the proposal based on the user's emotional state.

[1320] Overall system configuration

[1321] This invention is a system in which a user inputs information about the ingredients in their refrigerator and their health check results, and based on that data, it suggests appropriate menus and provides cooking instructions. The system consists of a user terminal, a central server, a database, and an emotion engine.

[1322] User terminal

[1323] User terminals refer to devices such as smartphones, tablets, and PCs. Users use these terminals to input information about the ingredients in their refrigerator and their health check results. The terminals are also used to display suggested menus and cooking instructions (audio and video).

[1324] Central server

[1325] The central server is the core component of the system, receiving and processing food ingredient information and health checkup results submitted by users. Based on the food ingredient information, the server generates optimal menus and provides cooking instructions via audio and video. It also generates suggestions for improving dietary habits based on health checkup results.

[1326] database

[1327] The database is a system that stores user-entered information on ingredients, recipes, and health checkup results. The database contains a large amount of pre-stored recipes and health information, and the server accesses this data to suggest menus and dietary recommendations.

[1328] Emotional Engine

[1329] The emotion engine is a component that recognizes the user's emotions by analyzing their facial expressions and voice. The emotion engine sends the user's emotional information to a central server in real time, and the server customizes the suggested content based on this information.

[1330] Program flow

[1331] 1. The user enters the ingredient information.

[1332] Users access the system and log in using their smartphones or PCs. After logging in, they enter information about the ingredients in their refrigerator. For example, if they enter "500g chicken, 2 onions, 3 tomatoes," that information is sent from their device to the server and stored in the database.

[1333] 2. The server suggests a menu.

[1334] The server uses the received ingredient information to query the ingredient database and recipe database to find appropriate menus. For example, it might select "Chicken Curry" and "Tomato Salad," both using chicken, onions, and tomatoes. The selected menus are sent to the terminal and displayed to the user.

[1335] 3. Audio and video support for cooking instructions

[1336] The server generates audio guides and video links for cooking instructions corresponding to the suggested menu. These are sent to the terminal for the user to use while cooking. For example, the video link might show steps such as "how to finely chop an onion."

[1337] 4. The user enters their health checkup results.

[1338] Users input their health checkup results (e.g., blood pressure and blood sugar levels) into the system. This information is sent from the terminal to the server and stored in a database.

[1339] 5. Dietary suggestions based on health checkup results

[1340] The server analyzes the stored health check results and generates suggestions for improving the user's diet based on their health status. For example, if the user has high blood sugar levels, a menu using low-carbohydrate ingredients will be suggested. This suggestion is also sent to the terminal and displayed to the user.

[1341] 6. Recognition of user emotions by an emotion engine

[1342] While the user is using the system, the emotion engine analyzes the user's facial expressions and voice to recognize their emotions. This information is sent to the server in real time.

[1343] 7. Customizing suggestions based on emotions

[1344] The server customizes suggested menus, cooking instructions, audio guides, and video content based on the user's perceived emotions. For example, if the server detects that the user is tired, it prioritizes suggesting easy-to-make recipes.

[1345] Specific example

[1346] For example, if a user enters "500g chicken, 2 onions, 3 tomatoes" as ingredients in their refrigerator, the server will suggest recipes for "chicken curry" and "tomato salad" using these ingredients. The device will also provide a video and audio guide demonstrating the cooking process. Furthermore, if the user's health check results indicate "high blood sugar," the server will advise using more low-carbohydrate ingredients. Additionally, if the emotion engine detects the user's fatigue from their facial expression, the server will suggest easy and quick recipes. In this way, the system can provide support tailored to the user's situation and condition, promoting better health management.

[1347] Example of a prompt

[1348] For example, the prompt text to be input to the generative AI model would be as follows:

[1349] "If a user's health check results show 'high blood sugar' and their refrigerator contains '500g chicken, 2 onions, and 3 tomatoes,' please list some simple low-carb recipes that can be provided."

[1350] This invention is a system that supports a balanced diet by suggesting appropriate menus based on the ingredients in the user's refrigerator, health check results, and emotions, thereby contributing to the improvement of the user's health.

[1351] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1352] Step 1:

[1353] Input: Users log in to the system using a smartphone or PC and enter information about the ingredients in their refrigerator. For example, they might enter "500g chicken, 2 onions, 3 tomatoes."

[1354] Operation: The ingredient information entered by the user is sent from the terminal to the server.

[1355] Output: The entered ingredient information is saved in the database.

[1356] Step 2:

[1357] Input: The server retrieves the stored ingredient information.

[1358] Operation: The server queries the ingredient database and recipe database to find a suitable menu based on the entered ingredients.

[1359] Output: The searched menu information is sent to the terminal.

[1360] Step 3:

[1361] Input: The terminal displays menu information received from the server.

[1362] Action: The user reviews the suggested menu. For example, recipes for "Chicken Curry" and "Tomato Salad" are displayed.

[1363] Output: Menu information displayed to the user.

[1364] Step 4:

[1365] Input: Menu information selected by the user.

[1366] Operation: The server generates cooking instructions corresponding to the selected menu item and creates audio guides and video links.

[1367] Output: The generated cooking instructions, along with a video link, will be sent to the device.

[1368] Step 5:

[1369] Input: The terminal displays the audio guide and video link received from the server.

[1370] Operation: The user refers to audio guides and video links while cooking. For example, a video on "how to finely chop an onion" is displayed.

[1371] Output: Support for cooking instructions provided to the user.

[1372] Step 6:

[1373] Input: The user enters their health checkup results. For example, they might enter "high blood sugar."

[1374] Operation: The entered health check results are sent from the terminal to the server and stored in the database.

[1375] Output: The entered health check results are saved in the database.

[1376] Step 7:

[1377] Input: The server retrieves the saved health check results.

[1378] Operation: The server analyzes health check results and generates suggestions for improving the user's diet based on their health status. For example, it might advise using low-carbohydrate foods for those with high blood sugar levels.

[1379] Output: The generated dietary improvement suggestions are sent to the terminal.

[1380] Step 8:

[1381] Input: The terminal displays dietary improvement suggestions received from the server.

[1382] Action: The user reviews the suggested diet.

[1383] Output: Suggestions for improving dietary habits provided to the user.

[1384] Step 9:

[1385] Input: User's facial expressions and voice data.

[1386] Operation: The emotion engine analyzes the user's facial expressions and voice to recognize their emotions. Emotional information is sent to the server in real time.

[1387] Output: Recognized emotion information is sent to the server.

[1388] Step 10:

[1389] Input: The server customizes suggestions based on emotional information.

[1390] Operation: If the recognized emotion is determined to be "tired," the server will prioritize suggesting menus that are easy to prepare.

[1391] Output: Customized menu information is sent to the terminal.

[1392] The above is the specific processing flow of this system.

[1393] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1394] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1395] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[1396] [Fourth Embodiment]

[1397] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[1398] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1399] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1400] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[1401] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1402] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1403] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1404] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[1405] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1406] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1407] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1408] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1409] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1410] This invention is a system in which the user inputs data on the ingredients stocked in their refrigerator, and based on that data, it suggests appropriate menus and provides cooking instructions via voice and video. Furthermore, it is a system that supports health promotion by inputting the user's health checkup results and suggesting a balanced diet based on those results. Specific embodiments for carrying out this invention are described below.

[1411] Overall system configuration

[1412] The system consists of user terminals, a central server, and a database.

[1413] User terminal: Refers to devices such as smartphones and PCs, which users use to input data or receive suggested information.

[1414] Central server: This is the central component of a system that processes and manages data and provides users with the information they need.

[1415] Database: Stores data such as ingredient information, recipe information, and health checkup results.

[1416] Program flow

[1417] 1. The user enters the ingredient information.

[1418] Users access and log in to the system using their smartphones or PCs. After logging in, they enter the types and quantities of food items in their refrigerator. For example, they might enter "6 eggs, 2 tomatoes, 500ml milk." The entered information is sent from the device to the server and stored in the database.

[1419] 2. The server suggests a menu.

[1420] The server uses the received ingredient information to query the ingredient database and recipe database to find appropriate menus. For example, it might select "omelet" and "tomato soup" using eggs, tomatoes, and milk. The selected menus are sent to the terminal and displayed to the user.

[1421] 3. Audio and video support for cooking instructions

[1422] The server generates audio guides and video links for cooking instructions corresponding to the suggested menu. These are sent to the terminal for the user to use while cooking. For example, the video link might show steps such as "how to whisk eggs."

[1423] 4. The user enters their health checkup results.

[1424] Users input their health checkup results (e.g., LDL cholesterol levels and blood sugar levels) into the system. This information is sent from the terminal to the server and stored in a database.

[1425] 5. Dietary suggestions based on health checkup results

[1426] The server analyzes the stored health check results and generates suggestions for improving the user's diet based on their health status. For example, if the user has high LDL cholesterol levels, menus using cholesterol-lowering ingredients will be suggested. These suggestions are also sent to the device and displayed to the user.

[1427] 6. Users create meals and promote health.

[1428] Users create and consume appropriate meals according to the server's suggestions, thereby promoting their health. This allows them to maintain a balanced diet.

[1429] Specific example

[1430] For example, if a user enters "6 eggs, 2 tomatoes, 500ml milk" as the ingredients in their refrigerator, the server will suggest recipes for "omelet" and "tomato soup" using these ingredients. The device will also provide a video and audio guide demonstrating the cooking process, making it easy for the user to prepare the dishes. Furthermore, if the user's health check results indicate "high LDL cholesterol levels," the server will advise using ingredients rich in cholesterol-lowering components.

[1431] This invention is a system that suggests appropriate menus based on the ingredients in the user's refrigerator and their health checkup results, supporting a balanced diet and contributing to the user's improved health.

[1432] The following describes the processing flow.

[1433] Step 1:

[1434] Users access the system using their smartphones or PCs and enter their login information. Upon successful login, the main screen is displayed.

[1435] Step 2:

[1436] The user enters information about the food items stocked in their refrigerator. For example, they might enter the type and quantity of food items such as "6 eggs, 2 tomatoes, 500ml milk."

[1437] Step 3:

[1438] The terminal sends the entered ingredient information to the server via the interface. This transmission is performed using methods such as HTTP requests or API calls.

[1439] Step 4:

[1440] The server saves the received ingredient information to the database. During the saving process, the type and quantity of ingredients are registered in the database as key-value pairs.

[1441] Step 5:

[1442] The server queries the ingredient database and recipe database based on the stored ingredient information. It then uses database queries to search for suitable recipes.

[1443] Step 6:

[1444] The server generates suggested menus from the recipes it finds. For example, it might select "Omelet" and "Tomato Soup" using eggs, tomatoes, and milk.

[1445] Step 7:

[1446] The server sends the generated suggested menu to the terminal. This is also done via HTTP requests or API responses.

[1447] Step 8:

[1448] The terminal displays suggested menus to the user. The user interface shows detailed recipes for "Omelet" and "Tomato Soup."

[1449] Step 9:

[1450] The server generates an audio guide and video link for the cooking procedure and sends it to the terminal. This process provides the audio file and the URL of the video.

[1451] Step 10:

[1452] The device displays the received audio guide and video link to the user. The user can then play the audio guide or watch the video.

[1453] Step 11:

[1454] The user obtains their own health checkup results and inputs them into the system. For example, they might input data such as "LDL cholesterol 150 mg / dL, blood sugar level 110 mg / dL".

[1455] Step 12:

[1456] The device sends the entered health check results to the server. This is also done via HTTP requests or API calls.

[1457] Step 13:

[1458] The server stores the received health check results in a database and evaluates the user's health status. For example, if the LDL cholesterol level is high, the system may advise avoiding certain foods.

[1459] Step 14:

[1460] The server generates dietary improvement suggestions based on the user's health status. For example, it advises users with high LDL cholesterol levels to use more tomatoes and other low-cholesterol foods.

[1461] Step 15:

[1462] The server sends generated dietary improvement suggestions to the terminal and displays them to the user. These include health-conscious recipes and advice on selecting ingredients.

[1463] Step 16:

[1464] Users prepare meals according to the server's suggestions, aiming to improve their health. Users practice cooking while referring to the provided recipes, audio guides, and videos.

[1465] (Example 1)

[1466] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1467] Current cooking support systems have limited functionality in providing appropriate menus and cooking procedures based on user input of ingredients in their refrigerators. Furthermore, few systems offer personalized dietary suggestions based on health checkup results, making them insufficient for users aiming to improve their health. As a result, users are forced to go through the trouble of searching for information themselves and devising menus that suit their health condition.

[1468] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1469] In this invention, the server includes means for the user to input ingredient information, means for generating suggested menus based on the input ingredient information, means for providing cooking procedures corresponding to the generated menus in audio and video, means for the user to input health checkup results, means for generating suggestions for improving dietary habits based on the input health checkup results, means for a central processing unit to store ingredient information and health checkup results in a database and generate menus and health suggestions based on the stored data, and means for displaying the suggested menus and dietary improvement suggestions on a terminal and providing audio guidance and video. As a result, the user can easily and effectively receive suggestions for appropriate menus based on the ingredients in their refrigerator and their health checkup results, and can cook while confirming the cooking procedures in audio and video. Furthermore, by receiving suggestions for improving dietary habits tailored to their individual health condition, health improvement is promoted.

[1470] A "user" refers to an end user who accesses the system, inputs information, and receives suggested menus and cooking instructions.

[1471] "Ingredient information" refers to information entered by the user regarding the types and quantities of ingredients in the refrigerator.

[1472] "Menu" refers to the recipes and types of dishes suggested based on the ingredient information entered by the user.

[1473] "Cooking instructions" refer to the specific methods for preparing the dishes corresponding to the suggested menu, and these are provided in audio and video formats.

[1474] "Audio guide" refers to audio instructions or guidance provided for users to refer to while cooking.

[1475] A "video link" refers to a link to a video provided to help users visually understand the cooking procedure.

[1476] "Health checkup results" refer to various diagnostic data indicating the user's health status, including blood test results, cholesterol levels, blood sugar levels, and so on.

[1477] A "central processing unit" refers to a server that receives input data from users, performs appropriate processing and data storage based on that data, and generates and provides necessary information.

[1478] A "database" refers to a collection of information used to store and manage information such as food ingredient information, recipe information, and health checkup results.

[1479] "Suggestions" refer to menus and health management methods for improving dietary habits that are generated based on user input data.

[1480] A "terminal" is a device used by a user to access a system, and specifically refers to smartphones and personal computers.

[1481] This invention is a system in which the user inputs data on the ingredients stocked in their refrigerator, and based on that data, it suggests appropriate menus and provides cooking instructions via voice and video. Furthermore, it is a system that supports health promotion by inputting the user's health checkup results and suggesting a balanced diet based on those results. Specific embodiments for carrying out this invention are described below.

[1482] Overall system configuration

[1483] The system consists of user terminals, a central processing unit, and a database.

[1484] User terminal: This refers to devices such as smartphones and personal computers that users use to input data or receive suggested information.

[1485] Central Processing Unit: This is the central component of a system that processes and manages data and provides users with the information they need.

[1486] Database: Stores data such as ingredient information, recipe information, and health checkup results.

[1487] Program processing

[1488] The user enters ingredient information.

[1489] Users access and log in to the system using their smartphones or personal computers. After logging in, they enter the types and quantities of food items in their refrigerator. For example, a user might enter: "6 eggs, 2 tomatoes, 500ml milk." This information is sent from the user's device to the server and stored in the database.

[1490] The server receives ingredient information and stores it in the database.

[1491] The server receives ingredient information sent from the user terminal and performs analysis. The received data is then stored in the database using appropriate SQL queries. In this process, the server specifies the table and field where the data will be stored and places it there.

[1492] The server suggests a menu.

[1493] The server queries the ingredient database and recipe database based on the received ingredient information to find a suitable menu. For example, it might suggest "Omelet with eggs, tomatoes, and milk, and tomato soup." The selected menu is sent to the user's terminal in JSON format and displayed to the user.

[1494] The server prepares and provides audio and video instructions for cooking.

[1495] The server prepares audio guides and video links for cooking instructions corresponding to the proposed menu. These resources are retrieved using existing databases or external APIs. Specifically, it generates video links such as "How to whisk eggs" and sends them to the user's terminal. The user then uses these as a reference to cook.

[1496] The user enters their health checkup results.

[1497] Users enter their health checkup results (e.g., LDL cholesterol levels and blood sugar levels) into a dedicated form. This information is also sent from the user's terminal to the server and stored in the database.

[1498] The server receives the health check results and saves them to the database.

[1499] The server receives health checkup results sent from user terminals, analyzes them, and stores them in a database. Specifically, it adds the health checkup data to the appropriate table and links it to user-specific data.

[1500] The server generates dietary suggestions based on health checkup results.

[1501] The server analyzes the user's health status based on their health checkup results and generates suggestions for appropriate dietary improvements. For example, if the user has high LDL cholesterol levels, the server will suggest menus using ingredients that help lower cholesterol. These suggestions are also sent to the user's terminal in JSON format and displayed to the user.

[1502] Users create meals according to generated suggestions, aiming to improve their health.

[1503] Users prepare dishes by following the menu and cooking instructions received from the server. Specifically, they play the provided audio guide and watch videos while cooking. Through this process, users can maintain a balanced diet and improve their health.

[1504] Specific example

[1505] For example, if a user enters "6 eggs, 2 tomatoes, 500ml milk" as the ingredients in their refrigerator, the server analyzes these ingredients and suggests recipes for "omelet" and "tomato soup." A video link and audio guide demonstrating the cooking process are provided to the user's device, allowing them to easily prepare the dishes. Furthermore, if the user's health check results indicate "high LDL cholesterol levels," the server advises using ingredients rich in cholesterol-lowering components.

[1506] Example of a prompt

[1507] "I have 6 eggs, 2 tomatoes, and 500ml of milk in my refrigerator. Please suggest some recipes using these ingredients. Also, please provide suggestions for what to do if I have high LDL cholesterol levels."

[1508] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1509] Step 1:

[1510] The user enters the ingredient information.

[1511] Operation: Users access the system and log in using a smartphone or personal computer. After logging in, they enter the types and quantities of food items in their refrigerator into a form. For example, "6 eggs, 2 tomatoes, 500ml milk."

[1512] Input: Information about the ingredients in the refrigerator.

[1513] Output: The entered ingredient information is sent from the terminal to the server.

[1514] Step 2:

[1515] The server receives ingredient information and stores it in the database.

[1516] Operation: The server receives ingredient information sent from the terminal, creates an SQL query, and saves it to the database.

[1517] Input: Ingredient information sent from the device.

[1518] Output: Ingredient information stored in the database.

[1519] Step 3:

[1520] The server suggests a menu.

[1521] Operation: The server queries the ingredient database and recipe database based on the stored ingredient information to find a suitable menu. For example, it might select recipes for "omelet" and "tomato soup" using "eggs, tomatoes, and milk."

[1522] Input: Food ingredient information stored in the database.

[1523] Output: The suggested menu is sent to the terminal.

[1524] Step 4:

[1525] The server prepares and provides audio and video instructions for the cooking process.

[1526] Operation: The server generates audio guides and video links for cooking instructions corresponding to the proposed menu and sends them to the terminal. For example, it provides audio guides and video links for "how to whisk eggs."

[1527] Input: Information about the suggested menu.

[1528] Output: An audio guide and video link for the cooking procedure are sent to the device.

[1529] Step 5:

[1530] The user enters their health checkup results.

[1531] Operation: The user enters their health check results into a dedicated form. Examples include LDL cholesterol levels and blood sugar levels. The entered data is sent from the terminal to the server.

[1532] Input: Health checkup results.

[1533] Output: The entered health check results are sent from the terminal to the server.

[1534] Step 6:

[1535] The server receives the health check results and saves them to the database.

[1536] Operation: The server receives health check results sent from the terminal, analyzes them, and saves them to the database. The server creates an SQL query and adds the data to the appropriate table.

[1537] Input: Health checkup results sent from the device.

[1538] Output: Health checkup results stored in the database.

[1539] Step 7:

[1540] The server generates dietary recommendations based on health checkup results.

[1541] Operation: The server analyzes stored health check results and generates dietary improvement suggestions based on the user's health status. For example, if the LDL cholesterol level is high, it will suggest menus using ingredients that contain components that lower cholesterol.

[1542] Input: Health checkup results stored in the database.

[1543] Output: The generated dietary suggestions are sent to the terminal.

[1544] Step 8:

[1545] The user prepares meals according to generated suggestions, aiming to improve their health.

[1546] Operation: The user reviews the suggested menu and cooking instructions, then prepares the dish. They listen to audio guides and watch videos while cooking. This process helps maintain a balanced diet and promotes health.

[1547] Input: Suggested menu and cooking instructions.

[1548] Output: Finished dish and improved health.

[1549] (Application Example 1)

[1550] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1551] Traditional recipe suggestion systems focused solely on providing menus that efficiently utilized the ingredients a user already possessed. However, they lacked features to improve dietary habits based on health checkup results and suggest menus containing the most suitable ingredients based on those results. Furthermore, the lack of a means to quickly obtain the suggested menus meant that user convenience was not adequately ensured. To address these issues, a system is needed that includes optimal menu suggestions based on ingredient information and health checkup results, as well as a delivery ordering function.

[1552] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1553] In this invention, the server includes means for the user to input ingredient information, means for generating suggested menus based on the input ingredient information, means for providing cooking procedures corresponding to the generated menus in audio and video, means for the user to input health checkup results, means for generating suggestions for improving dietary habits based on the input health checkup results, and means for suggesting the optimal menu from the ingredient information and health checkup results entered by the user and ordering it for delivery. As a result, the user can easily obtain a menu that allows for proper nutritional intake and can quickly obtain the necessary ingredients and dishes by using a delivery service.

[1554] "Means for users to input food information" refers to an interface for users to input information about the food items in their refrigerator into the system, and is an application used on a smartphone or computer.

[1555] "Means for generating menu suggestions based on input ingredient information" refers to algorithms and databases that search for and suggest available recipes based on ingredient information entered by the user.

[1556] "Means of providing cooking instructions corresponding to the generated menu in audio and video" refers to a function that provides cooking instructions based on the proposed recipe in an easily understandable visual format as audio or video.

[1557] "A means for users to input their health checkup results" refers to an interface for users to input their health checkup results into a system, and this is also an application used on smartphones and computers.

[1558] "Means for generating suggestions to improve dietary habits based on input health check results" refers to algorithms and databases that generate and propose dietary plans appropriate to the user's health condition based on their health check data.

[1559] "A means of suggesting the optimal menu based on the food information and health check results entered by the user, and ordering it for delivery" refers to a function that suggests the optimal menu based on the user's food information and health check results, and allows users to quickly order that menu for delivery.

[1560] Modes for carrying out the invention

[1561] One embodiment of the present invention is a system in which the user inputs information about the ingredients in their refrigerator and their health check results, and based on that information, the system suggests appropriate menus and even supports them in placing delivery orders. The system consists of a user terminal, a central server, and a database.

[1562] User terminal and input means

[1563] A user terminal refers to a device such as a smartphone or computer, which the user uses to input data and receive suggested information. A dedicated application is installed on this terminal, allowing the user to input information about the food items in their refrigerator. For example, if the user inputs "6 eggs, 2 tomatoes, 500ml milk," this data is sent to the server via the interface and stored there.

[1564] Data processing and menu suggestion means

[1565] The central server queries a database based on the received ingredient information and generates an appropriate menu. Specifically, the server refers to the ingredient database and the recipe database to search for recipes that can be made with the ingredients the user has. For example, it can suggest "omelet" or "tomato soup" using "eggs" and "tomatoes." This suggested menu is sent to the user's terminal and displayed to the user.

[1566] Means of providing cooking instructions

[1567] The cooking instructions corresponding to the suggested menu are provided as audio guides and video links. The server generates audio data and video links based on the cooking instructions for the relevant recipe and sends them to the user's terminal. The user can then easily cook by referring to these.

[1568] Input and analysis methods for health checkup results

[1569] Users input their health checkup results, and this data is sent to the server and stored in a database. The server analyzes this health checkup data and generates suggestions for improving the user's diet based on their health status. For example, a user with high LDL cholesterol levels will be suggested a menu that contains many ingredients that lower cholesterol.

[1570] Delivery ordering methods

[1571] Based on the ingredient information and health check results entered by the user, the system not only suggests the most suitable menu but also allows for quick delivery orders. The server displays the suggested menu on the user's terminal, and the user can place a delivery order directly if they wish.

[1572] Hardware and software to be used

[1573] Hardware: Smartphones, computers

[1574] Software: Flask (Python framework), JSON

[1575] Database: Food ingredient information, recipe information, health checkup results

[1576] Specific examples and prompt statements

[1577] For example, if a user enters their health checkup results and provides information such as "high LDL cholesterol levels," the server will suggest a menu based on that information. The suggested menu may include healthy recipes such as "vegetable soup" and "low-fat cooking."

[1578] Examples of prompts to input into a generative AI model:

[1579] text

[1580] The user's health check results indicate high LDL cholesterol. Based on the ingredients in the user's refrigerator, please suggest recipes that include cholesterol-lowering ingredients. The following is a list of the ingredients in the refrigerator.

[1581] Ingredients: 6 eggs, 2 tomatoes, 500ml milk, 1 cabbage

[1582] Thus, the system of the present invention makes it possible to suggest an optimal menu based on the user's food information and health check results, and to order it for delivery.

[1583] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1584] Step 1:

[1585] The user inputs food information. The user uses a smartphone or computer application to input information about the food in their refrigerator (for example, "6 eggs, 2 tomatoes, 500ml milk"). The entered data is sent from the terminal to the server. In this step, the input is the food information entered by the user, and the output is the food information stored on the server.

[1586] Step 2:

[1587] The server receives ingredient information and stores it in the database. The server stores ingredient information received from the terminal in the database. Specifically, each item of ingredient information (ingredient name, quantity, etc.) is stored in the appropriate field in the database. The input for this step is ingredient information sent by the user, and the output is ingredient information stored in the database.

[1588] Step 3:

[1589] The server queries the database to suggest a menu. Based on the stored ingredient information, the server searches the recipe database and retrieves available recipes. Specifically, it selects recipes that contain all the necessary ingredients. The input for this step is ingredient information, and the output is a list of suggested recipes.

[1590] Step 4:

[1591] The server sends the suggested menu to the user's terminal. The server sends the selected recipe as JSON data to the user's terminal. The user can view the suggested menu list on their terminal. The input for this step is the suggested recipe list, and the output is the menu information displayed on the user's terminal.

[1592] Step 5:

[1593] The server generates audio guides and video links for the cooking procedure. Based on the cooking procedure corresponding to the proposed menu, the server generates audio data and links to related videos and sends them to the user's terminal. Specifically, it generates and sends audio files explaining each step and URLs to the corresponding videos. The input for this step is the cooking procedure information for the proposed recipe, and the output is audio data and video links.

[1594] Step 6:

[1595] The user enters their health checkup results. The user enters their health checkup results (e.g., "LDL cholesterol level 120") into the application, and this data is sent to the server. In this step, the input is the health checkup results entered by the user, and the output is the health checkup results stored on the server.

[1596] Step 7:

[1597] The server receives the health check results and saves them to the database. The server saves the health check results received from the terminal to the database. Specifically, it stores each health data item (and its reference value, if any) in the appropriate field in the database. The input for this step is the health check results submitted by the user, and the output is the health data saved in the database.

[1598] Step 8:

[1599] The server generates suggestions for improving dietary habits based on health checkup results. The server analyzes the stored health checkup results and suggests menus and ingredients suitable for the user's health condition. For example, if the LDL cholesterol level is high, recipes using cholesterol-lowering ingredients will be suggested. The input for this step is the health checkup results, and the output is improvement suggestions.

[1600] Step 9:

[1601] The server sends dietary improvement suggestions to the user. The server sends the generated dietary improvement suggestions to the user's terminal. The user can view these suggestions on their terminal. The input for this step is the dietary improvement suggestions, and the output is the suggestion information displayed on the user's terminal.

[1602] Step 10:

[1603] The user places a delivery order. The user reviews the suggested menu and places a delivery order for the desired items. The server receives the user's order information and sends the order details to the partner delivery service. The input for this step is the user's order information, and the output is the order data sent to the delivery service.

[1604] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1605] This invention is a system in which the user inputs data on the ingredients stocked in their refrigerator, and based on that data, it suggests appropriate menus and provides cooking instructions in both audio and video. Furthermore, it is a system that supports health promotion by inputting the user's health checkup results and suggesting a balanced diet based on those results. Moreover, this invention is characterized by its integration of an emotion engine that recognizes the user's emotions, and by its ability to customize suggestions and support based on the user's emotions. Specific embodiments for carrying out this invention are described below.

[1606] Overall system configuration

[1607] The system consists of user terminals, a central server, a database, and an emotion engine.

[1608] User terminal: Refers to devices such as smartphones and PCs, which users use to input data or receive suggested information.

[1609] Central server: This is the central component of a system that processes and manages data and provides users with the information they need.

[1610] Database: Stores data such as ingredient information, recipe information, and health checkup results.

[1611] Emotion Engine: Analyzes the user's facial expressions and voice to recognize emotions and provides that information to the server.

[1612] Program flow

[1613] 1. The user enters the ingredient information.

[1614] Users access the system and log in using their smartphones or PCs. After logging in, they enter the types and quantities of food items in their refrigerator. For example, they might enter "6 eggs, 2 tomatoes, 500ml milk." The entered information is sent from the device to the server and stored in the database.

[1615] 2. The server suggests a menu.

[1616] The server uses the received ingredient information to query the ingredient database and recipe database to find appropriate menus. For example, it might select "omelet" and "tomato soup" using eggs, tomatoes, and milk. The selected menus are sent to the terminal and displayed to the user.

[1617] 3. Audio and video support for cooking instructions

[1618] The server generates audio guides and video links for cooking instructions corresponding to the suggested menu. These are sent to the terminal for the user to use while cooking. For example, the video link might show steps such as "how to whisk eggs."

[1619] 4. The user enters their health checkup results.

[1620] Users input their health checkup results (e.g., LDL cholesterol levels and blood sugar levels) into the system. This information is sent from the terminal to the server and stored in a database.

[1621] 5. Dietary suggestions based on health checkup results

[1622] The server analyzes the stored health check results and generates suggestions for improving the user's diet based on their health status. For example, if the user has high LDL cholesterol levels, menus using cholesterol-lowering ingredients will be suggested. These suggestions are also sent to the device and displayed to the user.

[1623] 6. Recognition of user emotions by an emotion engine

[1624] While the user is using the system, the emotion engine analyzes the user's facial expressions and voice to recognize their emotions. This information is sent to the server in real time.

[1625] 7. Customizing suggestions based on emotions

[1626] The server customizes suggested menus, cooking instructions, audio guides, and video content based on the user's perceived emotions. For example, if the server detects that the user is stressed, it prioritizes suggesting easy-to-make recipes.

[1627] 8. Users create meals and promote health.

[1628] Users aim to improve their health by preparing and consuming appropriate dishes according to the server's suggestions. Users practice cooking while referring to the provided recipes, audio guides, and videos.

[1629] Specific example

[1630] For example, if a user enters "6 eggs, 2 tomatoes, 500ml milk" as the ingredients in their refrigerator, the server will suggest recipes for "omelet" and "tomato soup" using these ingredients. The device will also provide a video and audio guide demonstrating the cooking process. Furthermore, if the user's health check results indicate "high LDL cholesterol levels," the server will advise using ingredients rich in cholesterol-lowering components.

[1631] Furthermore, if the emotion engine detects from the user's facial expressions that they are "tired," the server will suggest a simple and quick recipe. In this way, it is possible to provide support tailored to the user's situation and condition, promoting health management.

[1632] This invention is a system that supports a balanced diet by suggesting appropriate menus based on the ingredients in the user's refrigerator, health check results, and emotions, thereby contributing to the improvement of the user's health.

[1633] The following describes the processing flow.

[1634] Step 1:

[1635] Users access the system using their smartphones or PCs and enter their login information. Upon successful login, the main screen is displayed.

[1636] Step 2:

[1637] The user enters information about the food items stocked in their refrigerator. For example, they might enter the type and quantity of food items such as "6 eggs, 2 tomatoes, 500ml milk."

[1638] Step 3:

[1639] The terminal sends the entered ingredient information to the server via the interface. This transmission is performed using methods such as HTTP requests or API calls.

[1640] Step 4:

[1641] The server saves the received ingredient information to the database. During the saving process, the type and quantity of ingredients are registered in the database as key-value pairs.

[1642] Step 5:

[1643] The server queries the ingredient database and recipe database based on the stored ingredient information. It then uses database queries to search for suitable recipes.

[1644] Step 6:

[1645] The server generates suggested menus from the recipes it finds. For example, it might select "Omelet" and "Tomato Soup" using eggs, tomatoes, and milk.

[1646] Step 7:

[1647] The server sends the generated suggested menu to the terminal. This is also done via HTTP requests or API responses.

[1648] Step 8:

[1649] The terminal displays suggested menus to the user. The user interface shows detailed recipes for "Omelet" and "Tomato Soup."

[1650] Step 9:

[1651] The server generates an audio guide and video link for the cooking procedure and sends it to the terminal. This process provides the audio file and the URL of the video.

[1652] Step 10:

[1653] The device displays the received audio guide and video link to the user. The user can then play the audio guide or watch the video.

[1654] Step 11:

[1655] The user obtains their own health checkup results and inputs them into the system. For example, they might input data such as "LDL cholesterol 150 mg / dL, blood sugar level 110 mg / dL".

[1656] Step 12:

[1657] The device sends the entered health check results to the server. This is also done via HTTP requests or API calls.

[1658] Step 13:

[1659] The server stores the received health check results in a database and evaluates the user's health status. For example, if the LDL cholesterol level is high, the system may advise avoiding certain foods.

[1660] Step 14:

[1661] The server generates dietary improvement suggestions based on the user's health status. For example, it advises users with high LDL cholesterol levels to use more tomatoes and other low-cholesterol foods.

[1662] Step 15:

[1663] The server sends generated dietary improvement suggestions to the terminal and displays them to the user. These include health-conscious recipes and advice on selecting ingredients.

[1664] Step 16:

[1665] While the user is using the system, the emotion engine analyzes the user's facial expressions and voice to recognize their emotions. This information is sent to the server in real time.

[1666] Step 17:

[1667] The server customizes suggestions based on the user's recognized emotional state. For example, if the server detects that the user is stressed, it will prioritize suggesting easy-to-make recipes.

[1668] Step 18:

[1669] The server sends emotionally-based, customized suggestions to the device and displays them to the user. This provides information that is tailored to the user's emotions and state of mind.

[1670] Step 19:

[1671] The user creates a dish according to customized suggestions. For example, when the user is tired, they can select a simple recipe to reduce cooking time.

[1672] Step 20:

[1673] Users consume meals they have created to improve their health. The system provides balanced dietary suggestions and emotionally-based customization, enabling effective health management.

[1674] (Example 2)

[1675] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1676] Maintaining an efficient and healthy diet is difficult in today's busy lifestyle. In particular, there is a need for systems that offer menu suggestions utilizing ingredients already in the refrigerator, and suggestions for dietary improvements based on individual health checkup results. Customized suggestions that take user emotions into account are also necessary, but currently, no system exists that can effectively achieve this.

[1677] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1678] In this invention, the server includes means for the user to input ingredient information, means for generating suggested menus based on the input ingredient information, means for providing cooking procedures corresponding to the generated menus in audio and video, means for the user to input health checkup results, means for generating suggestions to improve dietary habits based on the input health checkup results, means for recognizing the user's emotions, and means for customizing the suggested content based on the recognized emotions. As a result, the user can effectively utilize the ingredients in their refrigerator, eat appropriate meals based on their health checkup results, and receive suggestions tailored to their emotional state on that day.

[1679] A "user" refers to someone who uses the system to input information about ingredients and health check results, and then utilizes the suggested menus and information.

[1680] "Ingredient information" refers to data on the types and quantities of ingredients in the refrigerator that the user enters into the system.

[1681] "Means for generating menus" refers to algorithms or functions that present appropriate recipes or cooking ideas based on input ingredient information.

[1682] "Means of providing cooking instructions via audio and video" refers to a function that informs the user of the cooking method corresponding to the proposed menu through audio guidance and video.

[1683] "Health checkup results" refers to information that users enter into the system, such as their blood test results and other information indicating their health status.

[1684] "Means for generating suggestions to improve eating habits" refers to a function that suggests appropriate meals and menus that take into account the user's health condition, based on the entered health checkup results.

[1685] "Means of recognizing emotions" refers to technologies that analyze a user's facial expressions, tone of voice, etc., to determine their emotional state at that time.

[1686] "Means for customizing suggestions" refers to a function that appropriately changes the content of suggested menus and information according to the recognized emotional state of the user.

[1687] A "database" refers to an information storage system used to accumulate and manage various types of data, such as ingredient information, recipe information, and health checkup results.

[1688] A "terminal" refers to a device that a user uses to input information or receive system suggestions.

[1689] This invention is a system that proposes an efficient and healthy diet to users. Based on ingredient information and health checkup results, the system proposes the optimal menu to the user and provides cooking instructions via voice and video. It is also distinctive in that it recognizes the user's emotions and provides customized suggestions according to the situation.

[1690] The entire system consists of user terminals, a central server, a database, and an emotion engine.

[1691] User terminal

[1692] A user terminal refers to a device such as a smartphone or PC, which the user uses to input food information and health checkup results, and to receive suggested information. This terminal accesses the system using a dedicated application or web browser.

[1693] Central server

[1694] The central server is the core of the system, processing and managing data and providing users with the information they need. The server has the following functions:

[1695] Data reception and storage: Receive food ingredient information and health checkup results submitted by users and store them in the database.

[1696] Menu Suggestion: Based on saved ingredient information, the system cross-references the ingredient database and recipe database to generate appropriate menus.

[1697] Cooking Instruction Generation: Generate links to provide audio and video instructions for cooking the proposed menu.

[1698] Suggestions based on health checkup results: Analyze saved health checkup results and generate appropriate dietary improvement suggestions.

[1699] Emotion recognition data processing: Receive data sent from the emotion engine and customize the suggested content.

[1700] database

[1701] The database stores various data, such as ingredient information, recipe information, and health checkup results. This allows the central server to quickly access the data when needed and provide optimal suggestions to users.

[1702] Emotional Engine

[1703] The emotion engine analyzes facial expressions and voice through the user's device's camera and microphone to recognize the user's emotions. The recognized emotion information is sent to a central server and used to customize the suggested content.

[1704] Specific example

[1705] For example, suppose a user enters "6 eggs, 2 tomatoes, 500ml milk" as the ingredients in their refrigerator. Based on this information, the central server suggests "omelet" and "tomato soup." The suggestions also include audio guides and video links showing the cooking procedure. Furthermore, if the user's health check results indicate "high LDL cholesterol levels," the central server will suggest menus that contain many cholesterol-lowering ingredients. Additionally, if the emotion engine recognizes from the user's facial expression that they are "tired," the central server will prioritize suggesting recipes that are easy to prepare and can be made quickly.

[1706] Example of a prompt

[1707] "Please enter the ingredients in your refrigerator (e.g., 6 eggs, 2 tomatoes, 500ml milk)."

[1708] "Please enter your health checkup results (e.g., LDL cholesterol level, blood sugar level)."

[1709] "Please suggest a dish. The emotion engine recognizes the user's emotions."

[1710] This allows users to make effective use of the ingredients in their refrigerator according to the system's suggestions and eat meals based on their health condition. Furthermore, they can receive optimal suggestions tailored to their emotional state at the time.

[1711] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1712] Step 1:

[1713] The user enters ingredient information.

[1714] Users access the system using their smartphones or PCs and log in by entering their authentication information on the login screen. Upon successful login, a screen appears where the user can enter the types and quantities of food items in their refrigerator. For example, "6 eggs, 2 tomatoes, 500ml milk." The device serializes this food information into JSON format and sends it to the server as an HTTP POST request.

[1715] Input: Information about the contents of the refrigerator (6 eggs, 2 tomatoes, 500ml milk)

[1716] Output: Ingredient information in JSON format sent to the server

[1717] Specific operation: The terminal collects the entered ingredient information, converts it to JSON format, and sends it to the server using the HTTPS protocol.

[1718] Step 2:

[1719] The server suggests a menu.

[1720] Based on the received ingredient information, the server cross-references the ingredient database and recipe database to find the most suitable menu. For example, using "eggs, tomatoes, and milk," it might select "omelet" and "tomato soup." The server then sorts these menus appropriately and sends them.

[1721] Input: Received ingredient information

[1722] Output: Suggestion menu information sent to the user terminal

[1723] Specific operation: The server executes an SQL query to retrieve recipes that match the ingredients, sorts the best recipes using an algorithm, and sends them to the terminal in JSON format.

[1724] Step 3:

[1725] Audio and video support for cooking instructions

[1726] The server generates cooking instructions corresponding to the proposed menu as audio guides and video links. The audio guides are generated using text-to-speech technology, and the video links are URLs to YouTube or internal storage showing the corresponding cooking instructions. This information is sent to the device, allowing the user to view it.

[1727] Input: Suggested menu information

[1728] Output: Audio guide and video link for cooking instructions sent to the user's device.

[1729] Specific operation: The server calls an API to convert text to speech, selects a video link, and sends it to the terminal along with the audio file.

[1730] Step 4:

[1731] The user enters their health checkup results.

[1732] Users input their health checkup results (e.g., LDL cholesterol levels and blood sugar levels) into the system. The entered data is securely transmitted from the terminal to the server and stored in a database.

[1733] Input: Health checkup results

[1734] Output: Health checkup results sent to the server and stored in the database.

[1735] Specific operation: The terminal serializes the health check results in JSON format and sends them to the server using the HTTPS protocol. The server receives them and stores them in a database.

[1736] Step 5:

[1737] Dietary suggestions based on health checkup results

[1738] The server analyzes stored health check results and generates dietary suggestions based on the user's health status. For example, if the user has high LDL cholesterol levels, it will suggest menus containing many cholesterol-lowering ingredients. These suggestions are also sent to the terminal and displayed to the user.

[1739] Input: Saved health check results

[1740] Output: Dietary improvement suggestions sent to the user's terminal.

[1741] Specific operation: The server executes an algorithm to analyze the health check results, generates suggestions, and sends them to the terminal in JSON format. The terminal displays the suggestions.

[1742] Step 6:

[1743] Recognition of user emotions by an emotion engine

[1744] The device uses its camera and microphone to transmit the user's facial expressions and voice in real time to the emotion engine while the system is in use. The emotion engine analyzes this data to understand the user's current emotional state. This information is then sent to the server.

[1745] Input: Real-time facial and audio data

[1746] Output: Emotion recognition data sent to the server

[1747] Specific operation: The device sends images captured by the camera and audio recorded by the microphone to the emotion engine, and then sends the results of the emotion analysis to the server.

[1748] Step 7:

[1749] Emotion-based customization of suggestions

[1750] The server customizes suggested menus, cooking instructions, audio guides, and video content based on the user's perceived emotions. For example, if the server detects that the user is stressed, it will suggest simple recipes that reduce the effort involved in cooking.

[1751] Input: Emotion recognition data

[1752] Output: Customized suggestions sent to the user terminal

[1753] Specific operation: The server analyzes emotional data, generates optimal suggestions based on the user's situation, and sends them to the terminal. The terminal then displays this information.

[1754] Step 8:

[1755] Users create meals and aim to improve their health.

[1756] Users aim to improve their health by preparing and consuming appropriate dishes according to the server's suggestions. Users practice cooking while referring to the provided recipes, audio guides, and videos.

[1757] Input: Recipe information, audio guide, and video links provided by the server.

[1758] Output: Health improvement obtained by consuming a healthy diet.

[1759] Specific actions: The user cooks and consumes the food based on the guide displayed on their device.

[1760] (Application Example 2)

[1761] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1762] Conventional systems often result in food waste because users are unable to effectively utilize the ingredients in their refrigerators. Furthermore, dietary suggestions aimed at health management are not tailored to the user's emotional needs, sometimes making them burdensome. This leads to food waste and a lack of consistent meal preparation as part of health management. To address these challenges, a system is needed that effectively utilizes ingredients and offers healthy dietary suggestions that resonate with the user's emotional needs.

[1763] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[1764] In this invention, the server includes means for the user to input ingredient information, means for generating suggested menus based on the input ingredient information, means for providing cooking procedures corresponding to the generated menus in audio and video, means for the user to input health checkup results, means for generating suggestions for improving dietary habits based on the input health checkup results, and means for recognizing the user's emotions and customizing the suggested content based on those emotions. This enables the server to suggest the optimal menu based on the ingredient information input by the user, as well as to provide customized suggestions according to the user's health condition and emotions.

[1765] "User" refers to a person who uses a service or system.

[1766] "Food ingredient information" refers to data about the types and quantities of food and ingredients owned by the user.

[1767] "Menu" refers to a list of suggested dishes.

[1768] "Cooking instructions" refer to a step-by-step guide that shows the process of preparing a dish.

[1769] "Audio and video" refers to multimedia content that provides users with visual and auditory instructions.

[1770] "Health checkup results" refers to medical data that indicates the user's health status.

[1771] "Dietary habits" refers to the user's daily eating habits and the types of food they consume.

[1772] "Suggestions" refer to recommendations or advice provided to users.

[1773] A "system" refers to a set of devices and software that operate by integrating the above-mentioned means.

[1774] A "database" refers to digital storage that stores information on food ingredients and health check results, and allows them to be retrieved as needed.

[1775] A "terminal" refers to a digital device that a user uses to interact with a system.

[1776] "Emotions" refers to the psychological and sensory states that a user exhibits while using the system.

[1777] "Recognizing emotions" refers to the process of determining a user's psychological state from their facial expressions and voice.

[1778] "Customizing the proposal" refers to adjusting and modifying the content of the proposal based on the user's emotional state.

[1779] Overall system configuration

[1780] This invention is a system in which a user inputs information about the ingredients in their refrigerator and their health check results, and based on that data, it suggests appropriate menus and provides cooking instructions. The system consists of a user terminal, a central server, a database, and an emotion engine.

[1781] User terminal

[1782] User terminals refer to devices such as smartphones, tablets, and PCs. Users use these terminals to input information about the ingredients in their refrigerator and their health check results. The terminals are also used to display suggested menus and cooking instructions (audio and video).

[1783] Central server

[1784] The central server is the core component of the system, receiving and processing food ingredient information and health checkup results submitted by users. Based on the food ingredient information, the server generates optimal menus and provides cooking instructions via audio and video. It also generates suggestions for improving dietary habits based on health checkup results.

[1785] database

[1786] The database is a system that stores user-entered information on ingredients, recipes, and health checkup results. The database contains a large amount of pre-stored recipes and health information, and the server accesses this data to suggest menus and dietary recommendations.

[1787] Emotional Engine

[1788] The emotion engine is a component that recognizes the user's emotions by analyzing their facial expressions and voice. The emotion engine sends the user's emotional information to a central server in real time, and the server customizes the suggested content based on this information.

[1789] Program flow

[1790] 1. The user enters the ingredient information.

[1791] Users access the system and log in using their smartphones or PCs. After logging in, they enter information about the ingredients in their refrigerator. For example, if they enter "500g chicken, 2 onions, 3 tomatoes," that information is sent from their device to the server and stored in the database.

[1792] 2. The server suggests a menu.

[1793] The server uses the received ingredient information to query the ingredient database and recipe database to find appropriate menus. For example, it might select "Chicken Curry" and "Tomato Salad," both using chicken, onions, and tomatoes. The selected menus are sent to the terminal and displayed to the user.

[1794] 3. Audio and video support for cooking instructions

[1795] The server generates audio guides and video links for cooking instructions corresponding to the suggested menu. These are sent to the terminal for the user to use while cooking. For example, the video link might show steps such as "how to finely chop an onion."

[1796] 4. The user enters their health checkup results.

[1797] Users input their health checkup results (e.g., blood pressure and blood sugar levels) into the system. This information is sent from the terminal to the server and stored in a database.

[1798] 5. Dietary suggestions based on health checkup results

[1799] The server analyzes the stored health check results and generates suggestions for improving the user's diet based on their health status. For example, if the user has high blood sugar levels, a menu using low-carbohydrate ingredients will be suggested. This suggestion is also sent to the terminal and displayed to the user.

[1800] 6. Recognition of user emotions by an emotion engine

[1801] While the user is using the system, the emotion engine analyzes the user's facial expressions and voice to recognize their emotions. This information is sent to the server in real time.

[1802] 7. Customizing suggestions based on emotions

[1803] The server customizes suggested menus, cooking instructions, audio guides, and video content based on the user's perceived emotions. For example, if the server detects that the user is tired, it prioritizes suggesting easy-to-make recipes.

[1804] Specific example

[1805] For example, if a user enters "500g chicken, 2 onions, 3 tomatoes" as ingredients in their refrigerator, the server will suggest recipes for "chicken curry" and "tomato salad" using these ingredients. The device will also provide a video and audio guide demonstrating the cooking process. Furthermore, if the user's health check results indicate "high blood sugar," the server will advise using more low-carbohydrate ingredients. Additionally, if the emotion engine detects the user's fatigue from their facial expression, the server will suggest easy and quick recipes. In this way, the system can provide support tailored to the user's situation and condition, promoting better health management.

[1806] Example of a prompt

[1807] For example, the prompt text to be input to the generative AI model would be as follows:

[1808] "If a user's health check results show 'high blood sugar' and their refrigerator contains '500g chicken, 2 onions, and 3 tomatoes,' please list some simple low-carb recipes that can be provided."

[1809] This invention is a system that supports a balanced diet by suggesting appropriate menus based on the ingredients in the user's refrigerator, health check results, and emotions, thereby contributing to the improvement of the user's health.

[1810] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1811] Step 1:

[1812] Input: Users log in to the system using a smartphone or PC and enter information about the ingredients in their refrigerator. For example, they might enter "500g chicken, 2 onions, 3 tomatoes."

[1813] Operation: The ingredient information entered by the user is sent from the terminal to the server.

[1814] Output: The entered ingredient information is saved in the database.

[1815] Step 2:

[1816] Input: The server retrieves the stored ingredient information.

[1817] Operation: The server queries the ingredient database and recipe database to find a suitable menu based on the entered ingredients.

[1818] Output: The searched menu information is sent to the terminal.

[1819] Step 3:

[1820] Input: The terminal displays menu information received from the server.

[1821] Action: The user reviews the suggested menu. For example, recipes for "Chicken Curry" and "Tomato Salad" are displayed.

[1822] Output: Menu information displayed to the user.

[1823] Step 4:

[1824] Input: Menu information selected by the user.

[1825] Operation: The server generates cooking instructions corresponding to the selected menu item and creates audio guides and video links.

[1826] Output: The generated cooking instructions, along with a video link, will be sent to the device.

[1827] Step 5:

[1828] Input: The terminal displays the audio guide and video link received from the server.

[1829] Operation: The user refers to audio guides and video links while cooking. For example, a video on "how to finely chop an onion" is displayed.

[1830] Output: Support for cooking instructions provided to the user.

[1831] Step 6:

[1832] Input: The user enters their health checkup results. For example, they might enter "high blood sugar."

[1833] Operation: The entered health check results are sent from the terminal to the server and stored in the database.

[1834] Output: The entered health check results are saved in the database.

[1835] Step 7:

[1836] Input: The server retrieves the saved health check results.

[1837] Operation: The server analyzes health check results and generates suggestions for improving the user's diet based on their health status. For example, it might advise using low-carbohydrate foods for those with high blood sugar levels.

[1838] Output: The generated dietary improvement suggestions are sent to the terminal.

[1839] Step 8:

[1840] Input: The terminal displays dietary improvement suggestions received from the server.

[1841] Action: The user reviews the suggested diet.

[1842] Output: Suggestions for improving dietary habits provided to the user.

[1843] Step 9:

[1844] Input: User's facial expressions and voice data.

[1845] Operation: The emotion engine analyzes the user's facial expressions and voice to recognize their emotions. Emotional information is sent to the server in real time.

[1846] Output: Recognized emotion information is sent to the server.

[1847] Step 10:

[1848] Input: The server customizes suggestions based on emotional information.

[1849] Operation: If the recognized emotion is determined to be "tired," the server will prioritize suggesting menus that are easy to prepare.

[1850] Output: Customized menu information is sent to the terminal.

[1851] The above is the specific processing flow of this system.

[1852] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1853] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1854] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[1855] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1856] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[1857] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[1858] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[1859] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[1860] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want...

Claims

1. A means for users to input ingredient information, A means of generating suggested menus based on input ingredient information, A means of providing cooking instructions corresponding to the generated menu in audio and video, A means for users to input their health checkup results, A means for generating suggestions for improving dietary habits based on input health check results, A system that includes this.

2. The system according to claim 1, further comprising means for storing food ingredient information and health checkup results in a database.

3. The system according to claim 1, further comprising means for displaying the proposed menu and dietary improvement suggestions on a terminal.

Images