System

Abstract

A system is provided.SOLUTION: The system includes a means for inputting information of foodstuffs in a refrigerator, a means for storing the foodstuff information in a database, a means for generating a menu considering nutritional balance based on the stored foodstuff information, and a means for displaying the generated menu information on a terminal.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In recent years, managing meals at home has become increasingly important. In particular, families with children are required to provide nutritionally balanced meals, and they must also take care not to overlap with the contents of school lunches. However, manually creating a menu that takes all of these factors into account is extremely time-consuming and labor-intensive. It is also difficult to create a menu for individuals with chronic illnesses or that emphasizes specific nutrients. Therefore, the present invention aims to solve these problems by providing a system that automatically creates nutritionally balanced menus based on information about ingredients in the refrigerator, and also by taking into account information about children's school lunches and health information. [Means for solving the problem]

[0005] In order to solve the above problems, the present invention provides the following means. The system of the present invention first comprises a means for inputting information about ingredients stored in a refrigerator. Next, it comprises a means for saving the input ingredient information in a database. It provides a means for generating a menu that takes nutritional balance into consideration based on the saved ingredient information, and includes a means for displaying the generated menu information on a terminal. It also comprises a means for inputting school lunch menu information at home and saving it in a database, and further comprises a means for generating a menu that does not overlap with school lunch menu information. It also comprises a means for inputting information about chronic illnesses and particular nutritional needs and saving it in a database, and includes a means for generating a menu that takes chronic illnesses and particular nutritional needs into consideration. This simplifies dietary management at home and makes it possible to provide menus that suit individual needs.

[0006] "Food information" refers to detailed data such as the type, quantity, and expiration date of ingredients in the refrigerator.

[0007] A "database" refers to a storage system that accumulates information on ingredients, school lunches, health information, etc., and allows for efficient management and retrieval.

[0008] A "menu" refers to the meal contents for a certain period of time, and includes specific dish items such as main dishes, side dishes, and desserts, as well as their combinations.

[0009] "Nutritional balance" refers to consuming the various nutrients necessary for the human body, such as proteins, lipids, carbohydrates, vitamins, and minerals, in appropriate proportions.

[0010] "Terminal" refers to an electronic device (e.g., PC, smartphone, tablet) that a user operates to input and display information.

[0011] "School lunch information" is data about the contents of the school lunches that children eat at school, including information about the type of food and nutrients.

[0012] "Health information" refers to information about the user's chronic illnesses and the nutrients they particularly want to consume.

[0013] A "menu generation algorithm" refers to the calculation procedures and rules for automatically generating optimal menus based on input ingredient information, school lunch information, and health information.

[0014] "Means of storage" refers to the processes and functions for recording various input information in a database.

[0015] "Display means" refers to functions and devices for visually presenting the generated menu information to the user. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11]FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] MODE FOR CARRYING OUT THE INVENTION

[0038] This invention is a system that automatically generates healthy menus based on information about ingredients in the refrigerator. The system is composed of three parties: the user, the terminal, and the server, and creates menus based on information stored in a database.

[0039] Enter ingredient information

[0040] First, the user inputs information about the ingredients in the refrigerator into the terminal. Specifically, the type of ingredient, quantity, expiration date, etc. are input. After confirming this information, the terminal sends it to the server. The server stores the received information in a database. The stored data is later used when creating menus.

[0041] Enter your child's school lunch information (optional)

[0042] For families with children, users input their children's school lunch information into the device. This information, like the food ingredient information, is sent to the server and stored in the database. The school lunch information includes the school lunch menu and its nutritional value. This information makes it possible to ensure that the breakfast and dinner menus do not overlap with those of school lunches.

[0043] Enter health information (optional)

[0044] If a user has a specific chronic illness or has specific nutrients they want to consume, they can enter that information into the device. This information is also sent to the server and stored in a database. For example, if a user wants to consume a lot of vitamin C, foods that fall into that category will be prioritized.

[0045] Menu generation

[0046] The server integrates information on ingredients, school meals, and health stored in a database to create menus that take into consideration nutritional balance, the amount of ingredients used, avoidance of overlaps with school meals, etc. The menu generation algorithm calculates the optimal meal plan that meets these conditions and proposes specific menus.

[0047] As a concrete example, consider a case where the refrigerator contains "chicken, tomatoes, and lettuce." If the user also inputs information such as "I want to consume a lot of vitamin C," the server can use these conditions to suggest "chicken stewed in tomato sauce" and "lettuce salad" for dinner. If the user has also input information about their child's school lunches, the server will adjust the suggestions to avoid overlapping with that information.

[0048] Display menu information

[0049] The menus generated by the server are sent to the device and displayed to the user, who can then check the suggested menus and incorporate them into their daily meal plans.

[0050] This system allows users to maintain a healthy and balanced diet while making effective use of the ingredients in their refrigerator. It also makes it easy to create optimal meal plans for the whole family by taking into account their children's school lunch information and health information.

[0051] The processing flow will be explained below.

[0052] Step 1:

[0053] The user inputs information about the ingredients in the refrigerator into the terminal, such as the type, quantity, and expiration date of the ingredients, and specific data is provided to the terminal.

[0054] Step 2:

[0055] The terminal checks the ingredient information entered by the user to ensure there are no errors, and then prepares to send this information to the server.

[0056] Step 3:

[0057] The device sends the ingredient information to the server as an HTTP request. The request includes all the ingredient information entered.

[0058] Step 4:

[0059] The server analyzes the received request, extracts the ingredient information, and stores it in the food_inventory table in the database.

[0060] Step 5:

[0061] The user inputs their child's school lunch information into the terminal, and detailed information about the school lunch menu and nutritional value is provided.

[0062] Step 6:

[0063] The terminal checks the lunch information to make sure there are no errors, and then prepares to send this information to the server.

[0064] Step 7:

[0065] The terminal sends the school lunch information to the server as an HTTP request. The request includes the entered school lunch menu information.

[0066] Step 8:

[0067] The server analyzes the received request, extracts the school meal information, and saves it in the school_meals table in the database.

[0068] Step 9:

[0069] The user inputs information about a specific chronic illness or the nutrients they want to consume into the device. For example, they input information such as "I want to consume a lot of vitamin C."

[0070] Step 10:

[0071] The device checks the health information to make sure there are no errors, and then prepares to send this information to the server.

[0072] Step 11:

[0073] The device sends the health information to the server as an HTTP request, which includes the entered health information.

[0074] Step 12:

[0075] The server analyzes the received request, extracts the health information, and stores it in the health_requirements table in the database.

[0076] Step 13:

[0077] The server queries the database to obtain information on ingredients, meal plans, and health information, which is then used as input data for the menu generation algorithm.

[0078] Step 14:

[0079] The server runs a menu generation algorithm to generate a menu that takes into consideration nutritional balance, the amount of ingredients used, avoidance of overlap with school lunches, health requirements, etc.

[0080] Step 15:

[0081] The server compiles the generated menu information and sends it to the terminal as an HTTP response, which includes the specific menu and its nutritional information.

[0082] Step 16:

[0083] The device analyzes the menu information received from the server and displays it in a user-friendly format. The user can then check the proposed menu through the device.

[0084] Example 1

[0085] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0086] Currently, there are applications and systems on the market that suggest meal plans based on the ingredients in the refrigerator, but most of them do not take into account nutritional balance, individual health information, or information about school lunches at home, and as a result, they are unable to provide efficient and healthy meal plans. This poses the problem that users have to go to the trouble of planning their own meals and assembling menus that take health information into account. Furthermore, it is difficult to create menus that do not overlap with information about children's school lunches.

[0087] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0088] In this invention, the server includes a means for inputting information about ingredients in the refrigerator, a means for saving the ingredient information in a database, a means for generating a menu that takes nutritional balance into consideration based on the saved ingredient information, a means for generating a menu based on a prompt sentence using a generation AI model, and a means for displaying the generated menu information on a terminal. This makes it possible to effectively use ingredients in the refrigerator and automatically provide a well-balanced menu that takes into consideration the user's health information and children's school lunch information.

[0089] "Information about food items in the refrigerator" refers to attribute information such as the type, quantity, and expiration date of the food items stored in the refrigerator.

[0090] "Database" means an organized collection of data for efficient storage, management, and retrieval of information.

[0091] "Nutritional balance" refers to a state in which the food you eat contains nutrients such as protein, lipids, carbohydrates, vitamins, and minerals in appropriate proportions.

[0092] "Menu" refers to the combination of menu items for a particular meal.

[0093] A "generative AI model" refers to an algorithm or system that uses AI (artificial intelligence) technology to make predictions or generate information based on text or data.

[0094] "Prompt sentence" refers to an instruction sentence input to a generative AI model to cause it to generate a specific output.

[0095] "Device" refers to an electronic device used by a user to input information or view output information. Examples include smartphones and personal computers.

[0096] "School lunch menu information" refers to information about the menu and nutritional value of meals provided to children at school or other facilities.

[0097] "Information about chronic conditions and specific nutritional needs" refers to information about nutrients that a user wishes to consume or avoid based on a user's specific health condition or goals.

[0098] "Means for inputting information about ingredients in the refrigerator" refers to an interface or method that allows a user to input attribute information about ingredients in the refrigerator into the system.

[0099] "Storage means" refers to a method or system for storing input information in a database.

[0100] "Generating means" refers to the methods or techniques used to create a particular output (e.g., a menu or recipe) based on the stored information.

[0101] "Means for displaying" refers to the method or interface for conveying the generated information to the user.

[0102] MODE FOR CARRYING OUT THE INVENTION

[0103] This invention is a system that automatically generates healthy menus based on information about ingredients in a refrigerator. This system is composed of three parties: a user, a terminal, and a server. The specific processing steps are explained below.

[0104] Enter and save ingredient information

[0105] First, the user inputs information about the ingredients in the refrigerator into the terminal. Specifically, the user inputs information such as the type of ingredient (e.g., chicken, tomato, lettuce), quantity, and expiration date. The terminal uses a common interface, such as a smartphone or PC. The terminal checks the input information and sends it to the server as JSON-formatted data. The server stores this information in a database system such as MySQL (registered trademark) or PostgreSQL.

[0106] Enter your child's school lunch information (optional)

[0107] If the user has information about school lunches for their children at home, they can also enter this information into the device. The school lunch menu and its nutritional value are entered, and the device confirms this and sends it to the server. This information is also saved in the database. For example, by saving detailed information such as "Today's school lunch is curry rice (calories: 450kcal, vitamin C: 10mg)," the server can use this information to generate menus later.

[0108] Enter health information (optional)

[0109] Furthermore, if a user has a specific chronic illness or has specific nutrients they want to consume, they can enter that information through the device. For example, they can enter information such as "I want to consume a lot of vitamin C." After confirming this, the device sends it to the server, which stores it in a database. By storing this health information, it is given priority when generating menus.

[0110] Automatic menu generation

[0111] The server generates menus based on the ingredient information, school meal information, and health information stored in the database, taking into consideration factors such as nutritional balance, the amount of ingredients used, and avoiding overlaps with school meals. This process uses artificial intelligence such as the popular GPT-4 (registered trademark) as a generative AI model. To generate a menu that meets specific conditions, a prompt is created and sent to the AI ​​model.

[0112] For example, use the following prompt:

[0113] I have the following ingredients in my fridge:

[0114] 500g chicken

[0115] 3 tomatoes

[0116] 1 head of lettuce

[0117] I would also like to get a lot of vitamin C. Could you please suggest a suitable menu for tonight's dinner?

[0118] (If there is information about children's school lunches)

[0119] Today's school lunch includes curry rice (calories: 450kcal, vitamin C: 10mg). Please make sure that the menu does not overlap with the school lunch menu.

[0120] Based on this prompt, the generative AI model generates a specific menu, such as "chicken stewed in tomato sauce" and "lettuce salad."

[0121] Display menu information

[0122] The menu information generated by the server is sent to the device, which receives the information and displays it in a visually easy-to-understand format for the user. The user can then review the proposed menu and incorporate it into their daily meal plan.

[0123] This system allows users to easily achieve a healthy and balanced diet while making effective use of the ingredients in their refrigerator. It also takes into account individual children's school lunch information and health information, making it possible to automatically provide optimal meal plans for the whole family.

[0124] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0125] Program processing flow

[0126] Step 1: Enter ingredient information

[0127] 1. The user enters the ingredients information into the terminal.

[0128] The user inputs the type, quantity, and expiration date of ingredients in the refrigerator into the terminal.

[0129] Input data: type of ingredients (e.g. chicken, tomato, lettuce), quantity, expiration date

[0130] 2. The device checks the entered information

[0131] The terminal automatically checks the input for errors.

[0132] Data processing: Check if the quantity field is a number and if the expiration date is in the future.

[0133] Output data: Ingredient information after verification (valid data)

[0134] 3. The device sends the information to the server

[0135] After checking the entered ingredient information, it is sent to the server in JSON format.

[0136] Data processing: Converting ingredient information into JSON format

[0137] Output data: Ingredient information in JSON format

[0138] Step 2: Save the ingredients

[0139] 1. The server receives the information

[0140] The server receives the ingredient information sent from the terminal.

[0141] Input data: JSON formatted ingredient information

[0142] 2. The server stores the information in a database

[0143] The server stores the received ingredient information in an appropriate database system (e.g., MySQL, PostgreSQL).

[0144] Data processing: Convert JSON format data into database format

[0145] Output data: Ingredient information stored in a database

[0146] Step 3: Enter school lunch information (optional)

[0147] 1. The user inputs the school lunch menu information into the terminal.

[0148] The user inputs the menu and nutritional value of the school lunch that the child will have at school into the terminal.

[0149] Input data: School lunch menu, nutritional value (e.g. curry rice, calories: 450kcal, vitamin C: 10mg)

[0150] 2. The device checks the information and sends it to the server

[0151] The terminal checks the input information for errors and sends it to the server in JSON format.

[0152] Data processing: Convert input information into JSON format

[0153] Output data: JSON format school lunch information

[0154] 3. The server stores the information in a database

[0155] The server stores the received meal information in a database.

[0156] Input data: JSON format school lunch information

[0157] Output data: Meal information stored in a database

[0158] Step 4: Enter your health information (optional)

[0159] 1. The user enters health information into the device

[0160] Users input information about specific medical conditions and the nutrients they want to consume into the device.

[0161] Input data: Health information (e.g., I want to take in more vitamin C)

[0162] 2. The device checks the information and sends it to the server

[0163] The device reviews the input information for errors and sends it to the server in JSON format.

[0164] Data processing: Convert input information into JSON format

[0165] Output data: Health information in JSON format

[0166] 3. The server stores the information in a database

[0167] The server stores the received health information in a database.

[0168] Input data: Health information in JSON format

[0169] Output data: Health information stored in a database

[0170] Step 5: Automatic menu generation

[0171] 1. The server retrieves the necessary information from the database

[0172] The server retrieves food ingredient information, school lunch information, and health information from the database.

[0173] Input data: Various information stored in the database

[0174] Output data: Consolidated information list

[0175] 2. The server generates a prompt using the generative AI model.

[0176] The server creates a prompt based on the information obtained.

[0177] Data processing: Converting information lists into prompt statements

[0178] Output data: Generated prompt statement

[0179] Examples:

[0180] I have the following ingredients in my fridge:

[0181] 500g chicken

[0182] 3 tomatoes

[0183] 1 head of lettuce

[0184] I would also like to get a lot of vitamin C. Could you please suggest a suitable menu for tonight's dinner?

[0185] (If there is information about children's school lunches)

[0186] Today's school lunch includes curry rice (calories: 450kcal, vitamin C: 10mg). Please make sure that the menu does not overlap with the school lunch menu.

[0187] 3. The server sends a prompt to the generative AI model (e.g., GPT-4).

[0188] The server sends the prompt to the AI ​​model, asking it to generate a menu.

[0189] Input data: Prompt statement

[0190] Output data: Generated menu information

[0191] 4. The generative AI model returns a response, which the server receives.

[0192] Receive menu information generated from an AI model.

[0193] Input data: AI model response

[0194] Output data: Menu information (e.g., chicken stewed in tomato sauce, lettuce salad)

[0195] Step 6: Display menu information

[0196] 1. The server sends the generated menu information to the terminal.

[0197] The server sends the generated menu information to the terminal in JSON format.

[0198] Input data: Generated menu information

[0199] Output data: Menu information in JSON format

[0200] 2. The device analyzes the menu information received from the server.

[0201] The terminal parses the JSON formatted data and converts it into a format that is displayed to the user.

[0202] Data processing: Convert JSON format data into a data format for visualization

[0203] Output data: Visualized menu information

[0204] 3. The device displays the menu information to the user.

[0205] The device displays visualized menu information to the user, who can then review the suggested menu and incorporate it into their daily meal planning.

[0206] Input data: Visualized menu information

[0207] Output data: Menu information displayed to the user

[0208] This detailed process flow allows users to efficiently utilize the ingredients in their refrigerator and achieve a healthy and balanced diet.

[0209] (Application example 1)

[0210] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0211] In today's busy households, it is an important challenge to efficiently utilize the ingredients stored in the refrigerator and easily generate nutritionally balanced menus. To address this challenge, there is a demand for menu generation that takes into account health information and information about meals provided at home. However, current systems are unable to fully meet these demands. In particular, there is no system that allows users to easily input information about ingredients purchased at stores, which requires the user to enter the information manually. Furthermore, since conventional menu generation systems have difficulty taking into account health information and meal information, there is a demand for a system that can integrate such information and generate optimal menus.

[0212] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0213] In this invention, the server includes a means for inputting information about ingredients stored in the refrigerator, a means for saving the ingredient information in a database, a means for generating a nutritionally balanced menu based on the saved ingredient information, a means for scanning purchased ingredients and inputting information, and a means for displaying the generated menu information on a terminal. This allows the user to easily input ingredients purchased at the store and easily generate an optimal nutritionally balanced menu. In addition, by taking into account menu information and health information for school lunches at home and avoiding duplication, the health of the entire family can be supported.

[0214] "Information about ingredients in the refrigerator" refers to detailed information such as the type, quantity, and expiration date of ingredients that the user has in the refrigerator.

[0215] "Means for saving to a database" refers to the process and system for storing the input information in a database on a server so that it can be referenced later.

[0216] The "means for generating a menu that takes nutritional balance into consideration" refers to an algorithm and system that automatically creates a menu with optimal nutritional balance based on stored ingredient information, the user's health condition, and nutritional needs.

[0217] "Means for scanning purchased ingredients and entering information" refers to a function that automatically enters information about ingredients by scanning the barcode or QR code (registered trademark) of ingredients purchased at a store using a smartphone or other device.

[0218] The "means for displaying the generated menu information on the terminal" is a system that displays the menu generated by the server on the user's terminal such as a smartphone or tablet, making it available for the user to refer to.

[0219] "Information about school lunch menus at home" refers to information about the menus and nutritional components of school lunches that children in the home eat at school or other places.

[0220] "Information about chronic illnesses and particular nutrients you want to take" refers to information about specific health problems or specific nutrients you want to take in more of.

[0221] A "generative AI model" is an artificial intelligence model used to suggest optimal menus and recipes based on user input data.

[0222] A "prompt sentence" is a sentence-based instruction provided to a generative AI model as input data for generating a menu.

[0223] This invention is a system that automatically generates menus that take into consideration the user's health status and nutritional balance based on information about ingredients in a refrigerator. The system is mainly composed of three entities: a server, a terminal, and a user. Specific embodiments are described below.

[0224] Entering ingredient information

[0225] The user inputs information about ingredients in the refrigerator into a device such as a smartphone. This information includes the type, quantity, and expiration date. The device confirms the information entered by the user and then sends it to the server. The server stores the received information in a database and uses it later when creating menus.

[0226] Enter purchased food information

[0227] When a user purchases new ingredients at a store, they can quickly enter ingredient information by scanning the barcode or QR code using the smartphone camera. This information is also sent to the server and stored in the database.

[0228] Entering health and child meal information

[0229] If a user has health problems or special nutritional needs, they can enter that information into the device. If they have children at home, they can also enter information about their school lunches. This information is also sent to the server and stored in a database.

[0230] Menu generation

[0231] The server uses a generative AI model to automatically generate optimal menus based on stored ingredient information, health information, and school lunch information. The generated menus take into consideration nutritional balance, ingredient usage efficiency, and avoidance of overlap with school lunches. The generated menus are then evaluated for nutritional value using the Nutritics API.

[0232] Specifically, the server uses the following methods:

[0233] A way to input information about ingredients in the refrigerator

[0234] A means for storing the ingredient information in a database

[0235] A method for generating nutritionally balanced menus based on stored ingredient information

[0236] A way to scan purchased ingredients and enter information

[0237] A method for generating menus based on health information using a generative AI model

[0238] This allows users to easily input ingredients purchased at the store and generate optimal menus that take nutritional balance into consideration. It also takes into account information about school lunches and health information at home, helping to support the health of the entire family.

[0239] Display menu information

[0240] The generated menu information is sent from the server to the user's device and displayed on their smartphone, tablet, etc. Users can check it and incorporate it into their daily meal plans.

[0241] Specific examples

[0242] For example, if a user purchases "chicken, tomato, lettuce, and carrot" at a supermarket and scans each item with a smartphone app, and the user enters "high in vitamin C" as health information and "curry (low in vitamins)" as their child's school lunch information, the app will automatically suggest a menu of "chicken and tomato stew, lettuce and carrot salad (high in vitamin C)."

[0243] Example prompts to input to the generative AI model

[0244] Consider the user's health condition and generate a menu rich in Vitamin C.

[0245] Based on information about your child's school lunch, please suggest healthy menus that do not use duplicate ingredients.

[0246] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0247] Step 1:

[0248] The user inputs information about ingredients in the refrigerator into the terminal. Specifically, the type of ingredient, quantity, and expiration date are entered, and the information is sent from the terminal to the server. At this time, the input information is converted into a data format such as JSON.

[0249] input:

[0250] Food type

[0251] Ingredient quantity

[0252] expiration date

[0253] output:

[0254] JSON formatted ingredient information

[0255] Step 2:

[0256] The server receives the ingredient information sent from the terminal and stores it in a database, where the ingredient information is recorded in a table format.

[0257] input:

[0258] JSON formatted ingredient information

[0259] output:

[0260] Ingredient information stored in the database

[0261] Step 3:

[0262] Users scan ingredients purchased at the store using the device's camera and enter the information. By reading barcodes and QR codes, the product name, nutritional information, expiration date, etc. are automatically obtained and sent to the server.

[0263] input:

[0264] Barcode / QR code

[0265] output:

[0266] Purchased ingredients information in JSON format

[0267] Step 4:

[0268] The server receives the information about the purchased ingredients and stores it in an existing database. The purchased ingredient information is also managed as an integrated ingredient information.

[0269] input:

[0270] Purchased ingredients information in JSON format

[0271] output:

[0272] Ingredient information stored in a database (integrated with existing databases)

[0273] Step 5:

[0274] The user inputs specific health information and information about their child's school lunch into the terminal. Health information includes specific nutrient intake preferences and information about chronic illnesses, while child's school lunch information includes the school lunch menu and its nutritional components. This information is also sent to the server.

[0275] input:

[0276] health information

[0277] School lunch information

[0278] output:

[0279] Health and meal information in JSON format

[0280] Step 6:

[0281] The server receives the health information and feeding information and stores it in a database, so that all necessary information is collected in the database.

[0282] input:

[0283] Health and meal information in JSON format

[0284] output:

[0285] Health and feeding information stored in a database

[0286] Step 7:

[0287] The server uses a generative AI model to generate optimal menus based on stored information on ingredients, health, and school lunches. It inputs prompt statements into the generative AI model to calculate nutritionally balanced and efficient menus, and evaluates the nutritional value of the generated menus using the Nutritics API.

[0288] input:

[0289] Food ingredient information, health information, and school lunch information stored in the database

[0290] Prompt statement

[0291] output:

[0292] Optimal menu information

[0293] Specific prompt examples:

[0294] Consider the user's health condition and generate a menu rich in Vitamin C.

[0295] Step 8:

[0296] The generated menu information is sent from the server to the user's device and displayed on their smartphone, tablet, etc. The user can then review it and incorporate it into their daily meal plan.

[0297] input:

[0298] Optimal menu information

[0299] output:

[0300] Menu information displayed on the device

[0301] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0302] MODE FOR CARRYING OUT THE INVENTION

[0303] This invention is a system that generates healthy menus based on information about ingredients in the refrigerator and also takes into account the user's emotional information. This system is composed of a user, a terminal, a server, and an emotion engine.

[0304] Enter ingredient information

[0305] The user inputs information about ingredients in the refrigerator into the terminal. Specifically, the user provides information such as the type of ingredient, quantity, and expiration date. The terminal sends this information to the server, which then stores the received ingredient information in a database. The stored ingredient information is used as basic data for creating menus.

[0306] Enter your child's school lunch information (optional)

[0307] Users also input their child's school lunch information into the device. This information, including details about the menu and nutritional value, is sent to the server and stored in a database. Based on this information, a menu that does not overlap with school lunches is generated.

[0308] Enter health information (optional)

[0309] Users enter information about any medical conditions or specific nutritional needs they have into the device. For example, if they want to increase their intake of vitamin C, they enter that information. This information is also sent to the server and stored in a database. The health information is used to manage illnesses and create diet plans that emphasize specific nutrients.

[0310] emotional information recognition

[0311] The emotion engine recognizes emotions from the user's voice and input information. For example, if the user is feeling stressed, it will detect that emotion. The emotion engine sends this information to the server and stores it in a database. The emotion information is used to create a menu that reflects the user's current psychological state.

[0312] Menu generation

[0313] The server queries and obtains information on ingredients, school meals, health, and emotions from the database. Based on this information, a menu generation algorithm operates, taking into account nutritional balance, the amount of ingredients used, avoidance of overlaps with school meals, health conditions, and emotional state to generate an optimal menu. For example, for a user who is feeling stressed, ingredients with a relaxing effect will be selected.

[0314] As a concrete example, consider a situation where the refrigerator contains "chicken, tomatoes, and lettuce." If the user inputs "I want to get more vitamin C," and the emotion engine also detects "fatigue," the server will use these conditions to suggest "chicken stewed in tomato sauce," "lettuce salad," and "oranges," which are rich in vitamin C.

[0315] Display menu information

[0316] The menus generated by the server are sent to the terminal and displayed to the user. The user can check the suggested menus and incorporate them into their daily meal plans. In particular, the system is expected to have the effect of refreshing the mind through meals by suggesting menus that take emotions into consideration.

[0317] This system allows users to maintain a healthy and balanced diet while making the most of the ingredients in their refrigerator. It also takes into account their children's school lunch information, health information, and even their emotional state, making it easy to create optimal meal plans for the whole family.

[0318] The processing flow will be explained below.

[0319] Step 1:

[0320] The user inputs information about ingredients in the refrigerator into the terminal, such as the type of ingredient (e.g., chicken, tomato, lettuce), quantity, and expiration date.

[0321] Step 2:

[0322] The terminal checks the entered ingredient information and checks for errors. After checking, it prepares to send this ingredient information to the server.

[0323] Step 3:

[0324] The device sends the ingredient information to the server as an HTTP request. The request includes all the ingredient information entered.

[0325] Step 4:

[0326] The server analyzes the received request, extracts the ingredient information, and stores it in the food_inventory table in the database.

[0327] Step 5:

[0328] The user inputs information about the child's school lunch into the terminal, including detailed information about the school lunch menu and nutritional value.

[0329] Step 6:

[0330] The terminal checks the entered school lunch information and checks for errors. After checking, the terminal prepares to send this school lunch information to the server.

[0331] Step 7:

[0332] The terminal sends the school lunch information to the server as an HTTP request. The request includes all the school lunch information that was entered.

[0333] Step 8:

[0334] The server analyzes the received request, extracts the school meal information, and saves it in the school_meals table in the database.

[0335] Step 9:

[0336] The user inputs information about their chronic illnesses and the nutrients they particularly want to take into the terminal. For example, they input information such as "I want to take in a lot of vitamin C."

[0337] Step 10:

[0338] The terminal checks the entered health information for errors, and then prepares to send the health information to the server.

[0339] Step 11:

[0340] The device sends the health information to the server as an HTTP request, which includes all the entered health information.

[0341] Step 12:

[0342] The server analyzes the received request, extracts the health information, and stores it in the health_requirements table in the database.

[0343] Step 13:

[0344] The emotion engine recognizes emotions from user input and voice. If the user is feeling stressed, it will detect that emotional information.

[0345] Step 14:

[0346] The emotion engine sends the recognized emotion information to the server as an HTTP request. The request includes the recognized emotion information.

[0347] Step 15:

[0348] The server analyzes the received request and extracts emotional information. The extracted information is stored in the emotion_data table in the database.

[0349] Step 16:

[0350] The server queries the database to obtain information on ingredients, meals, health, and emotions, and uses the information as input data for the menu generation algorithm.

[0351] Step 17:

[0352] The server runs a menu generation algorithm to generate a menu that takes into account nutritional balance, the amount of ingredients used, avoidance of overlaps with school meals, health conditions, and emotional state. For example, ingredients with a relaxing effect are selected for a stressed user.

[0353] Step 18:

[0354] The server compiles the generated menu information and sends it to the terminal as an HTTP response, which includes the specific menu and its nutritional information.

[0355] Step 19:

[0356] The device analyzes the menu information received from the server and displays it in a user-friendly format. The user can then check the proposed menu through the device.

[0357] This detailed process flow allows users to effectively use the ingredients in their refrigerator while creating meals that are in line with their health, nutritional balance, and even their own emotional state. It also makes it possible to suggest menus that do not overlap with children's school lunch information, providing optimal meal plans for the whole family.

[0358] Example 2

[0359] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0360] Conventional menu generation systems could take into account information about ingredients in the refrigerator and the user's health information, but they were unable to propose menus that took into account the user's emotional information. As a result, they were unable to provide more personalized menus that reflected the user's daily emotional fluctuations. Furthermore, even when generating menus that did not overlap or that emphasized specific nutrients, only limited information was taken into account, making it difficult to create optimal meal plans.

[0361] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0362] In this invention, the server includes a means for storing ingredient information in a database, a means for recognizing the user's emotional information, and a means for generating a menu that takes nutritional balance into consideration, thereby enabling the generation of a more personalized menu based on the ingredient information, health information, and emotional information in the user's refrigerator.

[0363] "Ingredient information" refers to data such as the type, quantity, and expiration date of food stored in the refrigerator.

[0364] A "database" is an information system for storing, managing, and retrieving data in an organized manner, and takes the form of a relational database management system (RDBMS).

[0365] The "means for generating a menu that takes nutritional balance into consideration" refers to an algorithm or program that calculates the balance of nutrients based on the input food ingredient information and other conditions and suggests the optimal meal.

[0366] A "terminal" is a device through which a user inputs information, and includes smartphones, tablets, personal computers, etc.

[0367] "Emotional information" is data that indicates the user's psychological state or mood, and is acquired through voice recognition, text input, or the like.

[0368] The "means for recognizing user's emotional information" is a combination of hardware and software for analyzing the user's voice and input data to identify the user's psychological state.

[0369] "Information on school lunch menus at home" refers to data on the menus and nutritional values ​​of meals provided to children at school or other facilities.

[0370] "Information about chronic illnesses and particularly desirable nutrients" is data about the health condition that the user wants to manage and the nutrients that the user wants to prioritize.

[0371] A "menu" is data that includes a meal menu and the ingredients and cooking procedures for serving it.

[0372] MODE FOR CARRYING OUT THE INVENTION

[0373] This invention is a system that generates healthy menus based on information about ingredients in a refrigerator and also takes into account the user's emotional information. This system is composed of a user, a terminal, a server, and an emotion engine.

[0374] Entering ingredient information

[0375] The user inputs information about ingredients in the refrigerator into a terminal. The terminal can be a smartphone or tablet. Specific examples of such devices include smartphones and tablets. For example, the user inputs information about ingredients such as "200g of chicken, 3 tomatoes, 1 lettuce, expiration date 2022-01-15."

[0376] Sending and saving food information

[0377] The device sends the entered ingredient information to the server via Wi-Fi or mobile data. This is done using the HTTP POST method to send data in JSON format. The server analyzes the received ingredient information and stores it in a database (e.g., MySQL or PostgreSQL).

[0378] Enter and submit health and meal information

[0379] Users also enter their own health information and their children's school lunch information into the device. Health information might include, for example, "I want to get more vitamin C," and school lunch information might include data such as "Monday: curry rice, Tuesday: fried fish." This information is also sent to the server via the device and stored in a database.

[0380] Recognition and transmission of emotional information

[0381] The emotion engine recognizes emotions from the user's voice and input information. For example, Google® Cloud Speech-to-Text API is used for voice recognition. When a user says, "I'm tired today," the emotion engine converts this into text and recognizes it as "fatigue." This information is sent to the server and stored in a database. This allows the user's emotional state to be reflected in the menu generation process.

[0382] Menu generation and display

[0383] The server queries and obtains information on ingredients, health, school meals, and emotions from the database. Based on the obtained information, a menu generation algorithm written in Python runs and generates an optimal menu taking into account nutritional balance, the amount of ingredients used, avoidance of overlaps with school meals, health conditions, and the user's emotional state. For example, if the ingredients in the refrigerator are "chicken, tomato, and lettuce," and the user says "I want to get more vitamin C," and their emotion is recognized as "fatigue," the menu suggested would be "chicken stewed in tomato sauce," "lettuce salad," and "orange."

[0384] The server sends the generated menu information to the device and displays it to the user via a dedicated app on the device. The user can then check the proposed menu and use it in their daily meal planning.

[0385] Examples of prompt statements

[0386] Next, as a concrete example, the following prompt sentence could be input to a generative AI model:

[0387] I have chicken, tomatoes, and lettuce in my fridge. I'd also like to get more vitamin C. I'm also feeling tired today. Could you recommend a meal plan for me?

[0388] Based on this prompt, the system generates a menu suggesting "chicken stewed in tomato sauce," "lettuce salad," and "orange."

[0389] This system allows users to maintain a healthy and balanced diet while making the most of the ingredients in their refrigerator. It also takes into account their children's school lunch information, health information, and even their emotional state, making it easy to create optimal meal plans for the whole family.

[0390] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0391] Step 1:

[0392] The user inputs information about the ingredients in the refrigerator into the terminal. For example, the input might be "200g of chicken, 3 tomatoes, 1 lettuce, expiration date 2022-01-15." The terminal receives this information and stores it in a data structure. Specifically, this is done using an application on a smartphone or tablet. When the user has finished entering information, they press the send button.

[0393] Step 2:

[0394] The device sends the entered ingredient information to the server. The input is a data structure, and the output is data converted to JSON format. This uses the HTTP POST method. Specifically, the device sends the data to the appropriate URL endpoint. After sending, the device waits for a successful response from the server.

[0395] Step 3:

[0396] The server analyzes the received ingredient information and saves it in a database. The input is JSON format data, which is converted into an internal data structure. The converted data is then stored in a database (e.g., MySQL or PostgreSQL). Specifically, it executes an SQL INSERT statement.

[0397] Step 4:

[0398] The user inputs information about their chronic illnesses and the nutrients they want to take into the device. For example, "I want to take in a lot of vitamin C." The device receives this information and stores it in a data structure. Specific operations include using a form for health information.

[0399] Step 5:

[0400] The device sends the health information to the server. Again, it is converted to JSON format and uses the HTTP POST method. The input is a data structure and the output is JSON formatted data. The device sends the data to the appropriate URL endpoint. If the transmission is successful, the data is stored on the server.

[0401] Step 6:

[0402] The server stores health information in a database. The input data is in JSON format and is stored in the database after parsing. Specifically, the data is saved using the SQL INSERT statement.

[0403] Step 7:

[0404] The user inputs the child's school lunch information into the terminal. For example, "Monday: curry rice, Tuesday: fried fish." The terminal receives the information and stores it in a data structure. Specific operations involve the use of a dedicated form.

[0405] Step 8:

[0406] The device sends the meal information to the server. As with the health information, it is converted to JSON format and the HTTP POST method is used. The input is a data structure and the output is JSON format data. The device sends the data to the server's URL endpoint and waits for a successful response.

[0407] Step 9:

[0408] The server saves the school lunch information in a database. It parses the JSON format data as input and stores it in the database. The specific operation is to execute an SQL INSERT statement.

[0409] Step 10:

[0410] The emotion engine recognizes emotions from the user's voice and input information. The input is the user's voice data, and natural language processing algorithms are applied. As a specific example, Google Cloud Speech-to-Text API is used for voice recognition. The detected emotion information is converted into JSON format.

[0411] Step 11:

[0412] The emotion engine sends the recognized emotion information to the server. The input is emotion data, and the output is JSON format data. The HTTP POST method is used to send it to the server's URL endpoint. The success of the transmission is confirmed.

[0413] Step 12:

[0414] The server saves the emotion information in a database. It parses the JSON format emotion data as input and stores it in the database. The specific operation is to execute an SQL INSERT statement.

[0415] Step 13:

[0416] The server queries and retrieves information about ingredients, meals, health, and emotions from the database. The input is an SQL query, and the output is the corresponding dataset. Specifically, multiple SELECT statements are used.

[0417] Step 14:

[0418] The server runs a menu generation algorithm based on the acquired information. The input data is ingredient information, school lunch information, health information, and emotional information, and the output is the generated menu information. The algorithm is implemented in Python and makes appropriate meal suggestions.

[0419] Step 15:

[0420] The server sends the generated menu information to the terminal. Here too, it is converted to JSON format and the HTTP POST method is used. The input is the menu data, and the output is JSON format data. It is sent to the terminal's URL endpoint.

[0421] Step 16:

[0422] The device then displays the received menu information to the user. The input is menu data in JSON format, which is processed by the application. Specifically, the menu information is visually presented using a dedicated interface. The user can then review the suggested menu and use it to plan their daily meals.

[0423] (Application example 2)

[0424] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0425] Conventional menu generation systems were unable to propose menus that comprehensively took into account information about ingredients in the user's refrigerator, health information, and even emotional information. Furthermore, ordering additional ingredients when necessary ingredients were in short supply was a hassle. This made it difficult for users to maintain a healthy diet and to refresh themselves psychologically. Furthermore, users tended to rely on manual input, which made managing each piece of information cumbersome.

[0426] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0427] In this invention, the server includes means for storing information about ingredients in the user's refrigerator in a database, means for generating a nutritionally balanced menu taking into account the user's health information and emotional information, means for ordering ingredients that are in short supply, and means for displaying the generated menu information on a terminal. This allows the user to receive optimal menu suggestions that reflect the user's health information and emotional information, and allows them to easily order additional ingredients even if they are in short supply, enabling them to lead a healthy and stress-reducing diet.

[0428] The "ingredient information input means" is a device or system that allows a user to input information such as the type, quantity, and expiration date of ingredients in the refrigerator and store it in a database.

[0429] The "health information input means" is a device or system that allows a user to input information about chronic illnesses and particular nutritional intake, and store the information in a database.

[0430] The "emotion information recognition means" is a device or system for recognizing a user's emotion from the user's voice or input information and storing it in a database.

[0431] The "menu generation means" is a device or system that synthesizes ingredient information, health information, and emotional information to generate an optimal menu that takes into consideration nutritional balance and the user's psychological state.

[0432] The "ingredient ordering means" is a device or system for ordering additional ingredients when there are insufficient ingredients required for the generated menu.

[0433] The "menu display means" is a device or system for displaying menu information generated by the server on the user's terminal.

[0434] The "school lunch information input means" is a device or system for inputting school lunch menu information for the home and storing it in a database.

[0435] The "chronic disease information storage means" is a device or system for storing information about a user's chronic disease in a database.

[0436] The "emotion information storage means" is a device or system for storing the recognized emotion information in a database.

[0437] The system for implementing this invention creates an optimal menu based on information about ingredients, health, and emotions in the user's refrigerator, and orders any ingredients that are missing. Specifically, the system uses the following hardware and software:

[0438] 1. Enter refrigerator food information

[0439] The user inputs information about ingredients in the refrigerator using a device such as a smartphone. This input information includes the type of ingredient, quantity, expiration date, etc. The device sends this information to the server, where it is stored in a database.

[0440] 2. Enter your health information

[0441] Users input information about their chronic illnesses and the nutrients they particularly want to take into account through a device such as a smartphone. This input information includes the nutrients the user wants to take and information about their chronic illness. The device sends this information to a server and stores it on a data bus on the server.

[0442] 3. Recognition of emotional information

[0443] The server uses the voice recognition function or camera of the smartphone or other device to collect emotional information about the user, including, for example, fatigue and stress levels. The device sends this information to the server and stores it on a data bus on the server.

[0444] 4. Menu generation

[0445] The server retrieves ingredient, health, and emotional information from the database and generates an optimal menu based on that information. The menu generation algorithm takes into account nutritional balance, ingredient amounts, health conditions, and emotional state. For example, if the user is feeling tired, it will generate a menu that includes ingredients with a relaxing effect.

[0446] 5. Ordering missing ingredients

[0447] If a required ingredient is in short supply when creating a menu, the server automatically places an additional order through the food delivery system, allowing users to have the ingredients they need delivered to them.

[0448] 6. Display of menu information

[0449] The menus generated by the server are sent to a device such as a smartphone and displayed to the user, allowing the user to easily check and implement their daily meal plan.

[0450] Specific examples

[0451] Specifically, use the following prompt:

[0452] I have chicken, tomatoes, and lettuce in my refrigerator. I'd like to get more vitamin C. I'm currently feeling fatigued. Based on this information, please suggest a meal plan that will be healthy and help reduce stress.

[0453] In this way, this invention realizes a system that integrates information on ingredients in the refrigerator, health information, and emotional information, creates nutritionally balanced menus, and allows users to easily order the necessary ingredients, enabling users to lead healthy and stress-reducing eating habits.

[0454] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0455] Step 1:

[0456] Enter ingredient information

[0457] The user inputs the type, quantity, and expiration date of ingredients in the refrigerator into a device such as a smartphone. The device sends this information to a server, which then stores the received ingredient information in a database. The input information might include, for example, "2 pieces of chicken," "3 tomatoes," "1 lettuce," and "expiration date 2023-10-10."

[0458] (Input: ingredient information, Output: ingredient information stored in the database)

[0459] Step 2:

[0460] Enter health information

[0461] Users input information about the nutrients they particularly want to consume and any chronic illnesses they have into a device such as a smartphone. The device then sends this information to a server, which then stores the received information in a database. Specific examples include information such as "I want to take in a lot of vitamin C" and "I have no chronic illnesses."

[0462] (Input: health information, Output: health information stored in the database)

[0463] Step 3:

[0464] emotional information recognition

[0465] The emotion engine uses the smartphone's voice recognition function and camera to recognize the user's emotions. For example, it can detect "fatigue" from the user's voice and facial expressions. The device sends this information to the server, which then stores it in a database.

[0466] (Input: Emotion information, Output: Emotion information stored in the database)

[0467] Step 4:

[0468] Database queries

[0469] The server queries and retrieves stored ingredient information, health information, and emotional information from the database. This aggregates the information set necessary for menu generation. For example, information such as "chicken, tomato, lettuce," "high intake of vitamin C," and "feeling tired" is retrieved.

[0470] (Input: database query, Output: retrieved information set)

[0471] Step 5:

[0472] Menu generation

[0473] The server then uses a generative AI model to generate an appropriate menu based on the acquired information set. The menu generation algorithm takes into account nutritional balance, the amount of ingredients used, health conditions, and emotional state. For example, it might suggest dishes such as chicken stewed in tomato sauce, which has a relaxing effect, lettuce salad, and oranges, which are rich in vitamin C.

[0474] (Input: information set, output: generated menu)

[0475] Step 6:

[0476] Ordering missing ingredients

[0477] The server checks whether the ingredients required for the generated menu are missing, and if so, automatically places an additional order through the food delivery system, for example, ordering an "orange."

[0478] (Input: Generated menu, check for missing ingredients, Output: Order request)

[0479] Step 7:

[0480] Display menu information

[0481] The server sends the generated menu and ordering information for missing ingredients to a device such as a smartphone, where the user can check the proposed menu and added order details on the device.

[0482] (Input: generated menu, order information, Output: information displayed on the terminal)

[0483] Through these steps, the system can integrate the user's ingredients, health status, and emotional state to generate an optimal menu and automatically order any missing ingredients.

[0484] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0485] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0486] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0487] [Second embodiment]

[0488] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0489] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0490] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0491] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0492] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0493] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0494] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0495] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0496] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0497] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0498] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0499] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0500] MODE FOR CARRYING OUT THE INVENTION

[0501] This invention is a system that automatically generates healthy menus based on information about ingredients in the refrigerator. The system is composed of three parties: the user, the terminal, and the server, and creates menus based on information stored in a database.

[0502] Enter ingredient information

[0503] First, the user inputs information about the ingredients in the refrigerator into the terminal. Specifically, the type of ingredient, quantity, expiration date, etc. are input. After confirming this information, the terminal sends it to the server. The server stores the received information in a database. The stored data is later used when creating menus.

[0504] Enter your child's school lunch information (optional)

[0505] For families with children, users input their children's school lunch information into the device. This information, like the food ingredient information, is sent to the server and stored in the database. The school lunch information includes the school lunch menu and its nutritional value. This information makes it possible to ensure that the breakfast and dinner menus do not overlap with those of school lunches.

[0506] Enter health information (optional)

[0507] If a user has a specific chronic illness or has specific nutrients they want to consume, they can enter that information into the device. This information is also sent to the server and stored in a database. For example, if a user wants to consume a lot of vitamin C, foods that fall into that category will be prioritized.

[0508] Menu generation

[0509] The server integrates information on ingredients, school meals, and health stored in a database to create menus that take into consideration nutritional balance, the amount of ingredients used, avoidance of overlaps with school meals, etc. The menu generation algorithm calculates the optimal meal plan that meets these conditions and proposes specific menus.

[0510] As a concrete example, consider a case where the refrigerator contains "chicken, tomatoes, and lettuce." If the user also inputs information such as "I want to consume a lot of vitamin C," the server can use these conditions to suggest "chicken stewed in tomato sauce" and "lettuce salad" for dinner. If the user has also input information about their child's school lunches, the server will adjust the suggestions to avoid overlapping with that information.

[0511] Display menu information

[0512] The menus generated by the server are sent to the device and displayed to the user, who can then check the suggested menus and incorporate them into their daily meal plans.

[0513] This system allows users to maintain a healthy and balanced diet while making effective use of the ingredients in their refrigerator. It also makes it easy to create optimal meal plans for the whole family by taking into account their children's school lunch information and health information.

[0514] The processing flow will be explained below.

[0515] Step 1:

[0516] The user inputs information about the ingredients in the refrigerator into the terminal, such as the type, quantity, and expiration date of the ingredients, and specific data is provided to the terminal.

[0517] Step 2:

[0518] The terminal checks the ingredient information entered by the user to ensure there are no errors, and then prepares to send this information to the server.

[0519] Step 3:

[0520] The device sends the ingredient information to the server as an HTTP request. The request includes all the ingredient information entered.

[0521] Step 4:

[0522] The server analyzes the received request, extracts the ingredient information, and stores it in the food_inventory table in the database.

[0523] Step 5:

[0524] The user inputs their child's school lunch information into the terminal, and detailed information about the school lunch menu and nutritional value is provided.

[0525] Step 6:

[0526] The terminal checks the lunch information to make sure there are no errors, and then prepares to send this information to the server.

[0527] Step 7:

[0528] The terminal sends the school lunch information to the server as an HTTP request. The request includes the entered school lunch menu information.

[0529] Step 8:

[0530] The server analyzes the received request, extracts the school meal information, and saves it in the school_meals table in the database.

[0531] Step 9:

[0532] The user inputs information about a specific chronic illness or the nutrients they want to consume into the device. For example, they input information such as "I want to consume a lot of vitamin C."

[0533] Step 10:

[0534] The device checks the health information to make sure there are no errors, and then prepares to send this information to the server.

[0535] Step 11:

[0536] The device sends the health information to the server as an HTTP request, which includes the entered health information.

[0537] Step 12:

[0538] The server analyzes the received request, extracts the health information, and stores it in the health_requirements table in the database.

[0539] Step 13:

[0540] The server queries the database to obtain information on ingredients, meal plans, and health information, which is then used as input data for the menu generation algorithm.

[0541] Step 14:

[0542] The server runs a menu generation algorithm to generate a menu that takes into consideration nutritional balance, the amount of ingredients used, avoidance of overlap with school lunches, health requirements, etc.

[0543] Step 15:

[0544] The server compiles the generated menu information and sends it to the terminal as an HTTP response, which includes the specific menu and its nutritional information.

[0545] Step 16:

[0546] The device analyzes the menu information received from the server and displays it in a user-friendly format. The user can then check the proposed menu through the device.

[0547] Example 1

[0548] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0549] Currently, there are applications and systems on the market that suggest meal plans based on the ingredients in the refrigerator, but most of them do not take into account nutritional balance, individual health information, or information about school lunches at home, and as a result, they are unable to provide efficient and healthy meal plans. This poses the problem that users have to go to the trouble of planning their own meals and assembling menus that take health information into account. Furthermore, it is difficult to create menus that do not overlap with information about children's school lunches.

[0550] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0551] In this invention, the server includes a means for inputting information about ingredients in the refrigerator, a means for saving the ingredient information in a database, a means for generating a menu that takes nutritional balance into consideration based on the saved ingredient information, a means for generating a menu based on a prompt sentence using a generation AI model, and a means for displaying the generated menu information on a terminal. This makes it possible to effectively use ingredients in the refrigerator and automatically provide a well-balanced menu that takes into consideration the user's health information and children's school lunch information.

[0552] "Information about food items in the refrigerator" refers to attribute information such as the type, quantity, and expiration date of the food items stored in the refrigerator.

[0553] "Database" means an organized collection of data for efficient storage, management, and retrieval of information.

[0554] "Nutritional balance" refers to a state in which the food you eat contains nutrients such as protein, lipids, carbohydrates, vitamins, and minerals in appropriate proportions.

[0555] "Menu" refers to the combination of menu items for a particular meal.

[0556] A "generative AI model" refers to an algorithm or system that uses AI (artificial intelligence) technology to make predictions or generate information based on text or data.

[0557] "Prompt sentence" refers to an instruction sentence input to a generative AI model to cause it to generate a specific output.

[0558] "Device" refers to an electronic device used by a user to input information or view output information. Examples include smartphones and personal computers.

[0559] "School lunch menu information" refers to information about the menu and nutritional value of meals provided to children at school or other facilities.

[0560] "Information about chronic conditions and specific nutritional needs" refers to information about nutrients that a user wishes to consume or avoid based on a user's specific health condition or goals.

[0561] "Means for inputting information about ingredients in the refrigerator" refers to an interface or method that allows a user to input attribute information about ingredients in the refrigerator into the system.

[0562] "Storage means" refers to a method or system for storing input information in a database.

[0563] "Generating means" refers to the methods or techniques used to create a particular output (e.g., a menu or recipe) based on the stored information.

[0564] "Means for displaying" refers to the method or interface for conveying the generated information to the user.

[0565] MODE FOR CARRYING OUT THE INVENTION

[0566] This invention is a system that automatically generates healthy menus based on information about ingredients in a refrigerator. This system is composed of three parties: a user, a terminal, and a server. The specific processing steps are explained below.

[0567] Enter and save ingredient information

[0568] First, the user enters information about the ingredients in the refrigerator into the terminal. Specifically, the user enters information such as the type of ingredient (e.g., chicken, tomato, lettuce), quantity, and expiration date. The terminal uses a common interface, such as a smartphone or PC. The terminal checks the entered information and sends it to the server as JSON-formatted data. The server stores this information in a database system such as MySQL or PostgreSQL.

[0569] Enter your child's school lunch information (optional)

[0570] If the user has information about school lunches for their children at home, they can also enter this information into the device. The school lunch menu and its nutritional value are entered, and the device confirms this and sends it to the server. This information is also saved in the database. For example, by saving detailed information such as "Today's school lunch is curry rice (calories: 450kcal, vitamin C: 10mg)," the server can use this information to generate menus later.

[0571] Enter health information (optional)

[0572] Furthermore, if a user has a specific chronic illness or has specific nutrients they want to consume, they can enter that information through the device. For example, they can enter information such as "I want to consume a lot of vitamin C." After confirming this, the device sends it to the server, which stores it in a database. By storing this health information, it is given priority when generating menus.

[0573] Automatic menu generation

[0574] The server generates menus based on the ingredient information, school meal information, and health information stored in the database, taking into consideration factors such as nutritional balance, ingredient amounts, and avoidance of overlaps with school meals. This process uses a common AI model, such as GPT-4, as a generative AI model. To generate a menu that meets specific conditions, a prompt is created and sent to the AI ​​model.

[0575] For example, use the following prompt:

[0576] I have the following ingredients in my fridge:

[0577] 500g chicken

[0578] 3 tomatoes

[0579] 1 head of lettuce

[0580] I would also like to get a lot of vitamin C. Could you please suggest a suitable menu for tonight's dinner?

[0581] (If there is information about children's school lunches)

[0582] Today's school lunch includes curry rice (calories: 450kcal, vitamin C: 10mg). Please make sure that the menu does not overlap with the school lunch menu.

[0583] Based on this prompt, the generative AI model generates a specific menu, such as "chicken stewed in tomato sauce" and "lettuce salad."

[0584] Display menu information

[0585] The menu information generated by the server is sent to the device, which receives the information and displays it in a visually easy-to-understand format for the user. The user can then review the proposed menu and incorporate it into their daily meal plan.

[0586] This system allows users to easily achieve a healthy and balanced diet while making effective use of the ingredients in their refrigerator. It also takes into account individual children's school lunch information and health information, making it possible to automatically provide optimal meal plans for the whole family.

[0587] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0588] Program processing flow

[0589] Step 1: Enter ingredient information

[0590] 1. The user enters the ingredients information into the terminal.

[0591] The user inputs the type, quantity, and expiration date of ingredients in the refrigerator into the terminal.

[0592] Input data: type of ingredients (e.g. chicken, tomato, lettuce), quantity, expiration date

[0593] 2. The device checks the entered information

[0594] The terminal automatically checks the input for errors.

[0595] Data processing: Check if the quantity field is a number and if the expiration date is in the future.

[0596] Output data: Ingredient information after verification (valid data)

[0597] 3. The device sends the information to the server

[0598] After checking the entered ingredient information, it is sent to the server in JSON format.

[0599] Data processing: Converting ingredient information into JSON format

[0600] Output data: Ingredient information in JSON format

[0601] Step 2: Save the ingredients

[0602] 1. The server receives the information

[0603] The server receives the ingredient information sent from the terminal.

[0604] Input data: JSON formatted ingredient information

[0605] 2. The server stores the information in a database

[0606] The server stores the received ingredient information in an appropriate database system (e.g., MySQL, PostgreSQL).

[0607] Data processing: Convert JSON format data into database format

[0608] Output data: Ingredient information stored in a database

[0609] Step 3: Enter school lunch information (optional)

[0610] 1. The user inputs the school lunch menu information into the terminal.

[0611] The user inputs the menu and nutritional value of the school lunch that the child will have at school into the terminal.

[0612] Input data: School lunch menu, nutritional value (e.g. curry rice, calories: 450kcal, vitamin C: 10mg)

[0613] 2. The device checks the information and sends it to the server

[0614] The terminal checks the input information for errors and sends it to the server in JSON format.

[0615] Data processing: Convert input information into JSON format

[0616] Output data: JSON format school lunch information

[0617] 3. The server stores the information in a database

[0618] The server stores the received meal information in a database.

[0619] Input data: JSON format school lunch information

[0620] Output data: Meal information stored in a database

[0621] Step 4: Enter your health information (optional)

[0622] 1. The user enters health information into the device

[0623] Users input information about specific medical conditions and the nutrients they want to consume into the device.

[0624] Input data: Health information (e.g., I want to take in more vitamin C)

[0625] 2. The device checks the information and sends it to the server

[0626] The device reviews the input information for errors and sends it to the server in JSON format.

[0627] Data processing: Convert input information into JSON format

[0628] Output data: Health information in JSON format

[0629] 3. The server stores the information in a database

[0630] The server stores the received health information in a database.

[0631] Input data: Health information in JSON format

[0632] Output data: Health information stored in a database

[0633] Step 5: Automatic menu generation

[0634] 1. The server retrieves the necessary information from the database

[0635] The server retrieves food ingredient information, school lunch information, and health information from the database.

[0636] Input data: Various information stored in the database

[0637] Output data: Consolidated information list

[0638] 2. The server generates a prompt using the generative AI model.

[0639] The server creates a prompt based on the information obtained.

[0640] Data processing: Converting information lists into prompt statements

[0641] Output data: Generated prompt statement

[0642] Examples:

[0643] I have the following ingredients in my fridge:

[0644] 500g chicken

[0645] 3 tomatoes

[0646] 1 head of lettuce

[0647] I would also like to get a lot of vitamin C. Could you please suggest a suitable menu for tonight's dinner?

[0648] (If there is information about children's school lunches)

[0649] Today's school lunch includes curry rice (calories: 450kcal, vitamin C: 10mg). Please make sure that the menu does not overlap with the school lunch menu.

[0650] 3. The server sends a prompt to the generative AI model (e.g., GPT-4).

[0651] The server sends the prompt to the AI ​​model, asking it to generate a menu.

[0652] Input data: Prompt statement

[0653] Output data: Generated menu information

[0654] 4. The generative AI model returns a response, which the server receives.

[0655] Receive menu information generated from an AI model.

[0656] Input data: AI model response

[0657] Output data: Menu information (e.g., chicken stewed in tomato sauce, lettuce salad)

[0658] Step 6: Display menu information

[0659] 1. The server sends the generated menu information to the terminal.

[0660] The server sends the generated menu information to the terminal in JSON format.

[0661] Input data: Generated menu information

[0662] Output data: Menu information in JSON format

[0663] 2. The device analyzes the menu information received from the server.

[0664] The terminal parses the JSON formatted data and converts it into a format that is displayed to the user.

[0665] Data processing: Convert JSON format data into a data format for visualization

[0666] Output data: Visualized menu information

[0667] 3. The device displays the menu information to the user.

[0668] The device displays visualized menu information to the user, who can then review the suggested menu and incorporate it into their daily meal planning.

[0669] Input data: Visualized menu information

[0670] Output data: Menu information displayed to the user

[0671] This detailed process flow allows users to efficiently utilize the ingredients in their refrigerator and achieve a healthy and balanced diet.

[0672] (Application example 1)

[0673] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0674] In today's busy households, it is an important challenge to efficiently utilize the ingredients stored in the refrigerator and easily generate nutritionally balanced menus. To address this challenge, there is a demand for menu generation that takes into account health information and information about meals provided at home. However, current systems are unable to fully meet these demands. In particular, there is no system that allows users to easily input information about ingredients purchased at stores, which requires the user to enter the information manually. Furthermore, since conventional menu generation systems have difficulty taking into account health information and meal information, there is a demand for a system that can integrate such information and generate optimal menus.

[0675] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0676] In this invention, the server includes a means for inputting information about ingredients stored in the refrigerator, a means for saving the ingredient information in a database, a means for generating a nutritionally balanced menu based on the saved ingredient information, a means for scanning purchased ingredients and inputting information, and a means for displaying the generated menu information on a terminal. This allows the user to easily input ingredients purchased at the store and easily generate an optimal nutritionally balanced menu. In addition, by taking into account menu information and health information for school lunches at home and avoiding duplication, the health of the entire family can be supported.

[0677] "Information about ingredients in the refrigerator" refers to detailed information such as the type, quantity, and expiration date of ingredients that the user has in the refrigerator.

[0678] "Means for saving to a database" refers to the process and system for storing the input information in a database on a server so that it can be referenced later.

[0679] The "means for generating a menu that takes nutritional balance into consideration" refers to an algorithm and system that automatically creates a menu with optimal nutritional balance based on stored ingredient information, the user's health condition, and nutritional needs.

[0680] "Means of scanning purchased ingredients and entering information" refers to a function that automatically enters information about ingredients by scanning the barcode or QR code of ingredients purchased at a store using a smartphone or other device.

[0681] The "means for displaying the generated menu information on the terminal" is a system that displays the menu generated by the server on the user's terminal such as a smartphone or tablet, making it available for the user to refer to.

[0682] "Information about school lunch menus at home" refers to information about the menus and nutritional components of school lunches that children in the home eat at school or other places.

[0683] "Information about chronic illnesses and particular nutrients you want to take" refers to information about specific health problems or specific nutrients you want to take in more of.

[0684] A "generative AI model" is an artificial intelligence model used to suggest optimal menus and recipes based on user input data.

[0685] A "prompt sentence" is a sentence-based instruction provided to a generative AI model as input data for generating a menu.

[0686] This invention is a system that automatically generates menus that take into consideration the user's health status and nutritional balance based on information about ingredients in a refrigerator. The system is mainly composed of three entities: a server, a terminal, and a user. Specific embodiments are described below.

[0687] Entering ingredient information

[0688] The user inputs information about ingredients in the refrigerator into a device such as a smartphone. This information includes the type, quantity, and expiration date. The device confirms the information entered by the user and then sends it to the server. The server stores the received information in a database and uses it later when creating menus.

[0689] Enter purchased food information

[0690] When a user purchases new ingredients at a store, they can quickly enter ingredient information by scanning the barcode or QR code using the smartphone camera. This information is also sent to the server and stored in the database.

[0691] Entering health and child meal information

[0692] If a user has health problems or special nutritional needs, they can enter that information into the device. If they have children at home, they can also enter information about their school lunches. This information is also sent to the server and stored in a database.

[0693] Menu generation

[0694] The server uses a generative AI model to automatically generate optimal menus based on stored ingredient information, health information, and school lunch information. The generated menus take into consideration nutritional balance, ingredient usage efficiency, and avoidance of overlap with school lunches. The generated menus are then evaluated for nutritional value using the Nutritics API.

[0695] Specifically, the server uses the following methods:

[0696] A way to input information about ingredients in the refrigerator

[0697] A means for storing the ingredient information in a database

[0698] A method for generating nutritionally balanced menus based on stored ingredient information

[0699] A way to scan purchased ingredients and enter information

[0700] A method for generating menus based on health information using a generative AI model

[0701] This allows users to easily input ingredients purchased at the store and generate optimal menus that take nutritional balance into consideration. It also takes into account information about school lunches and health information at home, helping to support the health of the entire family.

[0702] Display menu information

[0703] The generated menu information is sent from the server to the user's device and displayed on their smartphone, tablet, etc. Users can check it and incorporate it into their daily meal plans.

[0704] Specific examples

[0705] For example, if a user purchases "chicken, tomato, lettuce, and carrot" at a supermarket and scans each item with a smartphone app, and the user enters "high in vitamin C" as health information and "curry (low in vitamins)" as their child's school lunch information, the app will automatically suggest a menu of "chicken and tomato stew, lettuce and carrot salad (high in vitamin C)."

[0706] Example prompts to input to the generative AI model

[0707] Consider the user's health condition and generate a menu rich in Vitamin C.

[0708] Based on information about your child's school lunch, please suggest healthy menus that do not use duplicate ingredients.

[0709] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0710] Step 1:

[0711] The user inputs information about ingredients in the refrigerator into the terminal. Specifically, the type of ingredient, quantity, and expiration date are entered, and the information is sent from the terminal to the server. At this time, the input information is converted into a data format such as JSON.

[0712] input:

[0713] Food type

[0714] Ingredient quantity

[0715] expiration date

[0716] output:

[0717] JSON formatted ingredient information

[0718] Step 2:

[0719] The server receives the ingredient information sent from the terminal and stores it in a database, where the ingredient information is recorded in a table format.

[0720] input:

[0721] JSON formatted ingredient information

[0722] output:

[0723] Ingredient information stored in the database

[0724] Step 3:

[0725] Users scan ingredients purchased at the store using the device's camera and enter the information. By reading barcodes and QR codes, the product name, nutritional information, expiration date, etc. are automatically obtained and sent to the server.

[0726] input:

[0727] Barcode / QR code

[0728] output:

[0729] Purchased ingredients information in JSON format

[0730] Step 4:

[0731] The server receives the information about the purchased ingredients and stores it in an existing database. The purchased ingredient information is also managed as an integrated ingredient information.

[0732] input:

[0733] Purchased ingredients information in JSON format

[0734] output:

[0735] Ingredient information stored in a database (integrated with existing databases)

[0736] Step 5:

[0737] The user inputs specific health information and information about their child's school lunch into the terminal. Health information includes specific nutrient intake preferences and information about chronic illnesses, while child's school lunch information includes the school lunch menu and its nutritional components. This information is also sent to the server.

[0738] input:

[0739] health information

[0740] School lunch information

[0741] output:

[0742] Health and meal information in JSON format

[0743] Step 6:

[0744] The server receives the health information and feeding information and stores it in a database, so that all necessary information is collected in the database.

[0745] input:

[0746] Health and meal information in JSON format

[0747] output:

[0748] Health and feeding information stored in a database

[0749] Step 7:

[0750] The server uses a generative AI model to generate optimal menus based on stored information on ingredients, health, and school lunches. It inputs prompt statements into the generative AI model to calculate nutritionally balanced and efficient menus, and evaluates the nutritional value of the generated menus using the Nutritics API.

[0751] input:

[0752] Food ingredient information, health information, and school lunch information stored in the database

[0753] Prompt statement

[0754] output:

[0755] Optimal menu information

[0756] Specific prompt examples:

[0757] Consider the user's health condition and generate a menu rich in Vitamin C.

[0758] Step 8:

[0759] The generated menu information is sent from the server to the user's device and displayed on their smartphone, tablet, etc. The user can then review it and incorporate it into their daily meal plan.

[0760] input:

[0761] Optimal menu information

[0762] output:

[0763] Menu information displayed on the device

[0764] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0765] MODE FOR CARRYING OUT THE INVENTION

[0766] This invention is a system that generates healthy menus based on information about ingredients in the refrigerator and also takes into account the user's emotional information. This system is composed of a user, a terminal, a server, and an emotion engine.

[0767] Enter ingredient information

[0768] The user inputs information about ingredients in the refrigerator into the terminal. Specifically, the user provides information such as the type of ingredient, quantity, and expiration date. The terminal sends this information to the server, which then stores the received ingredient information in a database. The stored ingredient information is used as basic data for creating menus.

[0769] Enter your child's school lunch information (optional)

[0770] Users also input their child's school lunch information into the device. This information, including details about the menu and nutritional value, is sent to the server and stored in a database. Based on this information, a menu that does not overlap with school lunches is generated.

[0771] Enter health information (optional)

[0772] Users enter information about any medical conditions or specific nutritional needs they have into the device. For example, if they want to increase their intake of vitamin C, they enter that information. This information is also sent to the server and stored in a database. The health information is used to manage illnesses and create diet plans that emphasize specific nutrients.

[0773] emotional information recognition

[0774] The emotion engine recognizes emotions from the user's voice and input information. For example, if the user is feeling stressed, it will detect that emotion. The emotion engine sends this information to the server and stores it in a database. The emotion information is used to create a menu that reflects the user's current psychological state.

[0775] Menu generation

[0776] The server queries and obtains information on ingredients, school meals, health, and emotions from the database. Based on this information, a menu generation algorithm operates, taking into account nutritional balance, the amount of ingredients used, avoidance of overlaps with school meals, health conditions, and emotional state to generate an optimal menu. For example, for a user who is feeling stressed, ingredients with a relaxing effect will be selected.

[0777] As a concrete example, consider a situation where the refrigerator contains "chicken, tomatoes, and lettuce." If the user inputs "I want to get more vitamin C," and the emotion engine also detects "fatigue," the server will use these conditions to suggest "chicken stewed in tomato sauce," "lettuce salad," and "oranges," which are rich in vitamin C.

[0778] Display menu information

[0779] The menus generated by the server are sent to the terminal and displayed to the user. The user can check the suggested menus and incorporate them into their daily meal plans. In particular, the system is expected to have the effect of refreshing the mind through meals by suggesting menus that take emotions into consideration.

[0780] This system allows users to maintain a healthy and balanced diet while making the most of the ingredients in their refrigerator. It also takes into account their children's school lunch information, health information, and even their emotional state, making it easy to create optimal meal plans for the whole family.

[0781] The processing flow will be explained below.

[0782] Step 1:

[0783] The user inputs information about ingredients in the refrigerator into the terminal, such as the type of ingredient (e.g., chicken, tomato, lettuce), quantity, and expiration date.

[0784] Step 2:

[0785] The terminal checks the entered ingredient information and checks for errors. After checking, it prepares to send this ingredient information to the server.

[0786] Step 3:

[0787] The device sends the ingredient information to the server as an HTTP request. The request includes all the ingredient information entered.

[0788] Step 4:

[0789] The server analyzes the received request, extracts the ingredient information, and stores it in the food_inventory table in the database.

[0790] Step 5:

[0791] The user inputs information about the child's school lunch into the terminal, including detailed information about the school lunch menu and nutritional value.

[0792] Step 6:

[0793] The terminal checks the entered school lunch information and checks for errors. After checking, the terminal prepares to send this school lunch information to the server.

[0794] Step 7:

[0795] The terminal sends the school lunch information to the server as an HTTP request. The request includes all the school lunch information that was entered.

[0796] Step 8:

[0797] The server analyzes the received request, extracts the school meal information, and saves it in the school_meals table in the database.

[0798] Step 9:

[0799] The user inputs information about their chronic illnesses and the nutrients they particularly want to take into the terminal. For example, they input information such as "I want to take in a lot of vitamin C."

[0800] Step 10:

[0801] The terminal checks the entered health information for errors, and then prepares to send the health information to the server.

[0802] Step 11:

[0803] The device sends the health information to the server as an HTTP request, which includes all the entered health information.

[0804] Step 12:

[0805] The server analyzes the received request, extracts the health information, and stores it in the health_requirements table in the database.

[0806] Step 13:

[0807] The emotion engine recognizes emotions from user input and voice. If the user is feeling stressed, it will detect that emotional information.

[0808] Step 14:

[0809] The emotion engine sends the recognized emotion information to the server as an HTTP request. The request includes the recognized emotion information.

[0810] Step 15:

[0811] The server analyzes the received request and extracts emotional information. The extracted information is stored in the emotion_data table in the database.

[0812] Step 16:

[0813] The server queries the database to obtain information on ingredients, meals, health, and emotions, and uses the information as input data for the menu generation algorithm.

[0814] Step 17:

[0815] The server runs a menu generation algorithm to generate a menu that takes into account nutritional balance, the amount of ingredients used, avoidance of overlaps with school meals, health conditions, and emotional state. For example, ingredients with a relaxing effect are selected for a stressed user.

[0816] Step 18:

[0817] The server compiles the generated menu information and sends it to the terminal as an HTTP response, which includes the specific menu and its nutritional information.

[0818] Step 19:

[0819] The device analyzes the menu information received from the server and displays it in a user-friendly format. The user can then check the proposed menu through the device.

[0820] This detailed process flow allows users to effectively use the ingredients in their refrigerator while creating meals that are in line with their health, nutritional balance, and even their own emotional state. It also makes it possible to suggest menus that do not overlap with children's school lunch information, providing optimal meal plans for the whole family.

[0821] Example 2

[0822] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0823] Conventional menu generation systems could take into account information about ingredients in the refrigerator and the user's health information, but they were unable to propose menus that took into account the user's emotional information. As a result, they were unable to provide more personalized menus that reflected the user's daily emotional fluctuations. Furthermore, even when generating menus that did not overlap or that emphasized specific nutrients, only limited information was taken into account, making it difficult to create optimal meal plans.

[0824] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0825] In this invention, the server includes a means for storing ingredient information in a database, a means for recognizing the user's emotional information, and a means for generating a menu that takes nutritional balance into consideration, thereby enabling the generation of a more personalized menu based on the ingredient information, health information, and emotional information in the user's refrigerator.

[0826] "Ingredient information" refers to data such as the type, quantity, and expiration date of food stored in the refrigerator.

[0827] A "database" is an information system for storing, managing, and retrieving data in an organized manner, and takes the form of a relational database management system (RDBMS).

[0828] The "means for generating a menu that takes nutritional balance into consideration" refers to an algorithm or program that calculates the balance of nutrients based on the input food ingredient information and other conditions and suggests the optimal meal.

[0829] A "terminal" is a device through which a user inputs information, and includes smartphones, tablets, personal computers, etc.

[0830] "Emotional information" is data that indicates the user's psychological state or mood, and is acquired through voice recognition, text input, or the like.

[0831] The "means for recognizing user's emotional information" is a combination of hardware and software for analyzing the user's voice and input data to identify the user's psychological state.

[0832] "Information on school lunch menus at home" refers to data on the menus and nutritional values ​​of meals provided to children at school or other facilities.

[0833] "Information about chronic illnesses and particularly desirable nutrients" is data about the health condition that the user wants to manage and the nutrients that the user wants to prioritize.

[0834] A "menu" is data that includes a meal menu and the ingredients and cooking procedures for serving it.

[0835] MODE FOR CARRYING OUT THE INVENTION

[0836] This invention is a system that generates healthy menus based on information about ingredients in a refrigerator and also takes into account the user's emotional information. This system is composed of a user, a terminal, a server, and an emotion engine.

[0837] Entering ingredient information

[0838] The user inputs information about ingredients in the refrigerator into a terminal. The terminal can be a smartphone or tablet. Specific examples of such devices include smartphones and tablets. For example, the user inputs information about ingredients such as "200g of chicken, 3 tomatoes, 1 lettuce, expiration date 2022-01-15."

[0839] Sending and saving food information

[0840] The device sends the entered ingredient information to the server via Wi-Fi or mobile data. This is done using the HTTP POST method to send data in JSON format. The server analyzes the received ingredient information and stores it in a database (e.g., MySQL or PostgreSQL).

[0841] Enter and submit health and meal information

[0842] Users also enter their own health information and their children's school lunch information into the device. Health information might include, for example, "I want to get more vitamin C," and school lunch information might include data such as "Monday: curry rice, Tuesday: fried fish." This information is also sent to the server via the device and stored in a database.

[0843] Recognition and transmission of emotional information

[0844] The emotion engine recognizes emotions from the user's voice and input information. For example, Google Cloud Speech-to-Text API is used for voice recognition. When a user says, "I'm tired today," the emotion engine converts this into text and recognizes it as "fatigue." This information is sent to the server and stored in a database. This allows the user's emotional state to be reflected in the menu generation process.

[0845] Menu generation and display

[0846] The server queries and obtains information on ingredients, health, school meals, and emotions from the database. Based on the obtained information, a menu generation algorithm written in Python runs and generates an optimal menu taking into account nutritional balance, the amount of ingredients used, avoidance of overlaps with school meals, health conditions, and the user's emotional state. For example, if the ingredients in the refrigerator are "chicken, tomato, and lettuce," and the user says "I want to get more vitamin C," and their emotion is recognized as "fatigue," the menu suggested would be "chicken stewed in tomato sauce," "lettuce salad," and "orange."

[0847] The server sends the generated menu information to the device and displays it to the user via a dedicated app on the device. The user can then check the proposed menu and use it in their daily meal planning.

[0848] Examples of prompt statements

[0849] Next, as a concrete example, the following prompt sentence could be input to a generative AI model:

[0850] I have chicken, tomatoes, and lettuce in my fridge. I'd also like to get more vitamin C. I'm also feeling tired today. Could you recommend a meal plan for me?

[0851] Based on this prompt, the system generates a menu suggesting "chicken stewed in tomato sauce," "lettuce salad," and "orange."

[0852] This system allows users to maintain a healthy and balanced diet while making the most of the ingredients in their refrigerator. It also takes into account their children's school lunch information, health information, and even their emotional state, making it easy to create optimal meal plans for the whole family.

[0853] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0854] Step 1:

[0855] The user inputs information about the ingredients in the refrigerator into the terminal. For example, the input might be "200g of chicken, 3 tomatoes, 1 lettuce, expiration date 2022-01-15." The terminal receives this information and stores it in a data structure. Specifically, this is done using an application on a smartphone or tablet. When the user has finished entering information, they press the send button.

[0856] Step 2:

[0857] The device sends the entered ingredient information to the server. The input is a data structure, and the output is data converted to JSON format. This uses the HTTP POST method. Specifically, the device sends the data to the appropriate URL endpoint. After sending, the device waits for a successful response from the server.

[0858] Step 3:

[0859] The server analyzes the received ingredient information and saves it in a database. The input is JSON format data, which is converted into an internal data structure. The converted data is then stored in a database (e.g., MySQL or PostgreSQL). Specifically, it executes an SQL INSERT statement.

[0860] Step 4:

[0861] The user inputs information about their chronic illnesses and the nutrients they want to take into the device. For example, "I want to take in a lot of vitamin C." The device receives this information and stores it in a data structure. Specific operations include using a form for health information.

[0862] Step 5:

[0863] The device sends the health information to the server. Again, it is converted to JSON format and uses the HTTP POST method. The input is a data structure and the output is JSON formatted data. The device sends the data to the appropriate URL endpoint. If the transmission is successful, the data is stored on the server.

[0864] Step 6:

[0865] The server stores health information in a database. The input data is in JSON format and is stored in the database after parsing. Specifically, the data is saved using the SQL INSERT statement.

[0866] Step 7:

[0867] The user inputs the child's school lunch information into the terminal. For example, "Monday: curry rice, Tuesday: fried fish." The terminal receives the information and stores it in a data structure. Specific operations involve the use of a dedicated form.

[0868] Step 8:

[0869] The device sends the meal information to the server. As with the health information, it is converted to JSON format and the HTTP POST method is used. The input is a data structure and the output is JSON format data. The device sends the data to the server's URL endpoint and waits for a successful response.

[0870] Step 9:

[0871] The server saves the school lunch information in a database. It parses the JSON format data as input and stores it in the database. The specific operation is to execute an SQL INSERT statement.

[0872] Step 10:

[0873] The emotion engine recognizes emotions from the user's voice and input information. The input is the user's voice data, and natural language processing algorithms are applied. As a specific example, Google Cloud Speech-to-Text API is used for voice recognition. The detected emotion information is converted into JSON format.

[0874] Step 11:

[0875] The emotion engine sends the recognized emotion information to the server. The input is emotion data, and the output is JSON format data. The HTTP POST method is used to send it to the server's URL endpoint. The success of the transmission is confirmed.

[0876] Step 12:

[0877] The server saves the emotion information in a database. It parses the JSON format emotion data as input and stores it in the database. The specific operation is to execute an SQL INSERT statement.

[0878] Step 13:

[0879] The server queries and retrieves information about ingredients, meals, health, and emotions from the database. The input is an SQL query, and the output is the corresponding dataset. Specifically, multiple SELECT statements are used.

[0880] Step 14:

[0881] The server runs a menu generation algorithm based on the acquired information. The input data is ingredient information, school lunch information, health information, and emotional information, and the output is the generated menu information. The algorithm is implemented in Python and makes appropriate meal suggestions.

[0882] Step 15:

[0883] The server sends the generated menu information to the terminal. Here too, it is converted to JSON format and the HTTP POST method is used. The input is the menu data, and the output is JSON format data. It is sent to the terminal's URL endpoint.

[0884] Step 16:

[0885] The device then displays the received menu information to the user. The input is menu data in JSON format, which is processed by the application. Specifically, the menu information is visually presented using a dedicated interface. The user can then review the suggested menu and use it to plan their daily meals.

[0886] (Application example 2)

[0887] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0888] Conventional menu generation systems were unable to propose menus that comprehensively took into account information about ingredients in the user's refrigerator, health information, and even emotional information. Furthermore, ordering additional ingredients when necessary ingredients were in short supply was a hassle. This made it difficult for users to maintain a healthy diet and to refresh themselves psychologically. Furthermore, users tended to rely on manual input, which made managing each piece of information cumbersome.

[0889] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0890] In this invention, the server includes means for storing information about ingredients in the user's refrigerator in a database, means for generating a nutritionally balanced menu taking into account the user's health information and emotional information, means for ordering ingredients that are in short supply, and means for displaying the generated menu information on a terminal. This allows the user to receive optimal menu suggestions that reflect the user's health information and emotional information, and allows them to easily order additional ingredients even if they are in short supply, enabling them to lead a healthy and stress-reducing diet.

[0891] The "ingredient information input means" is a device or system that allows a user to input information such as the type, quantity, and expiration date of ingredients in the refrigerator and store it in a database.

[0892] The "health information input means" is a device or system that allows a user to input information about chronic illnesses and particular nutritional intake, and store the information in a database.

[0893] The "emotion information recognition means" is a device or system for recognizing a user's emotion from the user's voice or input information and storing it in a database.

[0894] The "menu generation means" is a device or system that synthesizes ingredient information, health information, and emotional information to generate an optimal menu that takes into consideration nutritional balance and the user's psychological state.

[0895] The "ingredient ordering means" is a device or system for ordering additional ingredients when there are insufficient ingredients required for the generated menu.

[0896] The "menu display means" is a device or system for displaying menu information generated by the server on the user's terminal.

[0897] The "school lunch information input means" is a device or system for inputting school lunch menu information for the home and storing it in a database.

[0898] The "chronic disease information storage means" is a device or system for storing information about a user's chronic disease in a database.

[0899] The "emotion information storage means" is a device or system for storing the recognized emotion information in a database.

[0900] The system for implementing this invention creates an optimal menu based on information about ingredients, health, and emotions in the user's refrigerator, and orders any ingredients that are missing. Specifically, the system uses the following hardware and software:

[0901] 1. Enter refrigerator food information

[0902] The user inputs information about ingredients in the refrigerator using a device such as a smartphone. This input information includes the type of ingredient, quantity, expiration date, etc. The device sends this information to the server, where it is stored in a database.

[0903] 2. Enter your health information

[0904] Users input information about their chronic illnesses and the nutrients they particularly want to take into account through a device such as a smartphone. This input information includes the nutrients the user wants to take and information about their chronic illness. The device sends this information to a server and stores it on a data bus on the server.

[0905] 3. Recognition of emotional information

[0906] The server uses the voice recognition function or camera of the smartphone or other device to collect emotional information about the user, including, for example, fatigue and stress levels. The device sends this information to the server and stores it on a data bus on the server.

[0907] 4. Menu generation

[0908] The server retrieves ingredient, health, and emotional information from the database and generates an optimal menu based on that information. The menu generation algorithm takes into account nutritional balance, ingredient amounts, health conditions, and emotional state. For example, if the user is feeling tired, it will generate a menu that includes ingredients with a relaxing effect.

[0909] 5. Ordering missing ingredients

[0910] If a required ingredient is in short supply when creating a menu, the server automatically places an additional order through the food delivery system, allowing users to have the ingredients they need delivered to them.

[0911] 6. Display of menu information

[0912] The menus generated by the server are sent to a device such as a smartphone and displayed to the user, allowing the user to easily check and implement their daily meal plan.

[0913] Specific examples

[0914] Specifically, use the following prompt:

[0915] I have chicken, tomatoes, and lettuce in my refrigerator. I'd like to get more vitamin C. I'm currently feeling fatigued. Based on this information, please suggest a meal plan that will be healthy and help reduce stress.

[0916] In this way, this invention realizes a system that integrates information on ingredients in the refrigerator, health information, and emotional information, creates nutritionally balanced menus, and allows users to easily order the necessary ingredients, enabling users to lead healthy and stress-reducing eating habits.

[0917] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0918] Step 1:

[0919] Enter ingredient information

[0920] The user inputs the type, quantity, and expiration date of ingredients in the refrigerator into a device such as a smartphone. The device sends this information to a server, which then stores the received ingredient information in a database. The input information might include, for example, "2 pieces of chicken," "3 tomatoes," "1 lettuce," and "expiration date 2023-10-10."

[0921] (Input: ingredient information, Output: ingredient information stored in the database)

[0922] Step 2:

[0923] Enter health information

[0924] Users input information about the nutrients they particularly want to consume and any chronic illnesses they have into a device such as a smartphone. The device then sends this information to a server, which then stores the received information in a database. Specific examples include information such as "I want to take in a lot of vitamin C" and "I have no chronic illnesses."

[0925] (Input: health information, Output: health information stored in the database)

[0926] Step 3:

[0927] emotional information recognition

[0928] The emotion engine uses the smartphone's voice recognition function and camera to recognize the user's emotions. For example, it can detect "fatigue" from the user's voice and facial expressions. The device sends this information to the server, which then stores it in a database.

[0929] (Input: Emotion information, Output: Emotion information stored in the database)

[0930] Step 4:

[0931] Database queries

[0932] The server queries and retrieves stored ingredient information, health information, and emotional information from the database. This aggregates the information set necessary for menu generation. For example, information such as "chicken, tomato, lettuce," "high intake of vitamin C," and "feeling tired" is retrieved.

[0933] (Input: database query, Output: retrieved information set)

[0934] Step 5:

[0935] Menu generation

[0936] The server then uses a generative AI model to generate an appropriate menu based on the acquired information set. The menu generation algorithm takes into account nutritional balance, the amount of ingredients used, health conditions, and emotional state. For example, it might suggest dishes such as chicken stewed in tomato sauce, which has a relaxing effect, lettuce salad, and oranges, which are rich in vitamin C.

[0937] (Input: information set, output: generated menu)

[0938] Step 6:

[0939] Ordering missing ingredients

[0940] The server checks whether the ingredients required for the generated menu are missing, and if so, automatically places an additional order through the food delivery system, for example, ordering an "orange."

[0941] (Input: Generated menu, check for missing ingredients, Output: Order request)

[0942] Step 7:

[0943] Display menu information

[0944] The server sends the generated menu and ordering information for missing ingredients to a device such as a smartphone, where the user can check the proposed menu and added order details on the device.

[0945] (Input: generated menu, order information, Output: information displayed on the terminal)

[0946] Through these steps, the system can integrate the user's ingredients, health status, and emotional state to generate an optimal menu and automatically order any missing ingredients.

[0947] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0948] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0949] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0950] [Third embodiment]

[0951] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0952] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0953] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0954] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0955] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0956] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0957] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0958] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0959] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0960] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0961] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0962] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0963] MODE FOR CARRYING OUT THE INVENTION

[0964] This invention is a system that automatically generates healthy menus based on information about ingredients in the refrigerator. The system is composed of three parties: the user, the terminal, and the server, and creates menus based on information stored in a database.

[0965] Enter ingredient information

[0966] First, the user inputs information about the ingredients in the refrigerator into the terminal. Specifically, the type of ingredient, quantity, expiration date, etc. are input. After confirming this information, the terminal sends it to the server. The server stores the received information in a database. The stored data is later used when creating menus.

[0967] Enter your child's school lunch information (optional)

[0968] For families with children, users input their children's school lunch information into the device. This information, like the food ingredient information, is sent to the server and stored in the database. The school lunch information includes the school lunch menu and its nutritional value. This information makes it possible to ensure that the breakfast and dinner menus do not overlap with those of school lunches.

[0969] Enter health information (optional)

[0970] If a user has a specific chronic illness or has specific nutrients they want to consume, they can enter that information into the device. This information is also sent to the server and stored in a database. For example, if a user wants to consume a lot of vitamin C, foods that fall into that category will be prioritized.

[0971] Menu generation

[0972] The server integrates information on ingredients, school meals, and health stored in a database to create menus that take into consideration nutritional balance, the amount of ingredients used, avoidance of overlaps with school meals, etc. The menu generation algorithm calculates the optimal meal plan that meets these conditions and proposes specific menus.

[0973] As a concrete example, consider a case where the refrigerator contains "chicken, tomatoes, and lettuce." If the user also inputs information such as "I want to consume a lot of vitamin C," the server can use these conditions to suggest "chicken stewed in tomato sauce" and "lettuce salad" for dinner. If the user has also input information about their child's school lunches, the server will adjust the suggestions to avoid overlapping with that information.

[0974] Display menu information

[0975] The menus generated by the server are sent to the device and displayed to the user, who can then check the suggested menus and incorporate them into their daily meal plans.

[0976] This system allows users to maintain a healthy and balanced diet while making effective use of the ingredients in their refrigerator. It also makes it easy to create optimal meal plans for the whole family by taking into account their children's school lunch information and health information.

[0977] The processing flow will be explained below.

[0978] Step 1:

[0979] The user inputs information about the ingredients in the refrigerator into the terminal, such as the type, quantity, and expiration date of the ingredients, and specific data is provided to the terminal.

[0980] Step 2:

[0981] The terminal checks the ingredient information entered by the user to ensure there are no errors, and then prepares to send this information to the server.

[0982] Step 3:

[0983] The device sends the ingredient information to the server as an HTTP request. The request includes all the ingredient information entered.

[0984] Step 4:

[0985] The server analyzes the received request, extracts the ingredient information, and stores it in the food_inventory table in the database.

[0986] Step 5:

[0987] The user inputs their child's school lunch information into the terminal, and detailed information about the school lunch menu and nutritional value is provided.

[0988] Step 6:

[0989] The terminal checks the lunch information to make sure there are no errors, and then prepares to send this information to the server.

[0990] Step 7:

[0991] The terminal sends the school lunch information to the server as an HTTP request. The request includes the entered school lunch menu information.

[0992] Step 8:

[0993] The server analyzes the received request, extracts the school meal information, and saves it in the school_meals table in the database.

[0994] Step 9:

[0995] The user inputs information about a specific chronic illness or the nutrients they want to consume into the device. For example, they input information such as "I want to consume a lot of vitamin C."

[0996] Step 10:

[0997] The device checks the health information to make sure there are no errors, and then prepares to send this information to the server.

[0998] Step 11:

[0999] The device sends the health information to the server as an HTTP request, which includes the entered health information.

[1000] Step 12:

[1001] The server analyzes the received request, extracts the health information, and stores it in the health_requirements table in the database.

[1002] Step 13:

[1003] The server queries the database to obtain information on ingredients, meal plans, and health information, which is then used as input data for the menu generation algorithm.

[1004] Step 14:

[1005] The server runs a menu generation algorithm to generate a menu that takes into consideration nutritional balance, the amount of ingredients used, avoidance of overlap with school lunches, health requirements, etc.

[1006] Step 15:

[1007] The server compiles the generated menu information and sends it to the terminal as an HTTP response, which includes the specific menu and its nutritional information.

[1008] Step 16:

[1009] The device analyzes the menu information received from the server and displays it in a user-friendly format. The user can then check the proposed menu through the device.

[1010] Example 1

[1011] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1012] Currently, there are applications and systems on the market that suggest meal plans based on the ingredients in the refrigerator, but most of them do not take into account nutritional balance, individual health information, or information about school lunches at home, and as a result, they are unable to provide efficient and healthy meal plans. This poses the problem that users have to go to the trouble of planning their own meals and assembling menus that take health information into account. Furthermore, it is difficult to create menus that do not overlap with information about children's school lunches.

[1013] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1014] In this invention, the server includes a means for inputting information about ingredients in the refrigerator, a means for saving the ingredient information in a database, a means for generating a menu that takes nutritional balance into consideration based on the saved ingredient information, a means for generating a menu based on a prompt sentence using a generation AI model, and a means for displaying the generated menu information on a terminal. This makes it possible to effectively use ingredients in the refrigerator and automatically provide a well-balanced menu that takes into consideration the user's health information and children's school lunch information.

[1015] "Information about food items in the refrigerator" refers to attribute information such as the type, quantity, and expiration date of the food items stored in the refrigerator.

[1016] "Database" means an organized collection of data for efficient storage, management, and retrieval of information.

[1017] "Nutritional balance" refers to a state in which the food you eat contains nutrients such as protein, lipids, carbohydrates, vitamins, and minerals in appropriate proportions.

[1018] "Menu" refers to the combination of menu items for a particular meal.

[1019] A "generative AI model" refers to an algorithm or system that uses AI (artificial intelligence) technology to make predictions or generate information based on text or data.

[1020] "Prompt sentence" refers to an instruction sentence input to a generative AI model to cause it to generate a specific output.

[1021] "Device" refers to an electronic device used by a user to input information or view output information. Examples include smartphones and personal computers.

[1022] "School lunch menu information" refers to information about the menu and nutritional value of meals provided to children at school or other facilities.

[1023] "Information about chronic conditions and specific nutritional needs" refers to information about nutrients that a user wishes to consume or avoid based on a user's specific health condition or goals.

[1024] "Means for inputting information about ingredients in the refrigerator" refers to an interface or method that allows a user to input attribute information about ingredients in the refrigerator into the system.

[1025] "Storage means" refers to a method or system for storing input information in a database.

[1026] "Generating means" refers to the methods or techniques used to create a particular output (e.g., a menu or recipe) based on the stored information.

[1027] "Means for displaying" refers to the method or interface for conveying the generated information to the user.

[1028] MODE FOR CARRYING OUT THE INVENTION

[1029] This invention is a system that automatically generates healthy menus based on information about ingredients in a refrigerator. This system is composed of three parties: a user, a terminal, and a server. The specific processing steps are explained below.

[1030] Enter and save ingredient information

[1031] First, the user enters information about the ingredients in the refrigerator into the terminal. Specifically, the user enters information such as the type of ingredient (e.g., chicken, tomato, lettuce), quantity, and expiration date. The terminal uses a common interface, such as a smartphone or PC. The terminal checks the entered information and sends it to the server as JSON-formatted data. The server stores this information in a database system such as MySQL or PostgreSQL.

[1032] Enter your child's school lunch information (optional)

[1033] If the user has information about school lunches for their children at home, they can also enter this information into the device. The school lunch menu and its nutritional value are entered, and the device confirms this and sends it to the server. This information is also saved in the database. For example, by saving detailed information such as "Today's school lunch is curry rice (calories: 450kcal, vitamin C: 10mg)," the server can use this information to generate menus later.

[1034] Enter health information (optional)

[1035] Furthermore, if a user has a specific chronic illness or has specific nutrients they want to consume, they can enter that information through the device. For example, they can enter information such as "I want to consume a lot of vitamin C." After confirming this, the device sends it to the server, which stores it in a database. By storing this health information, it is given priority when generating menus.

[1036] Automatic menu generation

[1037] The server generates menus based on the ingredient information, school meal information, and health information stored in the database, taking into consideration factors such as nutritional balance, ingredient amounts, and avoidance of overlaps with school meals. This process uses a common AI model, such as GPT-4, as a generative AI model. To generate a menu that meets specific conditions, a prompt is created and sent to the AI ​​model.

[1038] For example, use the following prompt:

[1039] I have the following ingredients in my fridge:

[1040] 500g chicken

[1041] 3 tomatoes

[1042] 1 head of lettuce

[1043] I would also like to get a lot of vitamin C. Could you please suggest a suitable menu for tonight's dinner?

[1044] (If there is information about children's school lunches)

[1045] Today's school lunch includes curry rice (calories: 450kcal, vitamin C: 10mg). Please make sure that the menu does not overlap with the school lunch menu.

[1046] Based on this prompt, the generative AI model generates a specific menu, such as "chicken stewed in tomato sauce" and "lettuce salad."

[1047] Display menu information

[1048] The menu information generated by the server is sent to the device, which receives the information and displays it in a visually easy-to-understand format for the user. The user can then review the proposed menu and incorporate it into their daily meal plan.

[1049] This system allows users to easily achieve a healthy and balanced diet while making effective use of the ingredients in their refrigerator. It also takes into account individual children's school lunch information and health information, making it possible to automatically provide optimal meal plans for the whole family.

[1050] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1051] Program processing flow

[1052] Step 1: Enter ingredient information

[1053] 1. The user enters the ingredients information into the terminal.

[1054] The user inputs the type, quantity, and expiration date of ingredients in the refrigerator into the terminal.

[1055] Input data: type of ingredients (e.g. chicken, tomato, lettuce), quantity, expiration date

[1056] 2. The device checks the entered information

[1057] The terminal automatically checks the input for errors.

[1058] Data processing: Check if the quantity field is a number and if the expiration date is in the future.

[1059] Output data: Ingredient information after verification (valid data)

[1060] 3. The device sends the information to the server

[1061] After checking the entered ingredient information, it is sent to the server in JSON format.

[1062] Data processing: Converting ingredient information into JSON format

[1063] Output data: Ingredient information in JSON format

[1064] Step 2: Save the ingredients

[1065] 1. The server receives the information

[1066] The server receives the ingredient information sent from the terminal.

[1067] Input data: JSON formatted ingredient information

[1068] 2. The server stores the information in a database

[1069] The server stores the received ingredient information in an appropriate database system (e.g., MySQL, PostgreSQL).

[1070] Data processing: Convert JSON format data into database format

[1071] Output data: Ingredient information stored in a database

[1072] Step 3: Enter school lunch information (optional)

[1073] 1. The user inputs the school lunch menu information into the terminal.

[1074] The user inputs the menu and nutritional value of the school lunch that the child will have at school into the terminal.

[1075] Input data: School lunch menu, nutritional value (e.g. curry rice, calories: 450kcal, vitamin C: 10mg)

[1076] 2. The device checks the information and sends it to the server

[1077] The terminal checks the input information for errors and sends it to the server in JSON format.

[1078] Data processing: Convert input information into JSON format

[1079] Output data: JSON format school lunch information

[1080] 3. The server stores the information in a database

[1081] The server stores the received meal information in a database.

[1082] Input data: JSON format school lunch information

[1083] Output data: Meal information stored in a database

[1084] Step 4: Enter your health information (optional)

[1085] 1. The user enters health information into the device

[1086] Users input information about specific medical conditions and the nutrients they want to consume into the device.

[1087] Input data: Health information (e.g., I want to take in more vitamin C)

[1088] 2. The device checks the information and sends it to the server

[1089] The device reviews the input information for errors and sends it to the server in JSON format.

[1090] Data processing: Convert input information into JSON format

[1091] Output data: Health information in JSON format

[1092] 3. The server stores the information in a database

[1093] The server stores the received health information in a database.

[1094] Input data: Health information in JSON format

[1095] Output data: Health information stored in a database

[1096] Step 5: Automatic menu generation

[1097] 1. The server retrieves the necessary information from the database

[1098] The server retrieves food ingredient information, school lunch information, and health information from the database.

[1099] Input data: Various information stored in the database

[1100] Output data: Consolidated information list

[1101] 2. The server generates a prompt using the generative AI model.

[1102] The server creates a prompt based on the information obtained.

[1103] Data processing: Converting information lists into prompt statements

[1104] Output data: Generated prompt statement

[1105] Examples:

[1106] I have the following ingredients in my fridge:

[1107] 500g chicken

[1108] 3 tomatoes

[1109] 1 head of lettuce

[1110] I would also like to get a lot of vitamin C. Could you please suggest a suitable menu for tonight's dinner?

[1111] (If there is information about children's school lunches)

[1112] Today's school lunch includes curry rice (calories: 450kcal, vitamin C: 10mg). Please make sure that the menu does not overlap with the school lunch menu.

[1113] 3. The server sends a prompt to the generative AI model (e.g., GPT-4).

[1114] The server sends the prompt to the AI ​​model, asking it to generate a menu.

[1115] Input data: Prompt statement

[1116] Output data: Generated menu information

[1117] 4. The generative AI model returns a response, which the server receives.

[1118] Receive menu information generated from an AI model.

[1119] Input data: AI model response

[1120] Output data: Menu information (e.g., chicken stewed in tomato sauce, lettuce salad)

[1121] Step 6: Display menu information

[1122] 1. The server sends the generated menu information to the terminal.

[1123] The server sends the generated menu information to the terminal in JSON format.

[1124] Input data: Generated menu information

[1125] Output data: Menu information in JSON format

[1126] 2. The device analyzes the menu information received from the server.

[1127] The terminal parses the JSON formatted data and converts it into a format that is displayed to the user.

[1128] Data processing: Convert JSON format data into a data format for visualization

[1129] Output data: Visualized menu information

[1130] 3. The device displays the menu information to the user.

[1131] The device displays visualized menu information to the user, who can then review the suggested menu and incorporate it into their daily meal planning.

[1132] Input data: Visualized menu information

[1133] Output data: Menu information displayed to the user

[1134] This detailed process flow allows users to efficiently utilize the ingredients in their refrigerator and achieve a healthy and balanced diet.

[1135] (Application example 1)

[1136] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1137] In today's busy households, it is an important challenge to efficiently utilize the ingredients stored in the refrigerator and easily generate nutritionally balanced menus. To address this challenge, there is a demand for menu generation that takes into account health information and information about meals provided at home. However, current systems are unable to fully meet these demands. In particular, there is no system that allows users to easily input information about ingredients purchased at stores, which requires the user to enter the information manually. Furthermore, since conventional menu generation systems have difficulty taking into account health information and meal information, there is a demand for a system that can integrate such information and generate optimal menus.

[1138] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1139] In this invention, the server includes a means for inputting information about ingredients stored in the refrigerator, a means for saving the ingredient information in a database, a means for generating a nutritionally balanced menu based on the saved ingredient information, a means for scanning purchased ingredients and inputting information, and a means for displaying the generated menu information on a terminal. This allows the user to easily input ingredients purchased at the store and easily generate an optimal nutritionally balanced menu. In addition, by taking into account menu information and health information for school lunches at home and avoiding duplication, the health of the entire family can be supported.

[1140] "Information about ingredients in the refrigerator" refers to detailed information such as the type, quantity, and expiration date of ingredients that the user has in the refrigerator.

[1141] "Means for saving to a database" refers to the process and system for storing the input information in a database on a server so that it can be referenced later.

[1142] The "means for generating a menu that takes nutritional balance into consideration" refers to an algorithm and system that automatically creates a menu with optimal nutritional balance based on stored ingredient information, the user's health condition, and nutritional needs.

[1143] "Means of scanning purchased ingredients and entering information" refers to a function that automatically enters information about ingredients by scanning the barcode or QR code of ingredients purchased at a store using a smartphone or other device.

[1144] The "means for displaying the generated menu information on the terminal" is a system that displays the menu generated by the server on the user's terminal such as a smartphone or tablet, making it available for the user to refer to.

[1145] "Information about school lunch menus at home" refers to information about the menus and nutritional components of school lunches that children in the home eat at school or other places.

[1146] "Information about chronic illnesses and particular nutrients you want to take" refers to information about specific health problems or specific nutrients you want to take in more of.

[1147] A "generative AI model" is an artificial intelligence model used to suggest optimal menus and recipes based on user input data.

[1148] A "prompt sentence" is a sentence-based instruction provided to a generative AI model as input data for generating a menu.

[1149] This invention is a system that automatically generates menus that take into consideration the user's health status and nutritional balance based on information about ingredients in a refrigerator. The system is mainly composed of three entities: a server, a terminal, and a user. Specific embodiments are described below.

[1150] Entering ingredient information

[1151] The user inputs information about ingredients in the refrigerator into a device such as a smartphone. This information includes the type, quantity, and expiration date. The device confirms the information entered by the user and then sends it to the server. The server stores the received information in a database and uses it later when creating menus.

[1152] Enter purchased food information

[1153] When a user purchases new ingredients at a store, they can quickly enter ingredient information by scanning the barcode or QR code using the smartphone camera. This information is also sent to the server and stored in the database.

[1154] Entering health and child meal information

[1155] If a user has health problems or special nutritional needs, they can enter that information into the device. If they have children at home, they can also enter information about their school lunches. This information is also sent to the server and stored in a database.

[1156] Menu generation

[1157] The server uses a generative AI model to automatically generate optimal menus based on stored ingredient information, health information, and school lunch information. The generated menus take into consideration nutritional balance, ingredient usage efficiency, and avoidance of overlap with school lunches. The generated menus are then evaluated for nutritional value using the Nutritics API.

[1158] Specifically, the server uses the following methods:

[1159] A way to input information about ingredients in the refrigerator

[1160] A means for storing the ingredient information in a database

[1161] A method for generating nutritionally balanced menus based on stored ingredient information

[1162] A way to scan purchased ingredients and enter information

[1163] A method for generating menus based on health information using a generative AI model

[1164] This allows users to easily input ingredients purchased at the store and generate optimal menus that take nutritional balance into consideration. It also takes into account information about school lunches and health information at home, helping to support the health of the entire family.

[1165] Display menu information

[1166] The generated menu information is sent from the server to the user's device and displayed on their smartphone, tablet, etc. Users can check it and incorporate it into their daily meal plans.

[1167] Specific examples

[1168] For example, if a user purchases "chicken, tomato, lettuce, and carrot" at a supermarket and scans each item with a smartphone app, and the user enters "high in vitamin C" as health information and "curry (low in vitamins)" as their child's school lunch information, the app will automatically suggest a menu of "chicken and tomato stew, lettuce and carrot salad (high in vitamin C)."

[1169] Example prompts to input to the generative AI model

[1170] Consider the user's health condition and generate a menu rich in Vitamin C.

[1171] Based on information about your child's school lunch, please suggest healthy menus that do not use duplicate ingredients.

[1172] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1173] Step 1:

[1174] The user inputs information about ingredients in the refrigerator into the terminal. Specifically, the type of ingredient, quantity, and expiration date are entered, and the information is sent from the terminal to the server. At this time, the input information is converted into a data format such as JSON.

[1175] input:

[1176] Food type

[1177] Ingredient quantity

[1178] expiration date

[1179] output:

[1180] JSON formatted ingredient information

[1181] Step 2:

[1182] The server receives the ingredient information sent from the terminal and stores it in a database, where the ingredient information is recorded in a table format.

[1183] input:

[1184] JSON formatted ingredient information

[1185] output:

[1186] Ingredient information stored in the database

[1187] Step 3:

[1188] Users scan ingredients purchased at the store using the device's camera and enter the information. By reading barcodes and QR codes, the product name, nutritional information, expiration date, etc. are automatically obtained and sent to the server.

[1189] input:

[1190] Barcode / QR code

[1191] output:

[1192] Purchased ingredients information in JSON format

[1193] Step 4:

[1194] The server receives the information about the purchased ingredients and stores it in an existing database. The purchased ingredient information is also managed as an integrated ingredient information.

[1195] input:

[1196] Purchased ingredients information in JSON format

[1197] output:

[1198] Ingredient information stored in a database (integrated with existing databases)

[1199] Step 5:

[1200] The user inputs specific health information and information about their child's school lunch into the terminal. Health information includes specific nutrient intake preferences and information about chronic illnesses, while child's school lunch information includes the school lunch menu and its nutritional components. This information is also sent to the server.

[1201] input:

[1202] health information

[1203] School lunch information

[1204] output:

[1205] Health and meal information in JSON format

[1206] Step 6:

[1207] The server receives the health information and feeding information and stores it in a database, so that all necessary information is collected in the database.

[1208] input:

[1209] Health and meal information in JSON format

[1210] output:

[1211] Health and feeding information stored in a database

[1212] Step 7:

[1213] The server uses a generative AI model to generate optimal menus based on stored information on ingredients, health, and school lunches. It inputs prompt statements into the generative AI model to calculate nutritionally balanced and efficient menus, and evaluates the nutritional value of the generated menus using the Nutritics API.

[1214] input:

[1215] Food ingredient information, health information, and school lunch information stored in the database

[1216] Prompt statement

[1217] output:

[1218] Optimal menu information

[1219] Specific prompt examples:

[1220] Consider the user's health condition and generate a menu rich in Vitamin C.

[1221] Step 8:

[1222] The generated menu information is sent from the server to the user's device and displayed on their smartphone, tablet, etc. The user can then review it and incorporate it into their daily meal plan.

[1223] input:

[1224] Optimal menu information

[1225] output:

[1226] Menu information displayed on the device

[1227] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1228] MODE FOR CARRYING OUT THE INVENTION

[1229] This invention is a system that generates healthy menus based on information about ingredients in the refrigerator and also takes into account the user's emotional information. This system is composed of a user, a terminal, a server, and an emotion engine.

[1230] Enter ingredient information

[1231] The user inputs information about ingredients in the refrigerator into the terminal. Specifically, the user provides information such as the type of ingredient, quantity, and expiration date. The terminal sends this information to the server, which then stores the received ingredient information in a database. The stored ingredient information is used as basic data for creating menus.

[1232] Enter your child's school lunch information (optional)

[1233] Users also input their child's school lunch information into the device. This information, including details about the menu and nutritional value, is sent to the server and stored in a database. Based on this information, a menu that does not overlap with school lunches is generated.

[1234] Enter health information (optional)

[1235] Users enter information about any medical conditions or specific nutritional needs they have into the device. For example, if they want to increase their intake of vitamin C, they enter that information. This information is also sent to the server and stored in a database. The health information is used to manage illnesses and create diet plans that emphasize specific nutrients.

[1236] emotional information recognition

[1237] The emotion engine recognizes emotions from the user's voice and input information. For example, if the user is feeling stressed, it will detect that emotion. The emotion engine sends this information to the server and stores it in a database. The emotion information is used to create a menu that reflects the user's current psychological state.

[1238] Menu generation

[1239] The server queries and obtains information on ingredients, school meals, health, and emotions from the database. Based on this information, a menu generation algorithm operates, taking into account nutritional balance, the amount of ingredients used, avoidance of overlaps with school meals, health conditions, and emotional state to generate an optimal menu. For example, for a user who is feeling stressed, ingredients with a relaxing effect will be selected.

[1240] As a concrete example, consider a situation where the refrigerator contains "chicken, tomatoes, and lettuce." If the user inputs "I want to get more vitamin C," and the emotion engine also detects "fatigue," the server will use these conditions to suggest "chicken stewed in tomato sauce," "lettuce salad," and "oranges," which are rich in vitamin C.

[1241] Display menu information

[1242] The menus generated by the server are sent to the terminal and displayed to the user. The user can check the suggested menus and incorporate them into their daily meal plans. In particular, the system is expected to have the effect of refreshing the mind through meals by suggesting menus that take emotions into consideration.

[1243] This system allows users to maintain a healthy and balanced diet while making the most of the ingredients in their refrigerator. It also takes into account their children's school lunch information, health information, and even their emotional state, making it easy to create optimal meal plans for the whole family.

[1244] The processing flow will be explained below.

[1245] Step 1:

[1246] The user inputs information about ingredients in the refrigerator into the terminal, such as the type of ingredient (e.g., chicken, tomato, lettuce), quantity, and expiration date.

[1247] Step 2:

[1248] The terminal checks the entered ingredient information and checks for errors. After checking, it prepares to send this ingredient information to the server.

[1249] Step 3:

[1250] The device sends the ingredient information to the server as an HTTP request. The request includes all the ingredient information entered.

[1251] Step 4:

[1252] The server analyzes the received request, extracts the ingredient information, and stores it in the food_inventory table in the database.

[1253] Step 5:

[1254] The user inputs information about the child's school lunch into the terminal, including detailed information about the school lunch menu and nutritional value.

[1255] Step 6:

[1256] The terminal checks the entered school lunch information and checks for errors. After checking, the terminal prepares to send this school lunch information to the server.

[1257] Step 7:

[1258] The terminal sends the school lunch information to the server as an HTTP request. The request includes all the school lunch information that was entered.

[1259] Step 8:

[1260] The server analyzes the received request, extracts the school meal information, and saves it in the school_meals table in the database.

[1261] Step 9:

[1262] The user inputs information about their chronic illnesses and the nutrients they particularly want to take into the terminal. For example, they input information such as "I want to take in a lot of vitamin C."

[1263] Step 10:

[1264] The terminal checks the entered health information for errors, and then prepares to send the health information to the server.

[1265] Step 11:

[1266] The device sends the health information to the server as an HTTP request, which includes all the entered health information.

[1267] Step 12:

[1268] The server analyzes the received request, extracts the health information, and stores it in the health_requirements table in the database.

[1269] Step 13:

[1270] The emotion engine recognizes emotions from user input and voice. If the user is feeling stressed, it will detect that emotional information.

[1271] Step 14:

[1272] The emotion engine sends the recognized emotion information to the server as an HTTP request. The request includes the recognized emotion information.

[1273] Step 15:

[1274] The server analyzes the received request and extracts emotional information. The extracted information is stored in the emotion_data table in the database.

[1275] Step 16:

[1276] The server queries the database to obtain information on ingredients, meals, health, and emotions, and uses the information as input data for the menu generation algorithm.

[1277] Step 17:

[1278] The server runs a menu generation algorithm to generate a menu that takes into account nutritional balance, the amount of ingredients used, avoidance of overlaps with school meals, health conditions, and emotional state. For example, ingredients with a relaxing effect are selected for a stressed user.

[1279] Step 18:

[1280] The server compiles the generated menu information and sends it to the terminal as an HTTP response, which includes the specific menu and its nutritional information.

[1281] Step 19:

[1282] The device analyzes the menu information received from the server and displays it in a user-friendly format. The user can then check the proposed menu through the device.

[1283] This detailed process flow allows users to effectively use the ingredients in their refrigerator while creating meals that are in line with their health, nutritional balance, and even their own emotional state. It also makes it possible to suggest menus that do not overlap with children's school lunch information, providing optimal meal plans for the whole family.

[1284] Example 2

[1285] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1286] Conventional menu generation systems could take into account information about ingredients in the refrigerator and the user's health information, but they were unable to propose menus that took into account the user's emotional information. As a result, they were unable to provide more personalized menus that reflected the user's daily emotional fluctuations. Furthermore, even when generating menus that did not overlap or that emphasized specific nutrients, only limited information was taken into account, making it difficult to create optimal meal plans.

[1287] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1288] In this invention, the server includes a means for storing ingredient information in a database, a means for recognizing the user's emotional information, and a means for generating a menu that takes nutritional balance into consideration, thereby enabling the generation of a more personalized menu based on the ingredient information, health information, and emotional information in the user's refrigerator.

[1289] "Ingredient information" refers to data such as the type, quantity, and expiration date of food stored in the refrigerator.

[1290] A "database" is an information system for storing, managing, and retrieving data in an organized manner, and takes the form of a relational database management system (RDBMS).

[1291] The "means for generating a menu that takes nutritional balance into consideration" refers to an algorithm or program that calculates the balance of nutrients based on the input food ingredient information and other conditions and suggests the optimal meal.

[1292] A "terminal" is a device through which a user inputs information, and includes smartphones, tablets, personal computers, etc.

[1293] "Emotional information" is data that indicates the user's psychological state or mood, and is acquired through voice recognition, text input, or the like.

[1294] The "means for recognizing user's emotional information" is a combination of hardware and software for analyzing the user's voice and input data to identify the user's psychological state.

[1295] "Information on school lunch menus at home" refers to data on the menus and nutritional values ​​of meals provided to children at school or other facilities.

[1296] "Information about chronic illnesses and particularly desirable nutrients" is data about the health condition that the user wants to manage and the nutrients that the user wants to prioritize.

[1297] A "menu" is data that includes a meal menu and the ingredients and cooking procedures for serving it.

[1298] MODE FOR CARRYING OUT THE INVENTION

[1299] This invention is a system that generates healthy menus based on information about ingredients in a refrigerator and also takes into account the user's emotional information. This system is composed of a user, a terminal, a server, and an emotion engine.

[1300] Entering ingredient information

[1301] The user inputs information about ingredients in the refrigerator into a terminal. The terminal can be a smartphone or tablet. Specific examples of such devices include smartphones and tablets. For example, the user inputs information about ingredients such as "200g of chicken, 3 tomatoes, 1 lettuce, expiration date 2022-01-15."

[1302] Sending and saving food information

[1303] The device sends the entered ingredient information to the server via Wi-Fi or mobile data. This is done using the HTTP POST method to send data in JSON format. The server analyzes the received ingredient information and stores it in a database (e.g., MySQL or PostgreSQL).

[1304] Enter and submit health and meal information

[1305] Users also enter their own health information and their children's school lunch information into the device. Health information might include, for example, "I want to get more vitamin C," and school lunch information might include data such as "Monday: curry rice, Tuesday: fried fish." This information is also sent to the server via the device and stored in a database.

[1306] Recognition and transmission of emotional information

[1307] The emotion engine recognizes emotions from the user's voice and input information. For example, Google Cloud Speech-to-Text API is used for voice recognition. When a user says, "I'm tired today," the emotion engine converts this into text and recognizes it as "fatigue." This information is sent to the server and stored in a database. This allows the user's emotional state to be reflected in the menu generation process.

[1308] Menu generation and display

[1309] The server queries and obtains information on ingredients, health, school meals, and emotions from the database. Based on the obtained information, a menu generation algorithm written in Python runs and generates an optimal menu taking into account nutritional balance, the amount of ingredients used, avoidance of overlaps with school meals, health conditions, and the user's emotional state. For example, if the ingredients in the refrigerator are "chicken, tomato, and lettuce," and the user says "I want to get more vitamin C," and their emotion is recognized as "fatigue," the menu suggested would be "chicken stewed in tomato sauce," "lettuce salad," and "orange."

[1310] The server sends the generated menu information to the device and displays it to the user via a dedicated app on the device. The user can then check the proposed menu and use it in their daily meal planning.

[1311] Examples of prompt statements

[1312] Next, as a concrete example, the following prompt sentence could be input to a generative AI model:

[1313] I have chicken, tomatoes, and lettuce in my fridge. I'd also like to get more vitamin C. I'm also feeling tired today. Could you recommend a meal plan for me?

[1314] Based on this prompt, the system generates a menu suggesting "chicken stewed in tomato sauce," "lettuce salad," and "orange."

[1315] This system allows users to maintain a healthy and balanced diet while making the most of the ingredients in their refrigerator. It also takes into account their children's school lunch information, health information, and even their emotional state, making it easy to create optimal meal plans for the whole family.

[1316] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1317] Step 1:

[1318] The user inputs information about the ingredients in the refrigerator into the terminal. For example, the input might be "200g of chicken, 3 tomatoes, 1 lettuce, expiration date 2022-01-15." The terminal receives this information and stores it in a data structure. Specifically, this is done using an application on a smartphone or tablet. When the user has finished entering information, they press the send button.

[1319] Step 2:

[1320] The device sends the entered ingredient information to the server. The input is a data structure, and the output is data converted to JSON format. This uses the HTTP POST method. Specifically, the device sends the data to the appropriate URL endpoint. After sending, the device waits for a successful response from the server.

[1321] Step 3:

[1322] The server analyzes the received ingredient information and saves it in a database. The input is JSON format data, which is converted into an internal data structure. The converted data is then stored in a database (e.g., MySQL or PostgreSQL). Specifically, it executes an SQL INSERT statement.

[1323] Step 4:

[1324] The user inputs information about their chronic illnesses and the nutrients they want to take into the device. For example, "I want to take in a lot of vitamin C." The device receives this information and stores it in a data structure. Specific operations include using a form for health information.

[1325] Step 5:

[1326] The device sends the health information to the server. Again, it is converted to JSON format and uses the HTTP POST method. The input is a data structure and the output is JSON formatted data. The device sends the data to the appropriate URL endpoint. If the transmission is successful, the data is stored on the server.

[1327] Step 6:

[1328] The server stores health information in a database. The input data is in JSON format and is stored in the database after parsing. Specifically, the data is saved using the SQL INSERT statement.

[1329] Step 7:

[1330] The user inputs the child's school lunch information into the terminal. For example, "Monday: curry rice, Tuesday: fried fish." The terminal receives the information and stores it in a data structure. Specific operations involve the use of a dedicated form.

[1331] Step 8:

[1332] The device sends the meal information to the server. As with the health information, it is converted to JSON format and the HTTP POST method is used. The input is a data structure and the output is JSON format data. The device sends the data to the server's URL endpoint and waits for a successful response.

[1333] Step 9:

[1334] The server saves the school lunch information in a database. It parses the JSON format data as input and stores it in the database. The specific operation is to execute an SQL INSERT statement.

[1335] Step 10:

[1336] The emotion engine recognizes emotions from the user's voice and input information. The input is the user's voice data, and natural language processing algorithms are applied. As a specific example, Google Cloud Speech-to-Text API is used for voice recognition. The detected emotion information is converted into JSON format.

[1337] Step 11:

[1338] The emotion engine sends the recognized emotion information to the server. The input is emotion data, and the output is JSON format data. The HTTP POST method is used to send it to the server's URL endpoint. The success of the transmission is confirmed.

[1339] Step 12:

[1340] The server saves the emotion information in a database. It parses the JSON format emotion data as input and stores it in the database. The specific operation is to execute an SQL INSERT statement.

[1341] Step 13:

[1342] The server queries and retrieves information about ingredients, meals, health, and emotions from the database. The input is an SQL query, and the output is the corresponding dataset. Specifically, multiple SELECT statements are used.

[1343] Step 14:

[1344] The server runs a menu generation algorithm based on the acquired information. The input data is ingredient information, school lunch information, health information, and emotional information, and the output is the generated menu information. The algorithm is implemented in Python and makes appropriate meal suggestions.

[1345] Step 15:

[1346] The server sends the generated menu information to the terminal. Here too, it is converted to JSON format and the HTTP POST method is used. The input is the menu data, and the output is JSON format data. It is sent to the terminal's URL endpoint.

[1347] Step 16:

[1348] The device then displays the received menu information to the user. The input is menu data in JSON format, which is processed by the application. Specifically, the menu information is visually presented using a dedicated interface. The user can then review the suggested menu and use it to plan their daily meals.

[1349] (Application example 2)

[1350] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1351] Conventional menu generation systems were unable to propose menus that comprehensively took into account information about ingredients in the user's refrigerator, health information, and even emotional information. Furthermore, ordering additional ingredients when necessary ingredients were in short supply was a hassle. This made it difficult for users to maintain a healthy diet and to refresh themselves psychologically. Furthermore, users tended to rely on manual input, which made managing each piece of information cumbersome.

[1352] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1353] In this invention, the server includes means for storing information about ingredients in the user's refrigerator in a database, means for generating a nutritionally balanced menu taking into account the user's health information and emotional information, means for ordering ingredients that are in short supply, and means for displaying the generated menu information on a terminal. This allows the user to receive optimal menu suggestions that reflect the user's health information and emotional information, and allows them to easily order additional ingredients even if they are in short supply, enabling them to lead a healthy and stress-reducing diet.

[1354] The "ingredient information input means" is a device or system that allows a user to input information such as the type, quantity, and expiration date of ingredients in the refrigerator and store it in a database.

[1355] The "health information input means" is a device or system that allows a user to input information about chronic illnesses and particular nutritional intake, and store the information in a database.

[1356] The "emotion information recognition means" is a device or system for recognizing a user's emotion from the user's voice or input information and storing it in a database.

[1357] The "menu generation means" is a device or system that synthesizes ingredient information, health information, and emotional information to generate an optimal menu that takes into consideration nutritional balance and the user's psychological state.

[1358] The "ingredient ordering means" is a device or system for ordering additional ingredients when there are insufficient ingredients required for the generated menu.

[1359] The "menu display means" is a device or system for displaying menu information generated by the server on the user's terminal.

[1360] The "school lunch information input means" is a device or system for inputting school lunch menu information for the home and storing it in a database.

[1361] The "chronic disease information storage means" is a device or system for storing information about a user's chronic disease in a database.

[1362] The "emotion information storage means" is a device or system for storing the recognized emotion information in a database.

[1363] The system for implementing this invention creates an optimal menu based on information about ingredients, health, and emotions in the user's refrigerator, and orders any ingredients that are missing. Specifically, the system uses the following hardware and software:

[1364] 1. Enter refrigerator food information

[1365] The user inputs information about ingredients in the refrigerator using a device such as a smartphone. This input information includes the type of ingredient, quantity, expiration date, etc. The device sends this information to the server, where it is stored in a database.

[1366] 2. Enter your health information

[1367] Users input information about their chronic illnesses and the nutrients they particularly want to take into account through a device such as a smartphone. This input information includes the nutrients the user wants to take and information about their chronic illness. The device sends this information to a server and stores it on a data bus on the server.

[1368] 3. Recognition of emotional information

[1369] The server uses the voice recognition function or camera of the smartphone or other device to collect emotional information about the user, including, for example, fatigue and stress levels. The device sends this information to the server and stores it on a data bus on the server.

[1370] 4. Menu generation

[1371] The server retrieves ingredient, health, and emotional information from the database and generates an optimal menu based on that information. The menu generation algorithm takes into account nutritional balance, ingredient amounts, health conditions, and emotional state. For example, if the user is feeling tired, it will generate a menu that includes ingredients with a relaxing effect.

[1372] 5. Ordering missing ingredients

[1373] If a required ingredient is in short supply when creating a menu, the server automatically places an additional order through the food delivery system, allowing users to have the ingredients they need delivered to them.

[1374] 6. Display of menu information

[1375] The menus generated by the server are sent to a device such as a smartphone and displayed to the user, allowing the user to easily check and implement their daily meal plan.

[1376] Specific examples

[1377] Specifically, use the following prompt:

[1378] I have chicken, tomatoes, and lettuce in my refrigerator. I'd like to get more vitamin C. I'm currently feeling fatigued. Based on this information, please suggest a meal plan that will be healthy and help reduce stress.

[1379] In this way, this invention realizes a system that integrates information on ingredients in the refrigerator, health information, and emotional information, creates nutritionally balanced menus, and allows users to easily order the necessary ingredients, enabling users to lead healthy and stress-reducing eating habits.

[1380] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1381] Step 1:

[1382] Enter ingredient information

[1383] The user inputs the type, quantity, and expiration date of ingredients in the refrigerator into a device such as a smartphone. The device sends this information to a server, which then stores the received ingredient information in a database. The input information might include, for example, "2 pieces of chicken," "3 tomatoes," "1 lettuce," and "expiration date 2023-10-10."

[1384] (Input: ingredient information, Output: ingredient information stored in the database)

[1385] Step 2:

[1386] Enter health information

[1387] Users input information about the nutrients they particularly want to consume and any chronic illnesses they have into a device such as a smartphone. The device then sends this information to a server, which then stores the received information in a database. Specific examples include information such as "I want to take in a lot of vitamin C" and "I have no chronic illnesses."

[1388] (Input: health information, Output: health information stored in the database)

[1389] Step 3:

[1390] emotional information recognition

[1391] The emotion engine uses the smartphone's voice recognition function and camera to recognize the user's emotions. For example, it can detect "fatigue" from the user's voice and facial expressions. The device sends this information to the server, which then stores it in a database.

[1392] (Input: Emotion information, Output: Emotion information stored in the database)

[1393] Step 4:

[1394] Database queries

[1395] The server queries and retrieves stored ingredient information, health information, and emotional information from the database. This aggregates the information set necessary for menu generation. For example, information such as "chicken, tomato, lettuce," "high intake of vitamin C," and "feeling tired" is retrieved.

[1396] (Input: database query, Output: retrieved information set)

[1397] Step 5:

[1398] Menu generation

[1399] The server then uses a generative AI model to generate an appropriate menu based on the acquired information set. The menu generation algorithm takes into account nutritional balance, the amount of ingredients used, health conditions, and emotional state. For example, it might suggest dishes such as chicken stewed in tomato sauce, which has a relaxing effect, lettuce salad, and oranges, which are rich in vitamin C.

[1400] (Input: information set, output: generated menu)

[1401] Step 6:

[1402] Ordering missing ingredients

[1403] The server checks whether the ingredients required for the generated menu are missing, and if so, automatically places an additional order through the food delivery system, for example, ordering an "orange."

[1404] (Input: Generated menu, check for missing ingredients, Output: Order request)

[1405] Step 7:

[1406] Display menu information

[1407] The server sends the generated menu and ordering information for missing ingredients to a device such as a smartphone, where the user can check the proposed menu and added order details on the device.

[1408] (Input: generated menu, order information, Output: information displayed on the terminal)

[1409] Through these steps, the system can integrate the user's ingredients, health status, and emotional state to generate an optimal menu and automatically order any missing ingredients.

[1410] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1411] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1412] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1413] [Fourth embodiment]

[1414] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1415] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1416] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1417] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1418] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1419] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1420] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1421] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1422] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1423] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1424] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1425] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1426] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1427] MODE FOR CARRYING OUT THE INVENTION

[1428] This invention is a system that automatically generates healthy menus based on information about ingredients in the refrigerator. The system is composed of three parties: the user, the terminal, and the server, and creates menus based on information stored in a database.

[1429] Enter ingredient information

[1430] First, the user inputs information about the ingredients in the refrigerator into the terminal. Specifically, the type of ingredient, quantity, expiration date, etc. are input. After confirming this information, the terminal sends it to the server. The server stores the received information in a database. The stored data is later used when creating menus.

[1431] Enter your child's school lunch information (optional)

[1432] For families with children, users input their children's school lunch information into the device. This information, like the food ingredient information, is sent to the server and stored in the database. The school lunch information includes the school lunch menu and its nutritional value. This information makes it possible to ensure that the breakfast and dinner menus do not overlap with those of school lunches.

[1433] Enter health information (optional)

[1434] If a user has a specific chronic illness or has specific nutrients they want to consume, they can enter that information into the device. This information is also sent to the server and stored in a database. For example, if a user wants to consume a lot of vitamin C, foods that fall into that category will be prioritized.

[1435] Menu generation

[1436] The server integrates information on ingredients, school meals, and health stored in a database to create menus that take into consideration nutritional balance, the amount of ingredients used, avoidance of overlaps with school meals, etc. The menu generation algorithm calculates the optimal meal plan that meets these conditions and proposes specific menus.

[1437] As a concrete example, consider a case where the refrigerator contains "chicken, tomatoes, and lettuce." If the user also inputs information such as "I want to consume a lot of vitamin C," the server can use these conditions to suggest "chicken stewed in tomato sauce" and "lettuce salad" for dinner. If the user has also input information about their child's school lunches, the server will adjust the suggestions to avoid overlapping with that information.

[1438] Display menu information

[1439] The menus generated by the server are sent to the device and displayed to the user, who can then check the suggested menus and incorporate them into their daily meal plans.

[1440] This system allows users to maintain a healthy and balanced diet while making effective use of the ingredients in their refrigerator. It also makes it easy to create optimal meal plans for the whole family by taking into account their children's school lunch information and health information.

[1441] The processing flow will be explained below.

[1442] Step 1:

[1443] The user inputs information about the ingredients in the refrigerator into the terminal, such as the type, quantity, and expiration date of the ingredients, and specific data is provided to the terminal.

[1444] Step 2:

[1445] The terminal checks the ingredient information entered by the user to ensure there are no errors, and then prepares to send this information to the server.

[1446] Step 3:

[1447] The device sends the ingredient information to the server as an HTTP request. The request includes all the ingredient information entered.

[1448] Step 4:

[1449] The server analyzes the received request, extracts the ingredient information, and stores it in the food_inventory table in the database.

[1450] Step 5:

[1451] The user inputs their child's school lunch information into the terminal, and detailed information about the school lunch menu and nutritional value is provided.

[1452] Step 6:

[1453] The terminal checks the lunch information to make sure there are no errors, and then prepares to send this information to the server.

[1454] Step 7:

[1455] The terminal sends the school lunch information to the server as an HTTP request. The request includes the entered school lunch menu information.

[1456] Step 8:

[1457] The server analyzes the received request, extracts the school meal information, and saves it in the school_meals table in the database.

[1458] Step 9:

[1459] The user inputs information about a specific chronic illness or the nutrients they want to consume into the device. For example, they input information such as "I want to consume a lot of vitamin C."

[1460] Step 10:

[1461] The device checks the health information to make sure there are no errors, and then prepares to send this information to the server.

[1462] Step 11:

[1463] The device sends the health information to the server as an HTTP request, which includes the entered health information.

[1464] Step 12:

[1465] The server analyzes the received request, extracts the health information, and stores it in the health_requirements table in the database.

[1466] Step 13:

[1467] The server queries the database to obtain information on ingredients, meal plans, and health information, which is then used as input data for the menu generation algorithm.

[1468] Step 14:

[1469] The server runs a menu generation algorithm to generate a menu that takes into consideration nutritional balance, the amount of ingredients used, avoidance of overlap with school lunches, health requirements, etc.

[1470] Step 15:

[1471] The server compiles the generated menu information and sends it to the terminal as an HTTP response, which includes the specific menu and its nutritional information.

[1472] Step 16:

[1473] The device analyzes the menu information received from the server and displays it in a user-friendly format. The user can then check the proposed menu through the device.

[1474] Example 1

[1475] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1476] Currently, there are applications and systems on the market that suggest meal plans based on the ingredients in the refrigerator, but most of them do not take into account nutritional balance, individual health information, or information about school lunches at home, and as a result, they are unable to provide efficient and healthy meal plans. This poses the problem that users have to go to the trouble of planning their own meals and assembling menus that take health information into account. Furthermore, it is difficult to create menus that do not overlap with information about children's school lunches.

[1477] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1478] In this invention, the server includes a means for inputting information about ingredients in the refrigerator, a means for saving the ingredient information in a database, a means for generating a menu that takes nutritional balance into consideration based on the saved ingredient information, a means for generating a menu based on a prompt sentence using a generation AI model, and a means for displaying the generated menu information on a terminal. This makes it possible to effectively use ingredients in the refrigerator and automatically provide a well-balanced menu that takes into consideration the user's health information and children's school lunch information.

[1479] "Information about food items in the refrigerator" refers to attribute information such as the type, quantity, and expiration date of the food items stored in the refrigerator.

[1480] "Database" means an organized collection of data for efficient storage, management, and retrieval of information.

[1481] "Nutritional balance" refers to a state in which the food you eat contains nutrients such as protein, lipids, carbohydrates, vitamins, and minerals in appropriate proportions.

[1482] "Menu" refers to the combination of menu items for a particular meal.

[1483] A "generative AI model" refers to an algorithm or system that uses AI (artificial intelligence) technology to make predictions or generate information based on text or data.

[1484] "Prompt sentence" refers to an instruction sentence input to a generative AI model to cause it to generate a specific output.

[1485] "Device" refers to an electronic device used by a user to input information or view output information. Examples include smartphones and personal computers.

[1486] "School lunch menu information" refers to information about the menu and nutritional value of meals provided to children at school or other facilities.

[1487] "Information about chronic conditions and specific nutritional needs" refers to information about nutrients that a user wishes to consume or avoid based on a user's specific health condition or goals.

[1488] "Means for inputting information about ingredients in the refrigerator" refers to an interface or method that allows a user to input attribute information about ingredients in the refrigerator into the system.

[1489] "Storage means" refers to a method or system for storing input information in a database.

[1490] "Generating means" refers to the methods or techniques used to create a particular output (e.g., a menu or recipe) based on the stored information.

[1491] "Means for displaying" refers to the method or interface for conveying the generated information to the user.

[1492] MODE FOR CARRYING OUT THE INVENTION

[1493] This invention is a system that automatically generates healthy menus based on information about ingredients in a refrigerator. This system is composed of three parties: a user, a terminal, and a server. The specific processing steps are explained below.

[1494] Enter and save ingredient information

[1495] First, the user enters information about the ingredients in the refrigerator into the terminal. Specifically, the user enters information such as the type of ingredient (e.g., chicken, tomato, lettuce), quantity, and expiration date. The terminal uses a common interface, such as a smartphone or PC. The terminal checks the entered information and sends it to the server as JSON-formatted data. The server stores this information in a database system such as MySQL or PostgreSQL.

[1496] Enter your child's school lunch information (optional)

[1497] If the user has information about school lunches for their children at home, they can also enter this information into the device. The school lunch menu and its nutritional value are entered, and the device confirms this and sends it to the server. This information is also saved in the database. For example, by saving detailed information such as "Today's school lunch is curry rice (calories: 450kcal, vitamin C: 10mg)," the server can use this information to generate menus later.

[1498] Enter health information (optional)

[1499] Furthermore, if a user has a specific chronic illness or has specific nutrients they want to consume, they can enter that information through the device. For example, they can enter information such as "I want to consume a lot of vitamin C." After confirming this, the device sends it to the server, which stores it in a database. By storing this health information, it is given priority when generating menus.

[1500] Automatic menu generation

[1501] The server generates menus based on the ingredient information, school meal information, and health information stored in the database, taking into consideration factors such as nutritional balance, ingredient amounts, and avoidance of overlaps with school meals. This process uses a common AI model, such as GPT-4, as a generative AI model. To generate a menu that meets specific conditions, a prompt is created and sent to the AI ​​model.

[1502] For example, use the following prompt:

[1503] I have the following ingredients in my fridge:

[1504] 500g chicken

[1505] 3 tomatoes

[1506] 1 head of lettuce

[1507] I would also like to get a lot of vitamin C. Could you please suggest a suitable menu for tonight's dinner?

[1508] (If there is information about children's school lunches)

[1509] Today's school lunch includes curry rice (calories: 450kcal, vitamin C: 10mg). Please make sure that the menu does not overlap with the school lunch menu.

[1510] Based on this prompt, the generative AI model generates a specific menu, such as "chicken stewed in tomato sauce" and "lettuce salad."

[1511] Display menu information

[1512] The menu information generated by the server is sent to the device, which receives the information and displays it in a visually easy-to-understand format for the user. The user can then review the proposed menu and incorporate it into their daily meal plan.

[1513] This system allows users to easily achieve a healthy and balanced diet while making effective use of the ingredients in their refrigerator. It also takes into account individual children's school lunch information and health information, making it possible to automatically provide optimal meal plans for the whole family.

[1514] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1515] Program processing flow

[1516] Step 1: Enter ingredient information

[1517] 1. The user enters the ingredients information into the terminal.

[1518] The user inputs the type, quantity, and expiration date of ingredients in the refrigerator into the terminal.

[1519] Input data: type of ingredients (e.g. chicken, tomato, lettuce), quantity, expiration date

[1520] 2. The device checks the entered information

[1521] The terminal automatically checks the input for errors.

[1522] Data processing: Check if the quantity field is a number and if the expiration date is in the future.

[1523] Output data: Ingredient information after verification (valid data)

[1524] 3. The device sends the information to the server

[1525] After checking the entered ingredient information, it is sent to the server in JSON format.

[1526] Data processing: Converting ingredient information into JSON format

[1527] Output data: Ingredient information in JSON format

[1528] Step 2: Save the ingredients

[1529] 1. The server receives the information

[1530] The server receives the ingredient information sent from the terminal.

[1531] Input data: JSON formatted ingredient information

[1532] 2. The server stores the information in a database

[1533] The server stores the received ingredient information in an appropriate database system (e.g., MySQL, PostgreSQL).

[1534] Data processing: Convert JSON format data into database format

[1535] Output data: Ingredient information stored in a database

[1536] Step 3: Enter school lunch information (optional)

[1537] 1. The user inputs the school lunch menu information into the terminal.

[1538] The user inputs the menu and nutritional value of the school lunch that the child will have at school into the terminal.

[1539] Input data: School lunch menu, nutritional value (e.g. curry rice, calories: 450kcal, vitamin C: 10mg)

[1540] 2. The device checks the information and sends it to the server

[1541] The terminal checks the input information for errors and sends it to the server in JSON format.

[1542] Data processing: Convert input information into JSON format

[1543] Output data: JSON format school lunch information

[1544] 3. The server stores the information in a database

[1545] The server stores the received meal information in a database.

[1546] Input data: JSON format school lunch information

[1547] Output data: Meal information stored in a database

[1548] Step 4: Enter your health information (optional)

[1549] 1. The user enters health information into the device

[1550] Users input information about specific medical conditions and the nutrients they want to consume into the device.

[1551] Input data: Health information (e.g., I want to take in more vitamin C)

[1552] 2. The device checks the information and sends it to the server

[1553] The device reviews the input information for errors and sends it to the server in JSON format.

[1554] Data processing: Convert input information into JSON format

[1555] Output data: Health information in JSON format

[1556] 3. The server stores the information in a database

[1557] The server stores the received health information in a database.

[1558] Input data: Health information in JSON format

[1559] Output data: Health information stored in a database

[1560] Step 5: Automatic menu generation

[1561] 1. The server retrieves the necessary information from the database

[1562] The server retrieves food ingredient information, school lunch information, and health information from the database.

[1563] Input data: Various information stored in the database

[1564] Output data: Consolidated information list

[1565] 2. The server generates a prompt using the generative AI model.

[1566] The server creates a prompt based on the information obtained.

[1567] Data processing: Converting information lists into prompt statements

[1568] Output data: Generated prompt statement

[1569] Examples:

[1570] I have the following ingredients in my fridge:

[1571] 500g chicken

[1572] 3 tomatoes

[1573] 1 head of lettuce

[1574] I would also like to get a lot of vitamin C. Could you please suggest a suitable menu for tonight's dinner?

[1575] (If there is information about children's school lunches)

[1576] Today's school lunch includes curry rice (calories: 450kcal, vitamin C: 10mg). Please make sure that the menu does not overlap with the school lunch menu.

[1577] 3. The server sends a prompt to the generative AI model (e.g., GPT-4).

[1578] The server sends the prompt to the AI ​​model, asking it to generate a menu.

[1579] Input data: Prompt statement

[1580] Output data: Generated menu information

[1581] 4. The generative AI model returns a response, which the server receives.

[1582] Receive menu information generated from an AI model.

[1583] Input data: AI model response

[1584] Output data: Menu information (e.g., chicken stewed in tomato sauce, lettuce salad)

[1585] Step 6: Display menu information

[1586] 1. The server sends the generated menu information to the terminal.

[1587] The server sends the generated menu information to the terminal in JSON format.

[1588] Input data: Generated menu information

[1589] Output data: Menu information in JSON format

[1590] 2. The device analyzes the menu information received from the server.

[1591] The terminal parses the JSON formatted data and converts it into a format that is displayed to the user.

[1592] Data processing: Convert JSON format data into a data format for visualization

[1593] Output data: Visualized menu information

[1594] 3. The device displays the menu information to the user.

[1595] The device displays visualized menu information to the user, who can then review the suggested menu and incorporate it into their daily meal planning.

[1596] Input data: Visualized menu information

[1597] Output data: Menu information displayed to the user

[1598] This detailed process flow allows users to efficiently utilize the ingredients in their refrigerator and achieve a healthy and balanced diet.

[1599] (Application example 1)

[1600] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1601] In today's busy households, it is an important challenge to efficiently utilize the ingredients stored in the refrigerator and easily generate nutritionally balanced menus. To address this challenge, there is a demand for menu generation that takes into account health information and information about meals provided at home. However, current systems are unable to fully meet these demands. In particular, there is no system that allows users to easily input information about ingredients purchased at stores, which requires the user to enter the information manually. Furthermore, since conventional menu generation systems have difficulty taking into account health information and meal information, there is a demand for a system that can integrate such information and generate optimal menus.

[1602] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1603] In this invention, the server includes a means for inputting information about ingredients stored in the refrigerator, a means for saving the ingredient information in a database, a means for generating a nutritionally balanced menu based on the saved ingredient information, a means for scanning purchased ingredients and inputting information, and a means for displaying the generated menu information on a terminal. This allows the user to easily input ingredients purchased at the store and easily generate an optimal nutritionally balanced menu. In addition, by taking into account menu information and health information for school lunches at home and avoiding duplication, the health of the entire family can be supported.

[1604] "Information about ingredients in the refrigerator" refers to detailed information such as the type, quantity, and expiration date of ingredients that the user has in the refrigerator.

[1605] "Means for saving to a database" refers to the process and system for storing the input information in a database on a server so that it can be referenced later.

[1606] The "means for generating a menu that takes nutritional balance into consideration" refers to an algorithm and system that automatically creates a menu with optimal nutritional balance based on stored ingredient information, the user's health condition, and nutritional needs.

[1607] "Means of scanning purchased ingredients and entering information" refers to a function that automatically enters information about ingredients by scanning the barcode or QR code of ingredients purchased at a store using a smartphone or other device.

[1608] The "means for displaying the generated menu information on the terminal" is a system that displays the menu generated by the server on the user's terminal such as a smartphone or tablet, making it available for the user to refer to.

[1609] "Information about school lunch menus at home" refers to information about the menus and nutritional components of school lunches that children in the home eat at school or other places.

[1610] "Information about chronic illnesses and particular nutrients you want to take" refers to information about specific health problems or specific nutrients you want to take in more of.

[1611] A "generative AI model" is an artificial intelligence model used to suggest optimal menus and recipes based on user input data.

[1612] A "prompt sentence" is a sentence-based instruction provided to a generative AI model as input data for generating a menu.

[1613] This invention is a system that automatically generates menus that take into consideration the user's health status and nutritional balance based on information about ingredients in a refrigerator. The system is mainly composed of three entities: a server, a terminal, and a user. Specific embodiments are described below.

[1614] Entering ingredient information

[1615] The user inputs information about ingredients in the refrigerator into a device such as a smartphone. This information includes the type, quantity, and expiration date. The device confirms the information entered by the user and then sends it to the server. The server stores the received information in a database and uses it later when creating menus.

[1616] Enter purchased food information

[1617] When a user purchases new ingredients at a store, they can quickly enter ingredient information by scanning the barcode or QR code using the smartphone camera. This information is also sent to the server and stored in the database.

[1618] Entering health and child meal information

[1619] If a user has health problems or special nutritional needs, they can enter that information into the device. If they have children at home, they can also enter information about their school lunches. This information is also sent to the server and stored in a database.

[1620] Menu generation

[1621] The server uses a generative AI model to automatically generate optimal menus based on stored ingredient information, health information, and school lunch information. The generated menus take into consideration nutritional balance, ingredient usage efficiency, and avoidance of overlap with school lunches. The generated menus are then evaluated for nutritional value using the Nutritics API.

[1622] Specifically, the server uses the following methods:

[1623] A way to input information about ingredients in the refrigerator

[1624] A means for storing the ingredient information in a database

[1625] A method for generating nutritionally balanced menus based on stored ingredient information

[1626] A way to scan purchased ingredients and enter information

[1627] A method for generating menus based on health information using a generative AI model

[1628] This allows users to easily input ingredients purchased at the store and generate optimal menus that take nutritional balance into consideration. It also takes into account information about school lunches and health information at home, helping to support the health of the entire family.

[1629] Display menu information

[1630] The generated menu information is sent from the server to the user's device and displayed on their smartphone, tablet, etc. Users can check it and incorporate it into their daily meal plans.

[1631] Specific examples

[1632] For example, if a user purchases "chicken, tomato, lettuce, and carrot" at a supermarket and scans each item with a smartphone app, and the user enters "high in vitamin C" as health information and "curry (low in vitamins)" as their child's school lunch information, the app will automatically suggest a menu of "chicken and tomato stew, lettuce and carrot salad (high in vitamin C)."

[1633] Example prompts to input to the generative AI model

[1634] Consider the user's health condition and generate a menu rich in Vitamin C.

[1635] Based on information about your child's school lunch, please suggest healthy menus that do not use duplicate ingredients.

[1636] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1637] Step 1:

[1638] The user inputs information about ingredients in the refrigerator into the terminal. Specifically, the type of ingredient, quantity, and expiration date are entered, and the information is sent from the terminal to the server. At this time, the input information is converted into a data format such as JSON.

[1639] input:

[1640] Food type

[1641] Ingredient quantity

[1642] expiration date

[1643] output:

[1644] JSON formatted ingredient information

[1645] Step 2:

[1646] The server receives the ingredient information sent from the terminal and stores it in a database, where the ingredient information is recorded in a table format.

[1647] input:

[1648] JSON formatted ingredient information

[1649] output:

[1650] Ingredient information stored in the database

[1651] Step 3:

[1652] Users scan ingredients purchased at the store using the device's camera and enter the information. By reading barcodes and QR codes, the product name, nutritional information, expiration date, etc. are automatically obtained and sent to the server.

[1653] input:

[1654] Barcode / QR code

[1655] output:

[1656] Purchased ingredients information in JSON format

[1657] Step 4:

[1658] The server receives the information about the purchased ingredients and stores it in an existing database. The purchased ingredient information is also managed as an integrated ingredient information.

[1659] input:

[1660] Purchased ingredients information in JSON format

[1661] output:

[1662] Ingredient information stored in a database (integrated with existing databases)

[1663] Step 5:

[1664] The user inputs specific health information and information about their child's school lunch into the terminal. Health information includes specific nutrient intake preferences and information about chronic illnesses, while child's school lunch information includes the school lunch menu and its nutritional components. This information is also sent to the server.

[1665] input:

[1666] health information

[1667] School lunch information

[1668] output:

[1669] Health and meal information in JSON format

[1670] Step 6:

[1671] The server receives the health information and feeding information and stores it in a database, so that all necessary information is collected in the database.

[1672] input:

[1673] Health and meal information in JSON format

[1674] output:

[1675] Health and feeding information stored in a database

[1676] Step 7:

[1677] The server uses a generative AI model to generate optimal menus based on stored information on ingredients, health, and school lunches. It inputs prompt statements into the generative AI model to calculate nutritionally balanced and efficient menus, and evaluates the nutritional value of the generated menus using the Nutritics API.

[1678] input:

[1679] Food ingredient information, health information, and school lunch information stored in the database

[1680] Prompt statement

[1681] output:

[1682] Optimal menu information

[1683] Specific prompt examples:

[1684] Consider the user's health condition and generate a menu rich in Vitamin C.

[1685] Step 8:

[1686] The generated menu information is sent from the server to the user's device and displayed on their smartphone, tablet, etc. The user can then review it and incorporate it into their daily meal plan.

[1687] input:

[1688] Optimal menu information

[1689] output:

[1690] Menu information displayed on the device

[1691] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1692] MODE FOR CARRYING OUT THE INVENTION

[1693] This invention is a system that generates healthy menus based on information about ingredients in the refrigerator and also takes into account the user's emotional information. This system is composed of a user, a terminal, a server, and an emotion engine.

[1694] Enter ingredient information

[1695] The user inputs information about ingredients in the refrigerator into the terminal. Specifically, the user provides information such as the type of ingredient, quantity, and expiration date. The terminal sends this information to the server, which then stores the received ingredient information in a database. The stored ingredient information is used as basic data for creating menus.

[1696] Enter your child's school lunch information (optional)

[1697] Users also input their child's school lunch information into the device. This information, including details about the menu and nutritional value, is sent to the server and stored in a database. Based on this information, a menu that does not overlap with school lunches is generated.

[1698] Enter health information (optional)

[1699] Users enter information about any medical conditions or specific nutritional needs they have into the device. For example, if they want to increase their intake of vitamin C, they enter that information. This information is also sent to the server and stored in a database. The health information is used to manage illnesses and create diet plans that emphasize specific nutrients.

[1700] emotional information recognition

[1701] The emotion engine recognizes emotions from the user's voice and input information. For example, if the user is feeling stressed, it will detect that emotion. The emotion engine sends this information to the server and stores it in a database. The emotion information is used to create a menu that reflects the user's current psychological state.

[1702] Menu generation

[1703] The server queries and obtains information on ingredients, school meals, health, and emotions from the database. Based on this information, a menu generation algorithm operates, taking into account nutritional balance, the amount of ingredients used, avoidance of overlaps with school meals, health conditions, and emotional state to generate an optimal menu. For example, for a user who is feeling stressed, ingredients with a relaxing effect will be selected.

[1704] As a concrete example, consider a situation where the refrigerator contains "chicken, tomatoes, and lettuce." If the user inputs "I want to get more vitamin C," and the emotion engine also detects "fatigue," the server will use these conditions to suggest "chicken stewed in tomato sauce," "lettuce salad," and "oranges," which are rich in vitamin C.

[1705] Display menu information

[1706] The menus generated by the server are sent to the terminal and displayed to the user. The user can check the suggested menus and incorporate them into their daily meal plans. In particular, the system is expected to have the effect of refreshing the mind through meals by suggesting menus that take emotions into consideration.

[1707] This system allows users to maintain a healthy and balanced diet while making the most of the ingredients in their refrigerator. It also takes into account their children's school lunch information, health information, and even their emotional state, making it easy to create optimal meal plans for the whole family.

[1708] The processing flow will be explained below.

[1709] Step 1:

[1710] The user inputs information about ingredients in the refrigerator into the terminal, such as the type of ingredient (e.g., chicken, tomato, lettuce), quantity, and expiration date.

[1711] Step 2:

[1712] The terminal checks the entered ingredient information and checks for errors. After checking, it prepares to send this ingredient information to the server.

[1713] Step 3:

[1714] The device sends the ingredient information to the server as an HTTP request. The request includes all the ingredient information entered.

[1715] Step 4:

[1716] The server analyzes the received request, extracts the ingredient information, and stores it in the food_inventory table in the database.

[1717] Step 5:

[1718] The user inputs information about the child's school lunch into the terminal, including detailed information about the school lunch menu and nutritional value.

[1719] Step 6:

[1720] The terminal checks the entered school lunch information and checks for errors. After checking, the terminal prepares to send this school lunch information to the server.

[1721] Step 7:

[1722] The terminal sends the school lunch information to the server as an HTTP request. The request includes all the school lunch information that was entered.

[1723] Step 8:

[1724] The server analyzes the received request, extracts the school meal information, and saves it in the school_meals table in the database.

[1725] Step 9:

[1726] The user inputs information about their chronic illnesses and the nutrients they particularly want to take into the terminal. For example, they input information such as "I want to take in a lot of vitamin C."

[1727] Step 10:

[1728] The terminal checks the entered health information for errors, and then prepares to send the health information to the server.

[1729] Step 11:

[1730] The device sends the health information to the server as an HTTP request, which includes all the entered health information.

[1731] Step 12:

[1732] The server analyzes the received request, extracts the health information, and stores it in the health_requirements table in the database.

[1733] Step 13:

[1734] The emotion engine recognizes emotions from user input and voice. If the user is feeling stressed, it will detect that emotional information.

[1735] Step 14:

[1736] The emotion engine sends the recognized emotion information to the server as an HTTP request. The request includes the recognized emotion information.

[1737] Step 15:

[1738] The server analyzes the received request and extracts emotional information. The extracted information is stored in the emotion_data table in the database.

[1739] Step 16:

[1740] The server queries the database to obtain information on ingredients, meals, health, and emotions, and uses the information as input data for the menu generation algorithm.

[1741] Step 17:

[1742] The server runs a menu generation algorithm to generate a menu that takes into account nutritional balance, the amount of ingredients used, avoidance of overlaps with school meals, health conditions, and emotional state. For example, ingredients with a relaxing effect are selected for a stressed user.

[1743] Step 18:

[1744] The server compiles the generated menu information and sends it to the terminal as an HTTP response, which includes the specific menu and its nutritional information.

[1745] Step 19:

[1746] The device analyzes the menu information received from the server and displays it in a user-friendly format. The user can then check the proposed menu through the device.

[1747] This detailed process flow allows users to effectively use the ingredients in their refrigerator while creating meals that are in line with their health, nutritional balance, and even their own emotional state. It also makes it possible to suggest menus that do not overlap with children's school lunch information, providing optimal meal plans for the whole family.

[1748] Example 2

[1749] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1750] Conventional menu generation systems could take into account information about ingredients in the refrigerator and the user's health information, but they were unable to propose menus that took into account the user's emotional information. As a result, they were unable to provide more personalized menus that reflected the user's daily emotional fluctuations. Furthermore, even when generating menus that did not overlap or that emphasized specific nutrients, only limited information was taken into account, making it difficult to create optimal meal plans.

[1751] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1752] In this invention, the server includes a means for storing ingredient information in a database, a means for recognizing the user's emotional information, and a means for generating a menu that takes nutritional balance into consideration, thereby enabling the generation of a more personalized menu based on the ingredient information, health information, and emotional information in the user's refrigerator.

[1753] "Ingredient information" refers to data such as the type, quantity, and expiration date of food stored in the refrigerator.

[1754] A "database" is an information system for storing, managing, and retrieving data in an organized manner, and takes the form of a relational database management system (RDBMS).

[1755] The "means for generating a menu that takes nutritional balance into consideration" refers to an algorithm or program that calculates the balance of nutrients based on the input food ingredient information and other conditions and suggests the optimal meal.

[1756] A "terminal" is a device through which a user inputs information, and includes smartphones, tablets, personal computers, etc.

[1757] "Emotional information" is data that indicates the user's psychological state or mood, and is acquired through voice recognition, text input, or the like.

[1758] The "means for recognizing user's emotional information" is a combination of hardware and software for analyzing the user's voice and input data to identify the user's psychological state.

[1759] "Information on school lunch menus at home" refers to data on the menus and nutritional values ​​of meals provided to children at school or other facilities.

[1760] "Information about chronic illnesses and particularly desirable nutrients" is data about the health condition that the user wants to manage and the nutrients that the user wants to prioritize.

[1761] A "menu" is data that includes a meal menu and the ingredients and cooking procedures for serving it.

[1762] MODE FOR CARRYING OUT THE INVENTION

[1763] This invention is a system that generates healthy menus based on information about ingredients in a refrigerator and also takes into account the user's emotional information. This system is composed of a user, a terminal, a server, and an emotion engine.

[1764] Entering ingredient information

[1765] The user inputs information about ingredients in the refrigerator into a terminal. The terminal can be a smartphone or tablet. Specific examples of such devices include smartphones and tablets. For example, the user inputs information about ingredients such as "200g of chicken, 3 tomatoes, 1 lettuce, expiration date 2022-01-15."

[1766] Sending and saving food information

[1767] The device sends the entered ingredient information to the server via Wi-Fi or mobile data. This is done using the HTTP POST method to send data in JSON format. The server analyzes the received ingredient information and stores it in a database (e.g., MySQL or PostgreSQL).

[1768] Enter and submit health and meal information

[1769] Users also enter their own health information and their children's school lunch information into the device. Health information might include, for example, "I want to get more vitamin C," and school lunch information might include data such as "Monday: curry rice, Tuesday: fried fish." This information is also sent to the server via the device and stored in a database.

[1770] Recognition and transmission of emotional information

[1771] The emotion engine recognizes emotions from the user's voice and input information. For example, Google Cloud Speech-to-Text API is used for voice recognition. When a user says, "I'm tired today," the emotion engine converts this into text and recognizes it as "fatigue." This information is sent to the server and stored in a database. This allows the user's emotional state to be reflected in the menu generation process.

[1772] Menu generation and display

[1773] The server queries and obtains information on ingredients, health, school meals, and emotions from the database. Based on the obtained information, a menu generation algorithm written in Python runs and generates an optimal menu taking into account nutritional balance, the amount of ingredients used, avoidance of overlaps with school meals, health conditions, and the user's emotional state. For example, if the ingredients in the refrigerator are "chicken, tomato, and lettuce," and the user says "I want to get more vitamin C," and their emotion is recognized as "fatigue," the menu suggested would be "chicken stewed in tomato sauce," "lettuce salad," and "orange."

[1774] The server sends the generated menu information to the device and displays it to the user via a dedicated app on the device. The user can then check the proposed menu and use it in their daily meal planning.

[1775] Examples of prompt statements

[1776] Next, as a concrete example, the following prompt sentence could be input to a generative AI model:

[1777] I have chicken, tomatoes, and lettuce in my fridge. I'd also like to get more vitamin C. I'm also feeling tired today. Could you recommend a meal plan for me?

[1778] Based on this prompt, the system generates a menu suggesting "chicken stewed in tomato sauce," "lettuce salad," and "orange."

[1779] This system allows users to maintain a healthy and balanced diet while making the most of the ingredients in their refrigerator. It also takes into account their children's school lunch information, health information, and even their emotional state, making it easy to create optimal meal plans for the whole family.

[1780] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1781] Step 1:

[1782] The user inputs information about the ingredients in the refrigerator into the terminal. For example, the input might be "200g of chicken, 3 tomatoes, 1 lettuce, expiration date 2022-01-15." The terminal receives this information and stores it in a data structure. Specifically, this is done using an application on a smartphone or tablet. When the user has finished entering information, they press the send button.

[1783] Step 2:

[1784] The device sends the entered ingredient information to the server. The input is a data structure, and the output is data converted to JSON format. This uses the HTTP POST method. Specifically, the device sends the data to the appropriate URL endpoint. After sending, the device waits for a successful response from the server.

[1785] Step 3:

[1786] The server analyzes the received ingredient information and saves it in a database. The input is JSON format data, which is converted into an internal data structure. The converted data is then stored in a database (e.g., MySQL or PostgreSQL). Specifically, it executes an SQL INSERT statement.

[1787] Step 4:

[1788] The user inputs information about their chronic illnesses and the nutrients they want to take into the device. For example, "I want to take in a lot of vitamin C." The device receives this information and stores it in a data structure. Specific operations include using a form for health information.

[1789] Step 5:

[1790] The device sends the health information to the server. Again, it is converted to JSON format and uses the HTTP POST method. The input is a data structure and the output is JSON formatted data. The device sends the data to the appropriate URL endpoint. If the transmission is successful, the data is stored on the server.

[1791] Step 6:

[1792] The server stores health information in a database. The input data is in JSON format and is stored in the database after parsing. Specifically, the data is saved using the SQL INSERT statement.

[1793] Step 7:

[1794] The user inputs the child's school lunch information into the terminal. For example, "Monday: curry rice, Tuesday: fried fish." The terminal receives the information and stores it in a data structure. Specific operations involve the use of a dedicated form.

[1795] Step 8:

[1796] The device sends the meal information to the server. As with the health information, it is converted to JSON format and the HTTP POST method is used. The input is a data structure and the output is JSON format data. The device sends the data to the server's URL endpoint and waits for a successful response.

[1797] Step 9:

[1798] The server saves the school lunch information in a database. It parses the JSON format data as input and stores it in the database. The specific operation is to execute an SQL INSERT statement.

[1799] Step 10:

[1800] The emotion engine recognizes emotions from the user's voice and input information. The input is the user's voice data, and natural language processing algorithms are applied. As a specific example, Google Cloud Speech-to-Text API is used for voice recognition. The detected emotion information is converted into JSON format.

[1801] Step 11:

[1802] The emotion engine sends the recognized emotion information to the server. The input is emotion data, and the output is JSON format data. The HTTP POST method is used to send it to the server's URL endpoint. The success of the transmission is confirmed.

[1803] Step 12:

[1804] The server saves the emotion information in a database. It parses the JSON format emotion data as input and stores it in the database. The specific operation is to execute an SQL INSERT statement.

[1805] Step 13:

[1806] The server queries and retrieves information about ingredients, meals, health, and emotions from the database. The input is an SQL query, and the output is the corresponding dataset. Specifically, multiple SELECT statements are used.

[1807] Step 14:

[1808] The server runs a menu generation algorithm based on the acquired information. The input data is ingredient information, school lunch information, health information, and emotional information, and the output is the generated menu information. The algorithm is implemented in Python and makes appropriate meal suggestions.

[1809] Step 15:

[1810] The server sends the generated menu information to the terminal. Here too, it is converted to JSON format and the HTTP POST method is used. The input is the menu data, and the output is JSON format data. It is sent to the terminal's URL endpoint.

[1811] Step 16:

[1812] The device then displays the received menu information to the user. The input is menu data in JSON format, which is processed by the application. Specifically, the menu information is visually presented using a dedicated interface. The user can then review the suggested menu and use it to plan their daily meals.

[1813] (Application example 2)

[1814] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1815] Conventional menu generation systems were unable to propose menus that comprehensively took into account information about ingredients in the user's refrigerator, health information, and even emotional information. Furthermore, ordering additional ingredients when necessary ingredients were in short supply was a hassle. This made it difficult for users to maintain a healthy diet and to refresh themselves psychologically. Furthermore, users tended to rely on manual input, which made managing each piece of information cumbersome.

[1816] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1817] In this invention, the server includes means for storing information about ingredients in the user's refrigerator in a database, means for generating a nutritionally balanced menu taking into account the user's health information and emotional information, means for ordering ingredients that are in short supply, and means for displaying the generated menu information on a terminal. This allows the user to receive optimal menu suggestions that reflect the user's health information and emotional information, and allows them to easily order additional ingredients even if they are in short supply, enabling them to lead a healthy and stress-reducing diet.

[1818] The "ingredient information input means" is a device or system that allows a user to input information such as the type, quantity, and expiration date of ingredients in the refrigerator and store it in a database.

[1819] The "health information input means" is a device or system that allows a user to input information about chronic illnesses and particular nutritional intake, and store the information in a database.

[1820] The "emotion information recognition means" is a device or system for recognizing a user's emotion from the user's voice or input information and storing it in a database.

[1821] The "menu generation means" is a device or system that synthesizes ingredient information, health information, and emotional information to generate an optimal menu that takes into consideration nutritional balance and the user's psychological state.

[1822] The "ingredient ordering means" is a device or system for ordering additional ingredients when there are insufficient ingredients required for the generated menu.

[1823] The "menu display means" is a device or system for displaying menu information generated by the server on the user's terminal.

[1824] The "school lunch information input means" is a device or system for inputting school lunch menu information for the home and storing it in a database.

[1825] The "chronic disease information storage means" is a device or system for storing information about a user's chronic disease in a database.

[1826] The "emotion information storage means" is a device or system for storing the recognized emotion information in a database.

[1827] The system for implementing this invention creates an optimal menu based on information about ingredients, health, and emotions in the user's refrigerator, and orders any ingredients that are missing. Specifically, the system uses the following hardware and software:

[1828] 1. Enter refrigerator food information

[1829] The user inputs information about ingredients in the refrigerator using a device such as a smartphone. This input information includes the type of ingredient, quantity, expiration date, etc. The device sends this information to the server, where it is stored in a database.

[1830] 2. Enter your health information

[1831] Users input information about their chronic illnesses and the nutrients they particularly want to take into account through a device such as a smartphone. This input information includes the nutrients the user wants to take and information about their chronic illness. The device sends this information to a server and stores it on a data bus on the server.

[1832] 3. Recognition of emotional information

[1833] The server uses the voice recognition function or camera of the smartphone or other device to collect emotional information about the user, including, for example, fatigue and stress levels. The device sends this information to the server and stores it on a data bus on the server.

[1834] 4. Menu generation

[1835] The server retrieves ingredient, health, and emotional information from the database and generates an optimal menu based on that information. The menu generation algorithm takes into account nutritional balance, ingredient amounts, health conditions, and emotional state. For example, if the user is feeling tired, it will generate a menu that includes ingredients with a relaxing effect.

[1836] 5. Ordering missing ingredients

[1837] If a required ingredient is in short supply when creating a menu, the server automatically places an additional order through the food delivery system, allowing users to have the ingredients they need delivered to them.

[1838] 6. Display of menu information

[1839] The menus generated by the server are sent to a device such as a smartphone and displayed to the user, allowing the user to easily check and implement their daily meal plan.

[1840] Specific examples

[1841] Specifically, use the following prompt:

[1842] I have chicken, tomatoes, and lettuce in my refrigerator. I'd like to get more vitamin C. I'm currently feeling fatigued. Based on this information, please suggest a meal plan that will be healthy and help reduce stress.

[1843] In this way, this invention realizes a system that integrates information on ingredients in the refrigerator, health information, and emotional information, creates nutritionally balanced menus, and allows users to easily order the necessary ingredients, enabling users to lead healthy and stress-reducing eating habits.

[1844] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1845] Step 1:

[1846] Enter ingredient information

[1847] The user inputs the type, quantity, and expiration date of ingredients in the refrigerator into a device such as a smartphone. The device sends this information to a server, which then stores the received ingredient information in a database. The input information might include, for example, "2 pieces of chicken," "3 tomatoes," "1 lettuce," and "expiration date 2023-10-10."

[1848] (Input: ingredient information, Output: ingredient information stored in the database)

[1849] Step 2:

[1850] Enter health information

[1851] Users input information about the nutrients they particularly want to consume and any chronic illnesses they have into a device such as a smartphone. The device then sends this information to a server, which then stores the received information in a database. Specific examples include information such as "I want to take in a lot of vitamin C" and "I have no chronic illnesses."

[1852] (Input: health information, Output: health information stored in the database)

[1853] Step 3:

[1854] emotional information recognition

[1855] The emotion engine uses the smartphone's voice recognition function and camera to recognize the user's emotions. For example, it can detect "fatigue" from the user's voice and facial expressions. The device sends this information to the server, which then stores it in a database.

[1856] (Input: Emotion information, Output: Emotion information stored in the database)

[1857] Step 4:

[1858] Database queries

[1859] The server queries and retrieves stored ingredient information, health information, and emotional information from the database. This aggregates the information set necessary for menu generation. For example, information such as "chicken, tomato, lettuce," "high intake of vitamin C," and "feeling tired" is retrieved.

[1860] (Input: database query, Output: retrieved information set)

[1861] Step 5:

[1862] Menu generation

[1863] The server then uses a generative AI model to generate an appropriate menu based on the acquired information set. The menu generation algorithm takes into account nutritional balance, the amount of ingredients used, health conditions, and emotional state. For example, it might suggest dishes such as chicken stewed in tomato sauce, which has a relaxing effect, lettuce salad, and oranges, which are rich in vitamin C.

[1864] (Input: information set, output: generated menu)

[1865] Step 6:

[1866] Ordering missing ingredients

[1867] The server checks whether the ingredients required for the generated menu are missing, and if so, automatically places an additional order through the food delivery system, for example, ordering an "orange."

[1868] (Input: Generated menu, check for missing ingredients, Output: Order request)

[1869] Step 7:

[1870] Display menu information

[1871] The server sends the generated menu and ordering information for missing ingredients to a device such as a smartphone, where the user can check the proposed menu and added order details on the device.

[1872] (Input: generated menu, order information, Output: information displayed on the terminal)

[1873] Through these steps, the system can integrate the user's ingredients, health status, and emotional state to generate an optimal menu and automatically order any missing ingredients.

[1874] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1875] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1876] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1877] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1878] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1879] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1880] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1881] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1882] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1883] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1884] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1885] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1886] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1887] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1888] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1889] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1890] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1891] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1892] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1893] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1894] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1895] The following is further disclosed regarding the above embodiment.

[1896] (Claim 1)

[1897] A means to input information about ingredients in the refrigerator,

[1898] A means for storing the ingredient information in a database;

[1899] A means for generating a menu that takes into consideration nutritional balance based on the stored ingredient information;

[1900] A means for displaying the generated menu information on a terminal;

[1901] A system including:

[1902] (Claim 2)

[1903] A means for inputting information on school lunch menus at home;

[1904] A means for storing the menu information of the school lunch in a database;

[1905] A means for generating a menu that does not overlap with the stored school lunch menu information;

[1906] The system of claim 1 further comprising:

[1907] (Claim 3)

[1908] A way to input information about chronic illnesses and specific nutritional needs,

[1909] A means for storing information about the chronic illness and the nutritional intake that the patient particularly desires to take in a database;

[1910] A means for generating a menu that takes into account stored chronic illnesses and nutritional information you particularly want to consume;

[1911] The system of claim 1 further comprising:

[1912] "Example 1"

[1913] (Claim 1)

[1914] A means to input information about ingredients in the refrigerator,

[1915] A means for storing the ingredient information in a database;

[1916] A means for generating a menu that takes into consideration nutritional balance based on the stored ingredient information;

[1917] A means for generating a menu based on the prompt sentence using a generative AI model;

[1918] A means for displaying the generated menu information on a terminal;

[1919] A system including:

[1920] (Claim 2)

[1921] A means for inputting information on school lunch menus at home and storing the information in a database;

[1922] A means for generating a menu that does not overlap with the stored school lunch menu information;

[1923] The system of claim 1 further comprising:

[1924] (Claim 3)

[1925] A way to input information about chronic illnesses and specific nutritional needs and store it in a database,

[1926] A means for generating a menu that takes into account stored chronic illnesses and nutritional information you particularly want to consume;

[1927] The system of claim 1 further comprising:

[1928] "Application Example 1"

[1929] (Claim 1)

[1930] A means to input information about ingredients in the refrigerator,

[1931] A means for storing the ingredient information in a database;

[1932] A means for generating a menu that takes into consideration nutritional balance based on the stored ingredient information;

[1933] A way to scan purchased ingredients and enter information,

[1934] A means for displaying the generated menu information on a terminal;

[1935] A system including:

[1936] (Claim 2)

[1937] A means for inputting information on school lunch menus at home;

[1938] A means for storing the menu information of the school lunch in a database;

[1939] A means for generating a menu that does not overlap with the stored school lunch menu information;

[1940] Further including means for inputting purchased food ingredient information;

[1941] 10. The system of claim 1.

[1942] (Claim 3)

[1943] A way to input information about chronic illnesses and specific nutritional needs,

[1944] A means for storing information about the chronic illness and the nutritional intake that the patient particularly desires to take in a database;

[1945] A means for generating a menu that takes into account stored chronic illnesses and nutritional information you particularly want to consume;

[1946] A means for generating menus using a generative AI model based on health information;

[1947] The system of claim 1 further comprising:

[1948] "Example 2: Combining Emotion Engines"

[1949] (Claim 1)

[1950] a means for inputting information about ingredients;

[1951] A means for storing the ingredient information in a database;

[1952] A means for generating a menu that takes into consideration nutritional balance based on the stored ingredient information;

[1953] A means for displaying the generated menu information on a terminal;

[1954] means for recognizing user emotion information;

[1955] means for storing the emotion information in a database;

[1956] A means for generating a menu taking into account the stored emotional information;

[1957] A system including:

[1958] (Claim 2)

[1959] A means for inputting information on school lunch menus at home;

[1960] A means for storing the menu information of the school lunch in a database;

[1961] A means for generating a menu that does not overlap with the stored school lunch menu information;

[1962] The system of claim 1 further comprising:

[1963] (Claim 3)

[1964] A way to input information about chronic illnesses and specific nutritional needs,

[1965] A means for storing information about the chronic illness and the nutritional intake that the patient particularly desires to take in a database;

[1966] A means for generating a menu that takes into account stored chronic illnesses and nutritional information you particularly want to consume;

[1967] The system of claim 1 further comprising:

[1968] "Application example 2 when combining emotion engines"

[1969] (Claim 1)

[1970] A means to input information about ingredients in the refrigerator,

[1971] A means for storing the ingredient information in a database;

[1972] A means for generating a menu that takes into consideration nutritional balance based on the stored ingredient information;

[1973] a means for inputting a user's health information;

[1974] means for storing said health information in a database;

[1975] A means for generating a menu taking into account the stored health information;

[1976] means for recognizing user emotion information;

[1977] means for storing the emotion information in a database;

[1978] A means for generating a menu taking into account the stored emotional information;

[1979] A way to order ingredients that are missing when proposing a menu,

[1980] A means for displaying the generated menu information on a terminal;

[1981] A system including:

[1982] (Claim 2)

[1983] A means for inputting information on school lunch menus at home;

[1984] A means for storing the menu information of the school lunch in a database;

[1985] A means for generating a menu that does not overlap with the stored school lunch menu information;

[1986] The system of claim 1 further comprising:

[1987] (Claim 3)

[1988] A way to input information about chronic illnesses and specific nutritional needs,

[1989] A means for storing information about the chronic illness and the nutritional intake that the patient particularly desires to take in a database;

[1990] A means for generating a menu that takes into account stored chronic illnesses and nutritional information you particularly want to consume;

[1991] The system of claim 1 further comprising: [Explanation of symbols]

[1992] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means to input information about ingredients in the refrigerator, A means for storing the ingredient information in a database; A means for generating a menu that takes into consideration nutritional balance based on the stored ingredient information; A means for displaying the generated menu information on a terminal; A system including:

2. A means for inputting information on school lunch menus at home; A means for storing the menu information of the school lunch in a database; A means for generating a menu that does not overlap with the stored school lunch menu information; The system of claim 1 further comprising:

3. A way to input information about chronic illnesses and specific nutritional needs, A means for storing information about the chronic illness and the nutritional intake that the patient particularly desires to take in a database; A means for generating a menu that takes into account stored chronic illnesses and nutritional information you particularly want to consume; The system of claim 1 further comprising:

Images