System

Abstract

A system is provided.SOLUTION: The system includes a means for inputting basic information of a user, a means for acquiring price information of a residential area, a means for generating an optimum menu by analyzing the input basic information and the acquired price information, a means for generating a necessary food list and an estimated budget on the basis of the generated menu, and a means for presenting the food list and the estimated budget to the user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] With today's home cooking, it is difficult to create a menu that takes into account individual circumstances, such as health conditions, allergies, and food intolerances. Another problem is the time and effort required to plan a menu and shop for ingredients. This makes it difficult to maintain a healthy, balanced diet and further reduces motivation to continue cooking at home. Furthermore, food waste in each household is also a problem. The purpose of this invention is to solve these issues and enable users to enjoy stress-free home cooking. [Means for solving the problem]

[0005] The present invention includes a means for inputting a user's basic information and a means for acquiring price information for the user's area of ​​residence. It also includes a means for analyzing the input basic information and acquired price information to generate an optimal menu, a means for generating a list of necessary ingredients and a projected budget based on the generated menu, and a means for presenting the generated ingredient list and projected budget to the user. It also includes a means for inputting the user's ingredient information, a means for analyzing the input ingredient information and desired seasonings to suggest dishes that can be made, and a means for analyzing the cooking status during cooking and providing cooking advice in real time, thereby providing comprehensive support for the user's home cooking. It also includes a means for listing necessary ingredients based on the generated menu, a means for ordering the listed ingredients from partner companies, a means for setting up a delivery request after the order is confirmed, and a means for notifying the user of the delivery request, thereby providing a system that consistently supports the user from procuring ingredients to cooking.

[0006] "Basic user information" refers to information that indicates individual circumstances, such as the user's health condition, allergies, dislikes, family composition, desired budget, preparation period, and desired menu direction.

[0007] "Residential area price information" is the latest supermarket and market price information for the user's area, including price data for specific food ingredients.

[0008] "Menu generation" is the process of creating healthy and balanced meal combinations based on the user's basic information and price information for their area.

[0009] The "ingredient list" is a specific list of ingredients required to create the generated menu, and lists the items that need to be purchased.

[0010] The "projected budget" is the estimated total cost of ingredients required to execute the generated menu.

[0011] "Ingredient information" refers to information about specific ingredients that the user has, and refers to data entered in the form of images, audio, text, etc.

[0012] "Seasoning preference" is information indicating the type of seasoning the user desires for the dish (e.g., Japanese style, Western style, etc.).

[0013] "Real-time cooking advice" refers to specific cooking steps and advice that are instantly provided based on information entered by the user while cooking.

[0014] "Ingredient list creation" is the process of identifying ingredients that are in short supply or that need to be purchased additionally based on the generated menu, and compiling them into a list.

[0015] "Partner companies" refer to supermarkets and online retailers with which we partner to purchase and deliver the suggested ingredients.

[0016] "Order confirmation" is the process by which the user confirms the order details of the suggested ingredients and officially notifies the partner company of their intention to purchase.

[0017] "Delivery request" refers to the arrangement of having the purchased ingredients delivered to the location specified by the user after the order is confirmed.

[0018] "Notification" refers to communication that a system uses to convey important information to a user, and may take the form of email, app notification, or similar. [Brief explanation of the drawings]

[0019] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0020] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0021] First, the terms used in the following description will be explained.

[0022] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0023] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0024] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0025] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0026] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0027] [First embodiment]

[0028] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0029] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0030] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0031] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0032] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0033] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0034] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0035] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0036] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0037] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0038] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0039] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0040] The present invention provides a system that allows users to easily create menus that suit their needs and efficiently procure the necessary ingredients. Hereinafter, an embodiment of the system will be described in detail.

[0041] System configuration

[0042] This system operates in cooperation with user terminals, a server, and partner companies. The user terminal accepts user input and communicates with the server. The server analyzes the information received from the user and is responsible for creating menus, suggesting recipes, and managing ingredients. Partner companies also provide and deliver the necessary ingredients.

[0043] Creating a menu

[0044] 1. Enter basic user information

[0045] Users input their health condition, allergies, food preferences, family composition, desired budget, preparation period, and menu direction. This information is sent to the server from the user's terminal.

[0046] Example: "Health condition: good, Allergies: peanuts, Dislikes: tomatoes, Family size: 4, Desired budget: 5,000 yen per week, Menu preference: healthy"

[0047] 2. Data analysis and menu generation by the server

[0048] The server receives the basic information and obtains price information for the user's area. Based on this information, it analyzes and generates a menu for the specified period. The generated menu is optimized taking into account the user's health condition, family composition, and budget.

[0049] Example: The server compares the price information it has obtained and generates a menu such as "Breakfast: oatmeal, Lunch: salad, Dinner: stir-fried chicken and vegetables."

[0050] 3. Present the ingredients list and estimated budget

[0051] The server calculates the necessary ingredients and estimated budget based on the generated menu, and sends them to the user's device, allowing the user to understand the ingredients they need to purchase and their budget.

[0052] Example: Based on the estimated budget calculated by the server, the user is presented with a list of ingredients needed: oatmeal, lettuce, chicken, carrots, along with the total budget.

[0053] Recipe suggestions and cooking assistance

[0054] 1. Enter ingredient information

[0055] Users input information about the ingredients in their refrigerator. This information can be entered using images, voice, or text. They can also input their desired seasonings at the same time.

[0056] Example: Take a photo of the ingredients in your refrigerator and enter the text "I want it to have a Japanese flavor."

[0057] 2. Server analyzes ingredient information and suggests recipes

[0058] The server analyzes the submitted ingredient information and seasoning preferences to identify available ingredients, and then suggests recipes that match the user's desired seasonings based on these ingredients.

[0059] Example: Based on the ingredients identified through photo analysis: chicken, cabbage, and carrots, the system suggests "teriyaki chicken."

[0060] 3. Real-time cooking advice

[0061] While cooking, users can use their device to send cooking information to the server using a camera or voice input, and the server analyzes the received information and provides appropriate advice in real time.

[0062] For example, say, "The chicken is browned," and the server will advise, "Next, add the sauce and cook over medium heat for 5 minutes."

[0063] Food ordering and delivery

[0064] 1. List and order ingredients

[0065] The server automatically generates a list of ingredients required based on the menu, which then becomes the basis for ordering from partner companies.

[0066] For example, "Oatmeal, lettuce, chicken, and carrots" is listed and an order request is sent to the partner company.

[0067] 2. Confirm and confirm your order

[0068] The user confirms the order details and presses the confirmation button to complete the order. The confirmed order is then officially sent to the partner company via the server.

[0069] Example: When a user checks the order details and presses the confirm button, the order is confirmed and sent to the partner company.

[0070] 3. Delivery request and notification

[0071] After the order is confirmed, the server arranges for delivery and sets the delivery request details. The user is notified of the estimated delivery time, etc.

[0072] Example: The server requests delivery from a courier company and sends a notification to the user saying, "Estimated delivery time: tomorrow at 2:00 PM."

[0073] This system allows users to lead a healthy diet without hassle and to efficiently source ingredients. In addition, it provides recipe suggestions and real-time cooking advice, so even beginners can cook with confidence.

[0074] The processing flow will be explained below.

[0075] Creating a menu

[0076] Step 1:

[0077] Users use a device to enter information about their health condition, allergies, food preferences, family composition, desired budget, preparation period, and desired menu direction into a dedicated form.

[0078] Specifically, the user enters the following information: "health condition: good, allergies: peanuts, dislikes: tomatoes, family size: 4 people, desired budget: 5,000 yen per week, menu preference: healthy."

[0079] Step 2:

[0080] The device sends the user's input data to the server.

[0081] Specifically, the terminal uses a REST API to send the user's input data to the server in JSON format.

[0082] Step 3:

[0083] The server analyzes the basic information received from the user and obtains price information for the user's area of ​​residence via an API.

[0084] The server identifies the area where the user lives and obtains the latest price information from supermarkets and markets in that area.

[0085] Step 4:

[0086] The server generates a menu for the specified period based on the price information obtained and the user's basic information.

[0087] The menu generation algorithm creates a list of recipes that suit your health status, family composition, and budget, and selects the optimal menu.

[0088] Step 5:

[0089] The server calculates the required ingredients list and estimated budget based on the generated menu and sends this information to the user's terminal.

[0090] Based on the estimated budget calculated by the server, a list of "necessary ingredients: oatmeal, lettuce, chicken, carrots" and the total budget are created and sent to the terminal.

[0091] Recipe suggestions and cooking assistance

[0092] Step 1:

[0093] The user uses the terminal to input information about the ingredients in the refrigerator (either images, voice, or text) and desired seasonings.

[0094] As a specific example, a user takes a photo of the ingredients in the refrigerator and enters the text, "I want it to have a Japanese flavor."

[0095] Step 2:

[0096] The terminal transmits the inputted ingredient information and desired seasoning to the server.

[0097] Using the REST API, image data and seasoning requests are sent to the server in JSON or XML format.

[0098] Step 3:

[0099] The server analyzes the received ingredient information and suggests dishes that can be made based on the available ingredients and desired seasonings.

[0100] Image analysis algorithms identify the type of ingredient and select the most suitable dish from a database of recipes.

[0101] Step 4:

[0102] The server sends the suggested recipe information to the user's terminal and displays it.

[0103] A list of suggested dishes, cooking instructions, and any additional ingredients needed are sent in JSON format, which is then displayed on the device.

[0104] Step 5:

[0105] While the user is cooking, the cooking status is sent to the server in real time using the device's camera or voice input.

[0106] Specifically, the cooking status is input by voice, such as "The chicken is browned."

[0107] Step 6:

[0108] The server analyzes the cooking status received and sends appropriate real-time cooking advice to the user's device.

[0109] Check cooking instructions and send advice such as, "Next, add the sauce and cook over medium heat for another 5 minutes."

[0110] Food ordering and delivery

[0111] Step 1:

[0112] The server lists the missing ingredients based on the generated menu.

[0113] It compares the menu with the user's list of ingredients on hand and lists any ingredients that are lacking, such as "oatmeal, lettuce, chicken, and carrots."

[0114] Step 2:

[0115] The server generates an order request to the partner company based on the listed ingredient information.

[0116] Specifically, order information is generated in JSON format and sent to partner companies via API.

[0117] Step 3:

[0118] The server sends an order request to the user's terminal, which the user confirms.

[0119] The user receives an email or app notification confirming their order.

[0120] Step 4:

[0121] The user checks the order details and presses the confirmation button on the terminal.

[0122] When the confirm button is clicked, the confirmation information is sent to the server.

[0123] Step 5:

[0124] The server finally transmits the confirmed order information to the partner company and confirms the order.

[0125] Order confirmation information is sent to partner companies via API.

[0126] Step 6:

[0127] After the order is confirmed, the server sets up a delivery request and arranges an appropriate delivery time.

[0128] Use the delivery company's API to set up delivery and set the expected delivery time for the user.

[0129] Step 7:

[0130] The server notifies the user of the delivery request and provides delivery information.

[0131] Information such as "Estimated delivery time: tomorrow 2pm" is sent to users via email or app notifications.

[0132] Example 1

[0133] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0134] In today's busy lifestyles, it is extremely difficult for users to plan healthy, balanced meals and efficiently source the necessary ingredients. Creating meals that take into account specific health conditions or allergies, and receiving real-time cooking advice, are even more challenging. There is a need for a system that can solve these challenges and enable users to effortlessly create appropriate meals and efficiently obtain the necessary ingredients.

[0135] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0136] In this invention, the server includes means for inputting basic information about the user, means for acquiring price information for the user's area of ​​residence, means for analyzing the input basic information and the acquired price information to generate an optimal menu, means for generating a list of necessary ingredients and a projected budget based on the generated menu, means for presenting the ingredient list and the projected budget to the user, means for analyzing the input ingredient information, photo data, and audio data to identify available ingredients, means for suggesting appropriate recipes based on the user's desired seasonings, and means for analyzing the cooking status during cooking and providing cooking advice in real time. This allows users to effortlessly plan healthy menus, procure ingredients, and efficiently manage the cooking process.

[0137] "User" refers to an individual or family member who uses the system to create menus, source ingredients, and receive cooking advice.

[0138] "Basic information" refers to information necessary for creating a menu and selecting ingredients, such as the user's health condition, allergies, food dislikes, family composition, desired budget, and menu direction.

[0139] "Price information for residential area" refers to price information for specific items in the area where the user lives, and is used for planning meals and calculating budgets.

[0140] The "ingredient list" refers to a list of specific ingredients that should be purchased based on the generated menu.

[0141] "Projected Budget" refers to an estimate of the budget required to purchase the necessary ingredients within a specified period of time.

[0142] "Photo data" refers to image information of ingredients and cooking processes that users take and upload to the system.

[0143] "Voice data" refers to voice instructions, questions, and cooking status reports provided by the user to the system.

[0144] A "recipe" refers to a method of making a dish based on specific ingredients and seasonings.

[0145] "Cooking status" refers to information about the progress and current state of cooking, and is reported to the system in real time.

[0146] "Real-time advice" refers to advice or instructions provided to the user immediately during the cooking process.

[0147] The present invention provides a system that allows users to easily create menus that suit their needs and efficiently procure the necessary ingredients. This system operates in cooperation with a user terminal, a server, and partner companies. Detailed embodiments of the system are described below.

[0148] System configuration

[0149] User Device

[0150] The user terminal is a device such as a smartphone or PC that accepts user input and communicates with the server. The user uses the terminal to input basic information about themselves, ingredients, cooking status, etc.

[0151] server

[0152] The server analyzes the information received from the user and is responsible for generating menus, suggesting recipes, and managing ingredients. It uses a cloud computing platform (such as Google® Cloud) to obtain price information for the user's area. It also performs data analysis using Python and R, and generates menus and recipes using machine learning models such as TENSORFLOW® and PyTorch.

[0153] Overview of program processing

[0154] Enter basic user information

[0155] Users input information such as their health condition, allergies, food preferences, family composition, desired budget, preparation period, menu direction, etc. via their smartphone or PC, and this information is sent from the device to the server.

[0156] Example: "Health condition: good, Allergies: peanuts, Dislikes: tomatoes, Family size: 4, Desired budget: 5,000 yen per week, Menu preference: healthy"

[0157] Data analysis and menu generation by server

[0158] The server receives the basic information sent and retrieves price information for the area from a Google Cloud database. Based on this information, it performs data analysis and generates a menu for the specified period.

[0159] Example: The server generates a menu such as "Breakfast: oatmeal, Lunch: salad, Dinner: stir-fried chicken and vegetables" based on price information and basic user information.

[0160] Present the ingredients list and estimated budget

[0161] The server calculates the necessary ingredients and estimated budget based on the generated menu, and sends this to the user's device, allowing the user to understand the ingredients they need to purchase and their budget.

[0162] Example: Provide the user with a list of ingredients needed: oatmeal, lettuce, chicken, carrots, along with a budget total.

[0163] Entering ingredient information and suggesting recipes

[0164] Entering ingredient information

[0165] Users can input information about the ingredients in their refrigerator using images, voice, or text, and can also input their desired seasonings for cooking.

[0166] Example: Take a photo of the ingredients in your refrigerator and enter the text "I want to make it Japanese flavor."

[0167] Server analyzes ingredients and suggests recipes

[0168] The server analyzes the submitted ingredient information and seasoning preferences, identifies available ingredients, and then suggests recipes using those ingredients. Image analysis uses technologies such as OpenCV and YOLOv3.

[0169] Example: Suggest "Teriyaki chicken" based on "Ingredients identified through photo analysis: chicken, cabbage, carrots."

[0170] Real-time cooking advice

[0171] While cooking, users can use their device to send cooking status information to the server using a camera or voice input. The server then analyzes the received information and provides appropriate advice in real time.

[0172] For example, say, "The chicken is browned," and the server will advise, "Next, add the sauce and cook over medium heat for 5 minutes."

[0173] Food ordering and delivery

[0174] Listing and ordering ingredients

[0175] The server automatically generates a list of ingredients required based on the menu and sends an order request to the partner company.

[0176] For example, "Oatmeal, lettuce, chicken, and carrots" is listed and an order request is sent to the partner company.

[0177] Confirm and confirm your order

[0178] The user checks the order details in the app and presses the confirmation button to complete the order. The confirmed order is then sent to the partner company via the server.

[0179] Example: When the user confirms the order details and presses the confirm button, the order is confirmed with the partner company.

[0180] Delivery requests and notifications

[0181] After the order is confirmed, the server arranges delivery with a delivery company and notifies the user of the estimated delivery time.

[0182] Example: The server requests delivery from a courier company and the user is notified that "Estimated delivery time: tomorrow 2pm."

[0183] Prompt Sentence Examples

[0184] Please enter your health condition, allergy information, family composition, desired budget, creation period, and menu direction.

[0185] "Please enter a photo of the ingredients in your refrigerator and your desired seasoning."

[0186] Please check your order details and press the confirm button.

[0187] In this way, the system helps users effortlessly lead a healthy and balanced diet. Users can easily create menus, efficiently source ingredients, and receive recipe suggestions and real-time cooking advice.

[0188] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0189] Step 1:

[0190] Users enter basic information such as their health condition, allergies, food preferences, family composition, desired budget, preparation period, and menu direction. Using a smartphone or computer, they enter the information into a dedicated app or web form and press the send button. The server receives the basic information entered and stores it in a database.

[0191] Input: health condition, allergy information, disliked ingredients, family composition, desired budget, preparation period, menu direction.

[0192] Output: Notification that basic information has been saved to the database.

[0193] Specific action: The user opens the app on their smartphone, enters the required information into the input form, and taps the submit button.

[0194] Step 2:

[0195] The server retrieves price information from a Google Cloud database based on the user's residential area information. It analyzes the price information and generates the optimal menu for the period specified by the user. It performs data analysis using Python and R, and generates the menu using machine learning models in TensorFlow and PyTorch.

[0196] Input: User's basic information, price information for the area where you live.

[0197] Output: The generated menu information.

[0198] Specific operation: The server calls the Google Cloud API to obtain price data and runs a Python script to generate a menu.

[0199] Step 3:

[0200] The server calculates the necessary ingredients and estimated budget based on the generated menu information. The calculation results are converted into HTML format and sent to the user's device, allowing the user to understand the ingredients they need to purchase and their budget.

[0201] Input: Generated menu information.

[0202] Output: A list of ingredients needed and a projected budget.

[0203] Specific operation: The server runs the program to calculate the required ingredient list and budget, and sends the results as an HTML file to the user's terminal.

[0204] Step 4:

[0205] Users input information about ingredients in their refrigerator using images, voice, or text. They take photos of the ingredients with their smartphone camera and enter their desired seasonings in text. The device then sends this information to the server.

[0206] Input: Information about ingredients in the refrigerator (images, audio, text), desired seasoning.

[0207] Output: Ingredient information and seasoning requests sent to the server.

[0208] What it does: The user takes a photo of the contents of their refrigerator with their smartphone and sends the photo and a text message about the seasoning through the app.

[0209] Step 5:

[0210] The server analyzes the received ingredient information and seasoning preferences. It identifies ingredients using image recognition technologies such as OpenCV and YOLOv3, and understands the seasoning preferences using natural language processing. Based on the results, it suggests an appropriate recipe.

[0211] Input: Submitted ingredient information and seasoning preference.

[0212] Output: A suggested recipe.

[0213] Specific operation: The server launches an image analysis engine to identify ingredients, analyzes desired seasonings using a natural language processing engine, and searches for and suggests appropriate recipes from a database.

[0214] Step 6:

[0215] While cooking, the user sends the cooking status to the server using a camera or voice input, and the server analyzes the received cooking status in real time and provides appropriate cooking advice.

[0216] Input: Image and audio data of the cooking process.

[0217] Output: Real-time cooking advice.

[0218] How it works: The user records the cooking process in real time using the camera and microphone on their smartphone and sends the footage to the server, which analyzes it and provides instant cooking advice.

[0219] Step 7:

[0220] The server automatically generates a list of ingredients based on the menu and sends an order request to partner companies via an API.

[0221] Input: A list of ingredients needed.

[0222] Output: An order request to the partner company.

[0223] Specific operation: The server references the MySQL (registered trademark) database to obtain the necessary ingredient information, and then calls the partner company's API to send an order request.

[0224] Step 8:

[0225] The user checks the order details on the app and presses the confirmation button to complete the order. The server then sends the confirmed order information to the partner company.

[0226] Input: User confirms and confirms order details.

[0227] Output: Confirmed order submission.

[0228] Specific operation: The user opens the order confirmation screen of the app, checks the order details, and taps the confirm button. The server receives this information and sends a notification of order confirmation to the partner company.

[0229] Step 9:

[0230] After the order is confirmed, the server arranges delivery with a courier and notifies the user of the estimated delivery time.

[0231] Input: Confirmed order information.

[0232] Output: Notification of delivery request details and estimated delivery time.

[0233] Specific operation: The server calls the delivery company's API to request delivery and sends the estimated delivery time to the user via push notification.

[0234] (Application example 1)

[0235] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0236] In today's world, it is extremely time-consuming for users to plan appropriate menus and efficiently procure the necessary ingredients while taking into account their own health condition, allergies, and budget. It is also difficult to receive appropriate recipe suggestions and cooking advice in real time. Furthermore, the lack of online links to purchase ingredients persists, making food procurement a complex process. Given this background, there is a demand for a system that allows users to easily lead a healthy diet.

[0237] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0238] In this invention, the server includes means for inputting basic information such as the user's health condition, allergies, budget, etc., means for acquiring price information for the user's area of ​​residence, means for analyzing the input basic information and the acquired price information to generate an optimal menu, means for generating a list of necessary ingredients and a projected budget based on the generated menu, means for providing an online link for purchasing ingredients based on the generated menu, and means for presenting the ingredient list and projected budget to the user. This enables users to easily generate optimal menus based on their own health condition, allergies, and budget, efficiently procure the necessary ingredients, and easily purchase them via the online link.

[0239] - "Basic user information" refers to personal information such as the user's health condition, allergies, disliked ingredients, family composition, desired budget, preparation period, and menu direction.

[0240] "Price information for residential area" refers to price information for food and ingredients in the area where the user lives.

[0241] A "menu" refers to a plan of suitable meals for a specific period of time based on the user's basic information.

[0242] The "ingredient list" is a list of ingredients required based on the generated menu.

[0243] The "projected budget" is the total amount of purchase costs predicted based on the ingredient list.

[0244] "Online Links" are links to online stores that allow users to easily purchase the ingredients they need.

[0245] "Ingredient information" is detailed information about foods and ingredients currently owned by the user.

[0246] "Seasoning preference" refers to a user's preference regarding the type of seasoning or flavor they desire for a particular dish.

[0247] "Real-time advice" refers to instructions and advice on cooking methods and next steps that users receive immediately while cooking.

[0248] "Partners" are companies and service providers that work together to supply ingredients and food products and provide them to users.

[0249] A "delivery request" is an arrangement for a delivery service to deliver ordered ingredients or food to a location specified by the user.

[0250] This invention is a system that allows users to create optimal menus that take into consideration their own health conditions, allergies, budget, etc., and to efficiently procure the necessary ingredients. A specific embodiment of this system will be described in detail below.

[0251] System configuration

[0252] This system works in conjunction with user terminals, a server, and partner companies. The user terminal accepts user input and communicates with the server. The server analyzes the information received from the user and is responsible for creating menus, suggesting recipes, and managing ingredients. Partner companies also provide and deliver the necessary ingredients.

[0253] Entering basic user information

[0254] Users input their health condition, allergies, food preferences, family composition, desired budget, preparation period, and menu direction into a user terminal, which then transmits this information to the server.

[0255] Data analysis and menu generation by server

[0256] The server receives the basic information and obtains price information for the user's area. Based on this information, it analyzes and generates a menu for the specified period. The generated menu is optimized taking into account the user's health condition, family composition, and budget.

[0257] Present the ingredients list and estimated budget

[0258] The server calculates the necessary ingredients and estimated budget based on the generated menu and sends them to the user's device. This allows the user to understand the ingredients they need to purchase and their budget. An online link to purchase ingredients is also provided, making it easy for the user to purchase ingredients.

[0259] Food ordering and delivery

[0260] 1. The server lists the ingredients needed based on the generated menu.

[0261] 2. The listed ingredients are ordered from partner companies.

[0262] 3. Once the order details are confirmed, the server will arrange for delivery and notify the user of the delivery request.

[0263] Hardware and software used

[0264] The hardware used includes the smartphone operated by the user, servers, and network communication devices. The software used includes Python programs, REST APIs, and data analysis algorithms. Specific technologies include using generative AI models to generate menus and suggest recipes.

[0265] Examples of concrete examples and prompts

[0266] Examples:

[0267] When users open the app and enter their basic information, the app displays the optimal meal plan based on their health status and budget. It also provides a list of ingredients and links to online shopping sites. By clicking on the link, users can easily purchase the ingredients they need. Users can also enter information about the ingredients in their refrigerator and receive real-time recipe suggestions.

[0268] Example prompt sentence:

[0269] "Create an app that generates optimal meals based on the user's health, allergies, budget, etc., and provides the necessary ingredients and online purchasing links."

[0270] In this way, the present invention allows users to easily lead a healthy diet and efficiently source ingredients. Furthermore, the system provides real-time cooking advice, making cooking easy even for beginners.

[0271] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0272] Step 1:

[0273] The user uses a terminal to input basic information such as health status, allergies, food preferences, family composition, budget, and menu direction. The input basic information is sent from the terminal to the server.

[0274] Input: User's health condition, allergies, disliked ingredients, family composition, budget, menu direction

[0275] Output: User basic information sent to the server

[0276] Specific operation: The user enters information into the input form on the device and presses the "Submit" button.

[0277] Step 2:

[0278] The server retrieves price information for the user's area. The server uses the Internet or existing databases to retrieve the latest food price information for the area the user enters.

[0279] Input: User's residential area

[0280] Output: Price information for your area

[0281] What happens: The server sends requests to APIs and third-party databases to retrieve price information.

[0282] Step 3:

[0283] The server analyzes basic information and price information to generate the optimal menu for the specified period. Using a generative AI model, the server generates a menu based on food price information, taking into account the user's health condition, budget, and allergies.

[0284] Input: User basic information, price information

[0285] Output: Generated menu data

[0286] How it works: The generative AI model processes the input data and generates the optimal menu.

[0287] Step 4:

[0288] The server calculates the necessary ingredients and estimated budget based on the generated menu and sends it to the user's device. This allows the user to understand the ingredients they need to purchase and their budget. An online purchase link is also provided.

[0289] Input: Generated menu data

[0290] Output: Ingredients list, estimated budget, online purchase link

[0291] What it does: The server analyzes the menu data, calculates the ingredients list and estimated budget, and generates online purchasing links for each ingredient.

[0292] Step 5:

[0293] The user confirms the order details using the terminal and presses the confirm button. Once the order details are confirmed, the server sends an order request to the partner company, and the order is officially confirmed.

[0294] Inputs: Ingredient list, estimated budget, online purchasing links

[0295] Output: Order request to partner

[0296] Specific operation: When the user confirms the order details on the terminal and presses the confirmation button, the server sends the order request to the partner company.

[0297] Step 6:

[0298] After the order is confirmed, the server arranges for delivery and notifies the user of the delivery request. After receiving confirmation from the partner company, the server requests delivery from the delivery company and notifies the user of the estimated delivery time.

[0299] Input: Order confirmation information

[0300] Output: Notification of delivery request details

[0301] Specific operation: The server sends a delivery request to the delivery company and provides notification information to the user.

[0302] Step 7:

[0303] The user inputs the ingredient information they have stored into the server from their device. The ingredient information can be input as an image, voice, or text. The user's seasoning preferences are also input at the same time.

[0304] Input: User's food information, seasoning preference

[0305] Output: Ingredient information and desired seasoning data sent to the server

[0306] Specific operation: The user takes a photo of the ingredients in the refrigerator using the device and enters their desired seasoning.

[0307] Step 8:

[0308] The server analyzes the inputted ingredient information and seasoning preferences and proposes appropriate recipes. The server uses a generative AI model to analyze the inputted data and generate recipes that match the user's preferences.

[0309] Input: Ingredient information, desired seasoning

[0310] Output: Suggested recipe

[0311] How it works: The generative AI model analyzes ingredient information and seasoning preferences to generate an appropriate recipe.

[0312] Step 9:

[0313] While the user is cooking, the cooking status is sent to the server via camera or voice input from the device, and the server analyzes the received cooking status and provides appropriate advice in real time.

[0314] Input: Cooking status data (images, audio)

[0315] Output: Real-time cooking advice

[0316] Specific operation: While cooking, the user takes a photo or voice-over of the situation on the device, and the server analyzes it and advises the next step.

[0317] In this way, at each processing step, the user, server, and terminal work together to input, analyze, and output data, enabling users to easily create healthy menus and efficiently procure the ingredients needed to make them.

[0318] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0319] The present invention combines a system that allows users to easily create personalized menus and efficiently source the necessary ingredients with an emotion engine. This system analyzes the user's emotions and provides recipes and cooking advice accordingly, creating a more personalized experience. Below, we will explain in detail how to implement this system.

[0320] System configuration

[0321] This system operates by combining user devices, a server, collaboration with partner companies, and an emotion engine. The user device accepts user input, communicates with the server, and acquires emotion data through the emotion engine. The server analyzes the information received from the user and is responsible for generating menus, suggesting recipes, and managing ingredients. Partner companies also provide and deliver the necessary ingredients.

[0322] Creating a menu and incorporating emotions

[0323] 1. Enter basic user information

[0324] Users input their health condition, allergies, disliked ingredients, family composition, desired budget, preparation period, and desired menu direction. In addition, the user can also input their current emotional state. This information is sent to the server from the user's device.

[0325] Example: "Health condition: good, Allergies: peanuts, Dislikes: tomatoes, Family size: 4 people, Desired budget: 5,000 yen per week, Menu preference: healthy, Emotional state: tired"

[0326] 2. Data analysis and menu generation by the server

[0327] The server receives the basic information and emotional state sent, and obtains price information for the user's area via an API. Based on this information, it analyzes and generates a menu for the specified period. This also includes menu suggestions to soothe the user's emotions based on their emotional state.

[0328] For example, if you are tired, a simple yet balanced meal will be suggested: "Breakfast: Oatmeal, Lunch: Simple Salad, Dinner: Stir-fried Chicken and Vegetables."

[0329] 3. Present the ingredients list and estimated budget

[0330] The server calculates the necessary ingredients and estimated budget based on the generated menu, and sends them to the user's device, allowing the user to understand the ingredients they need to purchase and their budget.

[0331] Example: Based on the estimated budget calculated by the server, the user is presented with a list of ingredients needed: oatmeal, lettuce, chicken, carrots, along with the total budget.

[0332] Recipe suggestions and cooking assistance

[0333] 1. Enter ingredient information

[0334] Users can input information about the ingredients in their refrigerator using images, voice, or text, as well as their desired seasoning and current emotional state.

[0335] Example: Take a photo of the ingredients in the refrigerator and enter the text "I want it to have a Japanese flavor" and "Emotional state: Calm."

[0336] 2. Analysis of food information and emotional state by the server

[0337] The server analyzes the transmitted ingredient information and emotional state, and suggests dishes that can be made based on the available ingredients and desired seasonings. It also suggests recipes that will help you relax depending on your emotional state.

[0338] Example: Based on the ingredients identified through photo analysis: chicken, cabbage, and carrots, the system suggests "teriyaki chicken."

[0339] 3. Real-time cooking advice

[0340] While cooking, users can use their device to send cooking status information to the server using a camera or voice input. The server then analyzes the received cooking status and the user's emotional state and provides appropriate advice in real time.

[0341] For example, if you say, "The chicken is browned," the server will advise you, "Next, add the sauce and cook over medium heat for 5 minutes," and also send a relaxing message depending on your emotional state.

[0342] Food ordering and delivery

[0343] 1. List and order ingredients

[0344] The server automatically generates a list of ingredients required based on the menu, which then becomes the basis for ordering from partner companies.

[0345] For example, "Oatmeal, lettuce, chicken, and carrots" is listed and an order request is sent to the partner company.

[0346] 2. Confirm and confirm your order

[0347] The user confirms the order details and presses the confirmation button to complete the order. The confirmed order is then officially sent to the partner company via the server.

[0348] Example: When a user checks the order details and presses the confirm button, the order is confirmed and sent to the partner company.

[0349] 3. Delivery request and notification

[0350] After the order is confirmed, the server arranges for delivery and sets the delivery request details. The user is notified of the estimated delivery time, etc.

[0351] Example: The server requests delivery from a courier company and sends a notification to the user saying, "Estimated delivery time: tomorrow at 2:00 PM."

[0352] This system allows users to eat healthy and hassle-free, and to efficiently source ingredients. Furthermore, an emotion engine is incorporated into the recipe suggestions and real-time cooking advice, providing support tailored to the user's emotional state. This increases the enjoyment and satisfaction of cooking, and is expected to be even more practical.

[0353] The processing flow will be explained below.

[0354] Creating a menu and incorporating emotions

[0355] Step 1:

[0356] Using a device, users enter their health condition, allergies, food dislikes, family composition, desired budget, preparation period, desired menu direction, and current emotional state into a dedicated form.

[0357] Specifically, the user enters the following information: "Health condition: good, Allergies: peanuts, Dislikes: tomatoes, Family size: 4 people, Desired budget: 5,000 yen per week, Menu direction: healthy, Emotional state: tired."

[0358] Step 2:

[0359] The device sends the user's input data to the server.

[0360] Specifically, the device uses a REST API to send the user's input data in JSON format to the server, along with their emotional state.

[0361] Step 3:

[0362] The server analyzes the user's basic information and emotional state and obtains price information for the user's area via an API.

[0363] The server identifies the area where the user lives and obtains the latest price information from supermarkets and markets in that area.

[0364] Step 4:

[0365] The server generates a menu for the specified period based on the basic information, emotional state, and price information analyzed by the server.

[0366] Specifically, the menu generation algorithm selects recipes that suit a user's health status, family composition, and budget, and suggests easy-to-prepare dishes for tired users, such as "Breakfast: oatmeal, Lunch: simple salad, Dinner: stir-fried chicken with vegetables."

[0367] Step 5:

[0368] Based on the generated menu, the server calculates the list of ingredients needed and the estimated budget, and sends it to the user's terminal.

[0369] Based on the ingredient list and estimated budget calculated by the server, a list of "necessary ingredients: oatmeal, lettuce, chicken, carrots" and the total budget are created and sent to the terminal.

[0370] Recipe suggestions and cooking assistance

[0371] Step 1:

[0372] The user uses the terminal to input information about the ingredients in the refrigerator (either images, voice, or text), desired seasoning, and their current emotional state.

[0373] As a specific example, a user takes a photo of the ingredients in the refrigerator and enters text such as "I want it to have a Japanese flavor" and "Emotional state: calm."

[0374] Step 2:

[0375] The terminal transmits the inputted information on ingredients, desired seasoning, and emotional state to the server.

[0376] Using the REST API, image data and seasoning requests are sent in JSON or XML format.

[0377] Step 3:

[0378] The server analyzes the received ingredient information, emotional state, and seasoning preferences, and suggests dishes that match the available ingredients and the user's emotions.

[0379] An image analysis algorithm identifies the type of ingredient and suggests the most suitable dish, such as "Teriyaki Chicken," from recipes in the database according to "Emotional State: Calm."

[0380] Step 4:

[0381] The server sends the suggested recipe information to the user's terminal and displays it.

[0382] A list of suggested dishes, cooking instructions, and any additional ingredients needed are sent in JSON format and displayed on the device.

[0383] Step 5:

[0384] While the user is cooking, the cooking status is sent to the server in real time using the device's camera or voice input.

[0385] Specifically, the cooking status is input by voice, such as "the chicken is browned."

[0386] Step 6:

[0387] The server analyzes the cooking status and emotional state received and sends appropriate real-time cooking advice to the user's device.

[0388] It will check the cooking steps, advise you on how to cook the food, and then give you advice such as "Next, add the sauce and cook over medium heat for another 5 minutes," and also send you relaxing messages based on your emotional state.

[0389] Food ordering and delivery

[0390] Step 1:

[0391] The server lists the missing ingredients based on the generated menu.

[0392] It compares the menu with the user's list of ingredients on hand and lists any ingredients that are lacking, such as "oatmeal, lettuce, chicken, and carrots."

[0393] Step 2:

[0394] The server generates an order request to the partner company based on the listed ingredient information.

[0395] Specifically, order information is generated in JSON format and sent to partner companies via API.

[0396] Step 3:

[0397] The server sends an order request to the user's terminal, which the user confirms.

[0398] The user receives an email or app notification confirming their order.

[0399] Step 4:

[0400] The user checks the order details and presses the confirmation button on the terminal.

[0401] When the confirm button is clicked, the confirmation information is sent to the server.

[0402] Step 5:

[0403] The server finally transmits the confirmed order information to the partner company and confirms the order.

[0404] Order confirmation information is sent to partner companies via API.

[0405] Step 6:

[0406] After the order is confirmed, the server sets up a delivery request and arranges an appropriate delivery time.

[0407] Use the delivery company's API to set up delivery and set the expected delivery time for the user.

[0408] Step 7:

[0409] The server notifies the user of the delivery request and provides delivery information.

[0410] Information such as "Estimated delivery time: tomorrow 2pm" is sent to users via email or app notifications.

[0411] Example 2

[0412] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0413] Conventional menu generation systems only suggest menus based on the user's basic information and price information, making it impossible to realize personalized menu suggestions that take the user's emotional state into account. Furthermore, there was no system that properly analyzed the user's ingredient information and cooking status to provide real-time cooking advice, resulting in low user convenience and satisfaction. Furthermore, there was a lack of efficient ways to order the necessary ingredients and process delivery procedures, making it difficult for users to procure ingredients without hassle.

[0414] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for inputting the user's basic information and emotional state; means for acquiring price information for the user's residential area; means for analyzing the input basic information, emotional state, and acquired price information to generate an optimal menu; means for generating a necessary ingredient list and estimated budget based on the generated menu; and means for presenting the ingredient list and estimated budget to the user. This enables personalized menu suggestions that take the user's emotional state into consideration. The server also includes means for inputting the user's ingredient information and emotional state; means for suggesting dishes that can be made by analyzing the input ingredient information, emotional state, and seasoning preferences; and means for analyzing the cooking status and emotional state during cooking to provide cooking advice in real time. This allows the user to receive appropriate cooking advice in real time. The server also includes means for listing necessary ingredients based on the generated menu; means for ordering the listed ingredients from partner companies; means for setting a delivery request after the order is confirmed; and means for notifying the user of the delivery request. This allows the user to efficiently procure the necessary ingredients without any hassle.

[0415] "Basic user information" refers to information regarding the user's health condition, allergies, food preferences, family composition, desired budget, and creation period.

[0416] "Emotional state" is information that represents the user's current mood or mental state.

[0417] "Price information" is information about the prices of common goods and services in the user's area.

[0418] "Menu" means a plan of meals that a User intends to consume within a specified time period.

[0419] The "ingredient list" is a list of ingredients required based on the generated menu.

[0420] The "projected budget" is the estimated cost required to purchase ingredients calculated based on the ingredient list.

[0421] "Ingredient information" refers to information about the type, quantity, and condition of ingredients currently owned by the user.

[0422] "Seasoning preference" refers to the user's preference regarding the taste and cooking method of the food.

[0423] "Cooking status" is information about the progress and process of cooking.

[0424] "Real-time cooking advice" refers to specific cooking instructions and advice provided instantly based on the user's cooking status while cooking.

[0425] "Partner companies" refer to companies that provide and deliver ingredients.

[0426] "Ordering" refers to the act of formally requesting ingredients needed by a user from a partner company.

[0427] A "delivery request" is a request to arrange for the delivery of the ingredients to the user after the order is confirmed.

[0428] "Notification" is a means of informing users of delivery requests and other important information.

[0429] This invention is a system that allows users to easily create menus that suit their needs and efficiently procure the necessary ingredients, and it also combines an emotion engine. This system analyzes the user's emotions and provides recipes and cooking advice accordingly, realizing a more personalized experience.

[0430] System configuration

[0431] This system operates by combining a user device, a server, collaboration with partner companies, and an emotion engine. The user device accepts input of the user's basic information and emotional state, communicates with the server, and transmits the data to the server. The server analyzes the information received from the user and is responsible for generating menus, suggesting recipes, and managing ingredients. Partner companies also provide and deliver the necessary ingredients.

[0432] Entering the user's basic information and emotional state

[0433] Users can use devices such as smartphones or tablets to input their health status, allergies, food preferences, family composition, desired budget, preparation period, and desired menu direction. In addition, users can also input their current emotional state. This information is sent from the device to the server.

[0434] Examples:

[0435] "Health: Good, Allergies: Peanuts, Dislikes: Tomatoes, Family size: 4, Desired budget: 5,000 yen per week, Menu preference: Healthy, Emotional state: Tired"

[0436] Data analysis and menu generation by server

[0437] The server receives the basic information and emotional state sent, and obtains price information for the user's area via an API. Based on this information, it analyzes and generates a menu for the specified period. This also includes menu suggestions to soothe the user's emotions based on their emotional state.

[0438] Examples:

[0439] If you're tired, the app suggests easy-to-prepare, nutritionally balanced meals: "Breakfast: Oatmeal, Lunch: Simple Salad, Dinner: Stir-fried Chicken and Vegetables."

[0440] Present the ingredients list and estimated budget

[0441] The server calculates the necessary ingredients and estimated budget based on the generated menu and sends this to the user's terminal, allowing the user to understand the ingredients they need to purchase and their budget.

[0442] Examples:

[0443] The user is given a list of "Ingredients Needed: Oatmeal, Lettuce, Chicken, Carrots" along with a budget total.

[0444] Enter ingredients and emotional state

[0445] Users can input information about the ingredients in their refrigerator using images, voice, or text, as well as their seasoning preferences and current emotional state.

[0446] Examples:

[0447] Take a photo of the ingredients in your refrigerator and enter the text "I want it to have a Japanese flavor" and "Emotional state: calm."

[0448] Server-based analysis of ingredients and emotional state

[0449] The server analyzes the transmitted ingredient information and emotional state, and suggests dishes that can be made based on the available ingredients and desired seasonings. It also suggests recipes that will help you relax depending on your emotional state.

[0450] Examples:

[0451] Based on the ingredients identified through photo analysis: chicken, cabbage, and carrots, we suggest "chicken teriyaki."

[0452] Real-time cooking advice

[0453] While cooking, the user uses the device to send the cooking status to the server using the camera or voice input, and the server analyzes this information and the user's emotional state to provide appropriate advice in real time.

[0454] Examples:

[0455] If you say, "The chicken is browned," the server will advise you, "Next, add the sauce and cook over medium heat for 5 minutes," and it will also send you a relaxing message based on your emotional state.

[0456] Ordering and delivery of necessary ingredients

[0457] The server lists the ingredients needed based on the menu and sends an order request to the partner company. The user confirms the order on the device and completes the order by pressing the confirmation button. After confirmation, the server arranges for delivery and notifies the user of the delivery request.

[0458] Examples:

[0459] "Oatmeal, lettuce, chicken, carrots" is listed and an order request is sent to the partner company. When the user confirms the order and presses the confirm button, the server requests delivery from the delivery company and notifies the user that "Estimated delivery time: tomorrow 2:00 PM."

[0460] Prompt Sentence Examples

[0461] Based on the following information, it generates a weekly menu and ingredient list, and suggests recipes that take into account your emotional state.

[0462] "Health: Good, Allergies: Peanuts, Disliked Food: Tomatoes, Family Size: 4, Budget: 5,000 yen per week, Menu Direction: Healthy, Emotional State: Tired"

[0463] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0464] Step 1:

[0465] Entering the user's basic information and emotional state

[0466] Users input their health condition, allergies, food preferences, family composition, desired budget, preparation period, and desired menu direction into the device, along with their emotional state. Input is done through a form in the app.

[0467] Input: health condition, allergies, disliked ingredients, family composition, desired budget, preparation period, desired menu direction, emotional state

[0468] Output: Basic information and emotional state data

[0469] Specific actions: The user opens the app on their smartphone, enters the required information into the form, and presses the submit button.

[0470] Step 2:

[0471] Sending basic information and emotional state to the server

[0472] The terminal transmits the input basic information and emotional state to the server.

[0473] Input: Basic information and emotional state data

[0474] Output: Data sent to the server

[0475] Specific operation: When the user presses the send button, the device sends the data to the server via the Internet.

[0476] Step 3:

[0477] Data analysis and menu generation by server

[0478] The server analyzes the received basic information and emotional state, obtains price information for the user's area via an API, and runs a program to generate a menu based on this information.

[0479] Input: Basic information, emotional state, price information

[0480] Data processing / calculation: Match basic information with emotional state and generate a menu based on price information in the area of ​​residence.

[0481] Output: Generated menu

[0482] Specific operation: The server calls the API to obtain price information and then generates a menu using an algorithm.

[0483] Step 4:

[0484] Present the ingredients list and estimated budget

[0485] The server calculates the list of ingredients needed and the estimated budget based on the generated menu, and transmits this to the user terminal.

[0486] Input: Generated menu

[0487] Data processing / calculation: Calculate the ingredients list and budget based on the menu.

[0488] Output: Ingredient list, estimated budget

[0489] What it does: The server sends the calculation results to the device, and the app displays them.

[0490] Step 5:

[0491] Enter ingredients and emotional state

[0492] The user inputs information about the ingredients in the refrigerator in the form of images, voice, or text, as well as their desired seasoning and current emotional state.

[0493] Input: Ingredient information, seasoning preference, emotional state

[0494] Output: Data sent to the server

[0495] Specific operation: The user takes a photo of the ingredients in the refrigerator and sends it by text inputting their seasoning preference and emotional state.

[0496] Step 6:

[0497] Server-based analysis of ingredients and emotional state

[0498] The server analyzes the transmitted ingredient information and emotional state, and suggests dishes that can be made based on the available ingredients and desired seasonings.

[0499] Input: Ingredient information, emotional state, desired seasoning

[0500] Data processing / computation: Identifying ingredients using image recognition, using sentiment analysis algorithms

[0501] Output: Suggested dishes

[0502] Specific operation: The server analyzes the image, searches for recipes in the database, and suggests the most suitable dish.

[0503] Step 7:

[0504] Real-time cooking advice

[0505] While cooking, the user uses the device to send cooking information to the server using a camera or voice input, and the server analyzes the information and provides appropriate advice in real time.

[0506] Input: Cooking status, emotional state

[0507] Data processing / calculation: Analyzing cooking status and emotional state

[0508] Output: Real-time cooking advice

[0509] Specific behavior: The server analyzes the voice input and suggests the next cooking steps.

[0510] Step 8:

[0511] List and order the ingredients you need

[0512] The server lists the ingredients needed based on the menu and sends an order request to partner companies.

[0513] Input: Menu

[0514] Data processing / calculation: Creating an ingredient list

[0515] Output: Order request

[0516] What happens: The server creates a list and sends it to partner companies through the ordering system.

[0517] Step 9:

[0518] Confirm and confirm your order

[0519] The user checks the order details on the terminal and presses the confirmation button to complete the order. The server then officially sends the confirmed order to the partner company.

[0520] Input: Ingredients list

[0521] Output: Order confirmation notification

[0522] Specific operation: The user checks the order details on the confirmation screen and presses the confirm button.

[0523] Step 10:

[0524] Delivery requests and notifications

[0525] After the order is confirmed, the server arranges for delivery and sets the delivery request details. The user is notified of the estimated delivery time, etc.

[0526] Input: Order Information

[0527] Data processing / calculation: Delivery schedule setting

[0528] Output: Delivery notification

[0529] Specific operation: The server requests the delivery company and sends a notification to the user.

[0530] (Application example 2)

[0531] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0532] In recent years, it has become increasingly important for users to easily create menus that suit their own health and emotional state and to efficiently source ingredients. However, conventional systems do not provide menu suggestions or cooking advice that take the user's emotions into account, making it difficult to provide a personalized experience. In addition, sourcing ingredients and requesting delivery is time-consuming for users, which is also an issue that lacks efficiency.

[0533] The specification processing by the specification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for inputting basic information about the user, means for acquiring price information for the residential area, means for analyzing the input basic information and the acquired price information to generate an optimal menu, means for generating a list of necessary ingredients and a projected budget based on the generated menu, means for analyzing the user's emotions, means for suggesting a personalized menu based on the analyzed emotions, and means for presenting the ingredient list and the projected budget to the user. This enables personalized menu suggestions and efficient ingredient procurement according to the user's emotional state.

[0534] "Means for inputting basic user information" refers to the means by which users input their health condition, allergies, food preferences, family composition, desired budget, preparation period, desired menu direction, etc. into the system.

[0535] "Means for obtaining price information in the residential area" refers to means for obtaining price information in the area where the user lives from the Internet or an external database.

[0536] "Means for analyzing the input basic information and the acquired price information to generate the optimal menu" refers to means for automatically generating the optimal menu for the user based on the basic information input by the user and the acquired price information.

[0537] The "means for generating a list of necessary ingredients and a projected budget based on the generated menu" refers to a means for calculating and generating a list of necessary ingredients and a projected budget based on the generated menu.

[0538] "Means for analyzing user emotions" refers to means for analyzing the user's facial image and voice data to detect their emotional state at that time.

[0539] The "means for making personalized menu suggestions based on analyzed emotions" is a means for suggesting a menu that is suitable for each individual user according to the user's emotional state.

[0540] The "means for presenting the ingredient list and estimated budget to the user" refers to a means for displaying the generated ingredient list and estimated budget on the user's terminal.

[0541] "Means for inputting information about ingredients owned by the user" refers to means for inputting information about ingredients owned by the user using images, voice, text, etc.

[0542] "Means for suggesting dishes that can be made by analyzing the inputted ingredient information and desired seasoning" refers to means for suggesting dishes that can be made based on the ingredient information and desired seasoning that the user inputs.

[0543] The "means for analyzing the cooking situation during cooking and providing cooking advice in real time" is a means for analyzing the cooking situation using images and sounds and providing cooking advice in real time.

[0544] The "means for generating cooking advice based on the emotional state of the user" is a means for analyzing the emotional state of the user and generating cooking advice that matches that emotion.

[0545] The "means for providing feedback according to the cooking situation and emotional state" is a means for providing appropriate feedback to the user based on the cooking situation and the user's emotional state.

[0546] The "means for listing necessary ingredients based on the generated menu" is a means for listing necessary ingredients based on the generated menu.

[0547] The "means for ordering the listed ingredients from partner companies" is a means for automatically ordering the listed ingredients from external partner companies.

[0548] "Means for requesting delivery after confirming an order" refers to a means for a user to arrange delivery after confirming and confirming the order details.

[0549] "Means for notifying the user of the details of the delivery request" refers to means for notifying the user of information such as the order details and the estimated delivery time.

[0550] The "means for proposing an optimal delivery time based on the emotional state of the user" is a means for proposing an optimal delivery time by taking into account the emotional state of the user.

[0551] To implement this invention, a system is required that can efficiently acquire a user's basic information and emotional state, and then use that information to create menus, procure ingredients, and provide cooking advice. This system operates primarily in cooperation with the user's terminal, a server, and partner companies. Specific embodiments are described below.

[0552] Hardware and Software Configuration

[0553] Hardware:

[0554] User device: A smartphone or tablet with a camera and internet connection.

[0555] Server: Responsible for data processing and analysis. A server with internet access.

[0556] software:

[0557] OpenCV: A library used for face recognition and emotion detection.

[0558] Emotion Detection API: An external API that performs emotion analysis.

[0559] Flask: A Python framework for building server-side recipe generation engines.

[0560] Delivery Service API: An API for handling food orders with partner companies.

[0561] System Functions and Processing

[0562] 1. Sentiment analysis and menu suggestions

[0563] The server first analyzes the user's emotional state. It captures a facial image using the user's device's camera and sends the image data to the Emotion Detection API. The API analyzes the image and returns the user's emotional state. Based on this emotional state, it suggests an appropriate menu from a database stored on the Flask server.

[0564] Example: "Since we detected the emotion 'tired' from your facial image, we will suggest a simple yet nutritionally balanced meal: 'Stir-fried chicken with vegetables.'"

[0565] 2. List and order ingredients

[0566] The server automatically generates a list of ingredients based on the proposed menu and calculates the estimated budget. This is then displayed on the user's device, and once the user confirms and confirms, the order information is sent to the partner company via the Delivery Service API.

[0567] For example: "I ordered lettuce, chicken, and carrots. The total cost is 1000 yen."

[0568] 3. Analysis of cooking status and provision of advice

[0569] The server receives and analyzes camera images and voice input from the user while cooking, providing real-time advice at the appropriate time based on the cooking situation. By taking the user's emotional state into consideration, the system provides feedback aimed at maximizing relaxation.

[0570] For example: "I've determined that the chicken is browned. Now add the sauce and cook over medium heat for 5 minutes."

[0571] Example prompt:

[0572] Below are some example prompts that users can use to suggest meals based on their emotional state:

[0573] Perform "face photo capture," "detect emotional state," and suggest "emotion-based menus."

[0574] summary

[0575] This invention allows users to receive menu suggestions tailored to their emotional state, efficiently source the ingredients they need, and receive real-time advice based on the cooking situation, providing a more personalized cooking experience.

[0576] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0577] Program processing steps

[0578] Step 1: Enter basic information

[0579] The user inputs their health condition, allergies, food preferences, family composition, desired budget, preparation period, and desired menu direction. This information is sent from the user terminal.

[0580] Input: Personal information entered by the user

[0581] Output: Basic information sent from the user device to the server

[0582] Step 2: Sentiment Analysis

[0583] Users take a picture of their face with their smartphone camera and send the image data to the Emotion Detection API, which analyzes the image and returns the user's emotional state.

[0584] Input: User's face image

[0585] Output: Sentiment analysis result (e.g., tired, relaxed)

[0586] Step 3: Obtaining price information

[0587] The server obtains price information from the Internet or external databases based on the user's residential area information.

[0588] Input: User's residential area information

[0589] Output: Price information data

[0590] Step 4: Create a menu

[0591] The server generates the optimal menu based on the input basic information, emotional state, and acquired price information. It uses the recipe database on the Flask server to suggest personalized menus.

[0592] Input: Basic information, emotional state, price information

[0593] Output: Optimal menu data

[0594] Step 5: Generate a list of ingredients needed and a projected budget

[0595] The server calculates and generates a list of necessary ingredients and a projected budget based on the generated menu.

[0596] Input: Optimal menu data

[0597] Output: Ingredient list, estimated budget

[0598] Step 6: Present to the user

[0599] The server sends the generated ingredient list and estimated budget to the user's terminal, where the user can check it on the screen.

[0600] Input: Ingredient list, estimated budget

[0601] Output: Information displayed on the user's terminal

[0602] Step 7: Order ingredients

[0603] The user confirms the ingredients list and budget and confirms the order. The server then sends this order information to the partner company via the Delivery Service API.

[0604] Input: Ingredient list, estimated budget, user order confirmation

[0605] Output: Order data for partner companies

[0606] Step 8: Arrange delivery and notify

[0607] Once the order is confirmed, the server sends a delivery request to the delivery company and notifies the user of the estimated time. The server also suggests the optimal delivery time, taking into account the user's emotional state.

[0608] Input: Order data, user emotional state

[0609] Output: Request to courier, notification to user

[0610] Step 9: Cooking Tips

[0611] While cooking, users take pictures of their cooking progress with their smartphone camera and report the progress by voice. The server analyzes the information and sends cooking advice in real time. The server also analyzes the user's emotional state and sends corresponding relaxing messages.

[0612] Input: Images and audio of cooking status

[0613] Output: Real-time cooking advice, emotion-based messages

[0614] Through this series of processes, users can receive personalized menu suggestions, efficiently source the necessary ingredients, and receive accurate advice while cooking.

[0615] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0616] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0617] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0618] [Second embodiment]

[0619] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0620] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0621] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0622] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0623] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0624] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0625] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0626] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0627] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0628] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0629] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0630] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0631] The present invention provides a system that allows users to easily create menus that suit their needs and efficiently procure the necessary ingredients. Hereinafter, an embodiment of the system will be described in detail.

[0632] System configuration

[0633] This system operates in cooperation with user terminals, a server, and partner companies. The user terminal accepts user input and communicates with the server. The server analyzes the information received from the user and is responsible for creating menus, suggesting recipes, and managing ingredients. Partner companies also provide and deliver the necessary ingredients.

[0634] Creating a menu

[0635] 1. Enter basic user information

[0636] Users input their health condition, allergies, food preferences, family composition, desired budget, preparation period, and menu direction. This information is sent to the server from the user's terminal.

[0637] Example: "Health condition: good, Allergies: peanuts, Dislikes: tomatoes, Family size: 4, Desired budget: 5,000 yen per week, Menu preference: healthy"

[0638] 2. Data analysis and menu generation by the server

[0639] The server receives the basic information and obtains price information for the user's area. Based on this information, it analyzes and generates a menu for the specified period. The generated menu is optimized taking into account the user's health condition, family composition, and budget.

[0640] Example: The server compares the price information it has obtained and generates a menu such as "Breakfast: oatmeal, Lunch: salad, Dinner: stir-fried chicken and vegetables."

[0641] 3. Present the ingredients list and estimated budget

[0642] The server calculates the necessary ingredients and estimated budget based on the generated menu, and sends them to the user's device, allowing the user to understand the ingredients they need to purchase and their budget.

[0643] Example: Based on the estimated budget calculated by the server, the user is presented with a list of ingredients needed: oatmeal, lettuce, chicken, carrots, along with the total budget.

[0644] Recipe suggestions and cooking assistance

[0645] 1. Enter ingredient information

[0646] Users input information about the ingredients in their refrigerator. This information can be entered using images, voice, or text. They can also input their desired seasonings at the same time.

[0647] Example: Take a photo of the ingredients in your refrigerator and enter the text "I want it to have a Japanese flavor."

[0648] 2. Server analyzes ingredient information and suggests recipes

[0649] The server analyzes the submitted ingredient information and seasoning preferences to identify available ingredients, and then suggests recipes that match the user's desired seasonings based on these ingredients.

[0650] Example: Based on the ingredients identified through photo analysis: chicken, cabbage, and carrots, the system suggests "teriyaki chicken."

[0651] 3. Real-time cooking advice

[0652] While cooking, users can use their device to send cooking information to the server using a camera or voice input, and the server analyzes the received information and provides appropriate advice in real time.

[0653] For example, say, "The chicken is browned," and the server will advise, "Next, add the sauce and cook over medium heat for 5 minutes."

[0654] Food ordering and delivery

[0655] 1. List and order ingredients

[0656] The server automatically generates a list of ingredients required based on the menu, which then becomes the basis for ordering from partner companies.

[0657] For example, "Oatmeal, lettuce, chicken, and carrots" is listed and an order request is sent to the partner company.

[0658] 2. Confirm and confirm your order

[0659] The user confirms the order details and presses the confirmation button to complete the order. The confirmed order is then officially sent to the partner company via the server.

[0660] Example: When a user checks the order details and presses the confirm button, the order is confirmed and sent to the partner company.

[0661] 3. Delivery request and notification

[0662] After the order is confirmed, the server arranges for delivery and sets the delivery request details. The user is notified of the estimated delivery time, etc.

[0663] Example: The server requests delivery from a courier company and sends a notification to the user saying, "Estimated delivery time: tomorrow at 2:00 PM."

[0664] This system allows users to lead a healthy diet without hassle and to efficiently source ingredients. In addition, it provides recipe suggestions and real-time cooking advice, so even beginners can cook with confidence.

[0665] The processing flow will be explained below.

[0666] Creating a menu

[0667] Step 1:

[0668] Users use a device to enter information about their health condition, allergies, food preferences, family composition, desired budget, preparation period, and desired menu direction into a dedicated form.

[0669] Specifically, the user enters the following information: "health condition: good, allergies: peanuts, dislikes: tomatoes, family size: 4 people, desired budget: 5,000 yen per week, menu preference: healthy."

[0670] Step 2:

[0671] The device sends the user's input data to the server.

[0672] Specifically, the terminal uses a REST API to send the user's input data to the server in JSON format.

[0673] Step 3:

[0674] The server analyzes the basic information received from the user and obtains price information for the user's area of ​​residence via an API.

[0675] The server identifies the area where the user lives and obtains the latest price information from supermarkets and markets in that area.

[0676] Step 4:

[0677] The server generates a menu for the specified period based on the price information obtained and the user's basic information.

[0678] The menu generation algorithm creates a list of recipes that suit your health status, family composition, and budget, and selects the optimal menu.

[0679] Step 5:

[0680] The server calculates the required ingredients list and estimated budget based on the generated menu and sends this information to the user's terminal.

[0681] Based on the estimated budget calculated by the server, a list of "necessary ingredients: oatmeal, lettuce, chicken, carrots" and the total budget are created and sent to the terminal.

[0682] Recipe suggestions and cooking assistance

[0683] Step 1:

[0684] The user uses the terminal to input information about the ingredients in the refrigerator (either images, voice, or text) and desired seasonings.

[0685] As a specific example, a user takes a photo of the ingredients in the refrigerator and enters the text, "I want it to have a Japanese flavor."

[0686] Step 2:

[0687] The terminal transmits the inputted ingredient information and desired seasoning to the server.

[0688] Using the REST API, image data and seasoning requests are sent to the server in JSON or XML format.

[0689] Step 3:

[0690] The server analyzes the received ingredient information and suggests dishes that can be made based on the available ingredients and desired seasonings.

[0691] Image analysis algorithms identify the type of ingredient and select the most suitable dish from a database of recipes.

[0692] Step 4:

[0693] The server sends the suggested recipe information to the user's terminal and displays it.

[0694] A list of suggested dishes, cooking instructions, and any additional ingredients needed are sent in JSON format, which is then displayed on the device.

[0695] Step 5:

[0696] While the user is cooking, the cooking status is sent to the server in real time using the device's camera or voice input.

[0697] Specifically, the cooking status is input by voice, such as "The chicken is browned."

[0698] Step 6:

[0699] The server analyzes the cooking status received and sends appropriate real-time cooking advice to the user's device.

[0700] Check cooking instructions and send advice such as, "Next, add the sauce and cook over medium heat for another 5 minutes."

[0701] Food ordering and delivery

[0702] Step 1:

[0703] The server lists the missing ingredients based on the generated menu.

[0704] It compares the menu with the user's list of ingredients on hand and lists any ingredients that are lacking, such as "oatmeal, lettuce, chicken, and carrots."

[0705] Step 2:

[0706] The server generates an order request to the partner company based on the listed ingredient information.

[0707] Specifically, order information is generated in JSON format and sent to partner companies via API.

[0708] Step 3:

[0709] The server sends an order request to the user's terminal, which the user confirms.

[0710] The user receives an email or app notification confirming their order.

[0711] Step 4:

[0712] The user checks the order details and presses the confirmation button on the terminal.

[0713] When the confirm button is clicked, the confirmation information is sent to the server.

[0714] Step 5:

[0715] The server finally transmits the confirmed order information to the partner company and confirms the order.

[0716] Order confirmation information is sent to partner companies via API.

[0717] Step 6:

[0718] After the order is confirmed, the server sets up a delivery request and arranges an appropriate delivery time.

[0719] Use the delivery company's API to set up delivery and set the expected delivery time for the user.

[0720] Step 7:

[0721] The server notifies the user of the delivery request and provides delivery information.

[0722] Information such as "Estimated delivery time: tomorrow 2pm" is sent to users via email or app notifications.

[0723] Example 1

[0724] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0725] In today's busy lifestyles, it is extremely difficult for users to plan healthy, balanced meals and efficiently source the necessary ingredients. Creating meals that take into account specific health conditions or allergies, and receiving real-time cooking advice, are even more challenging. There is a need for a system that can solve these challenges and enable users to effortlessly create appropriate meals and efficiently obtain the necessary ingredients.

[0726] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0727] In this invention, the server includes means for inputting basic information about the user, means for acquiring price information for the user's area of ​​residence, means for analyzing the input basic information and the acquired price information to generate an optimal menu, means for generating a list of necessary ingredients and a projected budget based on the generated menu, means for presenting the ingredient list and the projected budget to the user, means for analyzing the input ingredient information, photo data, and audio data to identify available ingredients, means for suggesting appropriate recipes based on the user's desired seasonings, and means for analyzing the cooking status during cooking and providing cooking advice in real time. This allows users to effortlessly plan healthy menus, procure ingredients, and efficiently manage the cooking process.

[0728] "User" refers to an individual or family member who uses the system to create menus, source ingredients, and receive cooking advice.

[0729] "Basic information" refers to information necessary for creating a menu and selecting ingredients, such as the user's health condition, allergies, food dislikes, family composition, desired budget, and menu direction.

[0730] "Price information for residential area" refers to price information for specific items in the area where the user lives, and is used for planning meals and calculating budgets.

[0731] The "ingredient list" refers to a list of specific ingredients that should be purchased based on the generated menu.

[0732] "Projected Budget" refers to an estimate of the budget required to purchase the necessary ingredients within a specified period of time.

[0733] "Photo data" refers to image information of ingredients and cooking processes that users take and upload to the system.

[0734] "Voice data" refers to voice instructions, questions, and cooking status reports provided by the user to the system.

[0735] A "recipe" refers to a method of making a dish based on specific ingredients and seasonings.

[0736] "Cooking status" refers to information about the progress and current state of cooking, and is reported to the system in real time.

[0737] "Real-time advice" refers to advice or instructions provided to the user immediately during the cooking process.

[0738] The present invention provides a system that allows users to easily create menus that suit their needs and efficiently procure the necessary ingredients. This system operates in cooperation with a user terminal, a server, and partner companies. Detailed embodiments of the system are described below.

[0739] System configuration

[0740] User Device

[0741] The user terminal is a device such as a smartphone or PC that accepts user input and communicates with the server. The user uses the terminal to input basic information about themselves, ingredients, cooking status, etc.

[0742] server

[0743] The server analyzes the information received from the user and is responsible for generating menus, suggesting recipes, and managing ingredients. It uses a cloud computing platform (such as Google Cloud) to obtain price information for the user's area. It also performs data analysis using Python and R, and generates menus and recipes using machine learning models such as TensorFlow and PyTorch.

[0744] Overview of program processing

[0745] Enter basic user information

[0746] Users input information such as their health condition, allergies, food preferences, family composition, desired budget, preparation period, menu direction, etc. via their smartphone or PC, and this information is sent from the device to the server.

[0747] Example: "Health condition: good, Allergies: peanuts, Dislikes: tomatoes, Family size: 4, Desired budget: 5,000 yen per week, Menu preference: healthy"

[0748] Data analysis and menu generation by server

[0749] The server receives the basic information sent and retrieves price information for the area from a Google Cloud database. Based on this information, it performs data analysis and generates a menu for the specified period.

[0750] Example: The server generates a menu such as "Breakfast: oatmeal, Lunch: salad, Dinner: stir-fried chicken and vegetables" based on price information and basic user information.

[0751] Present the ingredients list and estimated budget

[0752] The server calculates the necessary ingredients and estimated budget based on the generated menu, and sends this to the user's device, allowing the user to understand the ingredients they need to purchase and their budget.

[0753] Example: Provide the user with a list of ingredients needed: oatmeal, lettuce, chicken, carrots, along with a budget total.

[0754] Entering ingredient information and suggesting recipes

[0755] Entering ingredient information

[0756] Users can input information about the ingredients in their refrigerator using images, voice, or text, and can also input their desired seasonings for cooking.

[0757] Example: Take a photo of the ingredients in your refrigerator and enter the text "I want to make it Japanese flavor."

[0758] Server analyzes ingredients and suggests recipes

[0759] The server analyzes the submitted ingredient information and seasoning preferences, identifies available ingredients, and then suggests recipes using those ingredients. Image analysis uses technologies such as OpenCV and YOLOv3.

[0760] Example: Suggest "Teriyaki chicken" based on "Ingredients identified through photo analysis: chicken, cabbage, carrots."

[0761] Real-time cooking advice

[0762] While cooking, users can use their device to send cooking status information to the server using a camera or voice input. The server then analyzes the received information and provides appropriate advice in real time.

[0763] For example, say, "The chicken is browned," and the server will advise, "Next, add the sauce and cook over medium heat for 5 minutes."

[0764] Food ordering and delivery

[0765] Listing and ordering ingredients

[0766] The server automatically generates a list of ingredients required based on the menu and sends an order request to the partner company.

[0767] For example, "Oatmeal, lettuce, chicken, and carrots" is listed and an order request is sent to the partner company.

[0768] Confirm and confirm your order

[0769] The user checks the order details in the app and presses the confirmation button to complete the order. The confirmed order is then sent to the partner company via the server.

[0770] Example: When the user confirms the order details and presses the confirm button, the order is confirmed with the partner company.

[0771] Delivery requests and notifications

[0772] After the order is confirmed, the server arranges delivery with a delivery company and notifies the user of the estimated delivery time.

[0773] Example: The server requests delivery from a courier company and the user is notified that "Estimated delivery time: tomorrow 2pm."

[0774] Prompt Sentence Examples

[0775] Please enter your health condition, allergy information, family composition, desired budget, creation period, and menu direction.

[0776] "Please enter a photo of the ingredients in your refrigerator and your desired seasoning."

[0777] Please check your order details and press the confirm button.

[0778] In this way, the system helps users effortlessly lead a healthy and balanced diet. Users can easily create menus, efficiently source ingredients, and receive recipe suggestions and real-time cooking advice.

[0779] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0780] Step 1:

[0781] Users enter basic information such as their health condition, allergies, food preferences, family composition, desired budget, preparation period, and menu direction. Using a smartphone or computer, they enter the information into a dedicated app or web form and press the send button. The server receives the basic information entered and stores it in a database.

[0782] Input: health condition, allergy information, disliked ingredients, family composition, desired budget, preparation period, menu direction.

[0783] Output: Notification that basic information has been saved to the database.

[0784] Specific action: The user opens the app on their smartphone, enters the required information into the input form, and taps the submit button.

[0785] Step 2:

[0786] The server retrieves price information from a Google Cloud database based on the user's residential area information. It analyzes the price information and generates the optimal menu for the period specified by the user. It performs data analysis using Python and R, and generates the menu using machine learning models in TensorFlow and PyTorch.

[0787] Input: User's basic information, price information for the area where you live.

[0788] Output: The generated menu information.

[0789] Specific operation: The server calls the Google Cloud API to obtain price data and runs a Python script to generate a menu.

[0790] Step 3:

[0791] The server calculates the necessary ingredients and estimated budget based on the generated menu information. The calculation results are converted into HTML format and sent to the user's device, allowing the user to understand the ingredients they need to purchase and their budget.

[0792] Input: Generated menu information.

[0793] Output: A list of ingredients needed and a projected budget.

[0794] Specific operation: The server runs the program to calculate the required ingredient list and budget, and sends the results as an HTML file to the user's terminal.

[0795] Step 4:

[0796] Users input information about ingredients in their refrigerator using images, voice, or text. They take photos of the ingredients with their smartphone camera and enter their desired seasonings in text. The device then sends this information to the server.

[0797] Input: Information about ingredients in the refrigerator (images, audio, text), desired seasoning.

[0798] Output: Ingredient information and seasoning requests sent to the server.

[0799] What it does: The user takes a photo of the contents of their refrigerator with their smartphone and sends the photo and a text message about the seasoning through the app.

[0800] Step 5:

[0801] The server analyzes the received ingredient information and seasoning preferences. It identifies ingredients using image recognition technologies such as OpenCV and YOLOv3, and understands the seasoning preferences using natural language processing. Based on the results, it suggests an appropriate recipe.

[0802] Input: Submitted ingredient information and seasoning preference.

[0803] Output: A suggested recipe.

[0804] Specific operation: The server launches an image analysis engine to identify ingredients, analyzes desired seasonings using a natural language processing engine, and searches for and suggests appropriate recipes from a database.

[0805] Step 6:

[0806] While cooking, the user sends the cooking status to the server using a camera or voice input, and the server analyzes the received cooking status in real time and provides appropriate cooking advice.

[0807] Input: Image and audio data of the cooking process.

[0808] Output: Real-time cooking advice.

[0809] How it works: The user records the cooking process in real time using the camera and microphone on their smartphone and sends the footage to the server, which analyzes it and provides instant cooking advice.

[0810] Step 7:

[0811] The server automatically generates a list of ingredients based on the menu and sends an order request to partner companies via an API.

[0812] Input: A list of ingredients needed.

[0813] Output: An order request to the partner company.

[0814] Specific operation: The server references the MySQL database to obtain the necessary ingredient information, then calls the partner company's API to send an order request.

[0815] Step 8:

[0816] The user checks the order details on the app and presses the confirmation button to complete the order. The server then sends the confirmed order information to the partner company.

[0817] Input: User confirms and confirms order details.

[0818] Output: Confirmed order submission.

[0819] Specific operation: The user opens the order confirmation screen of the app, checks the order details, and taps the confirm button. The server receives this information and sends a notification of order confirmation to the partner company.

[0820] Step 9:

[0821] After the order is confirmed, the server arranges delivery with a courier and notifies the user of the estimated delivery time.

[0822] Input: Confirmed order information.

[0823] Output: Notification of delivery request details and estimated delivery time.

[0824] Specific operation: The server calls the delivery company's API to request delivery and sends the estimated delivery time to the user via push notification.

[0825] (Application example 1)

[0826] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0827] In today's world, it is extremely time-consuming for users to plan appropriate menus and efficiently procure the necessary ingredients while taking into account their own health condition, allergies, and budget. It is also difficult to receive appropriate recipe suggestions and cooking advice in real time. Furthermore, the lack of online links to purchase ingredients persists, making food procurement a complex process. Given this background, there is a demand for a system that allows users to easily lead a healthy diet.

[0828] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0829] In this invention, the server includes means for inputting basic information such as the user's health condition, allergies, budget, etc., means for acquiring price information for the user's area of ​​residence, means for analyzing the input basic information and the acquired price information to generate an optimal menu, means for generating a list of necessary ingredients and a projected budget based on the generated menu, means for providing an online link for purchasing ingredients based on the generated menu, and means for presenting the ingredient list and projected budget to the user. This enables users to easily generate optimal menus based on their own health condition, allergies, and budget, efficiently procure the necessary ingredients, and easily purchase them via the online link.

[0830] - "Basic user information" refers to personal information such as the user's health condition, allergies, disliked ingredients, family composition, desired budget, preparation period, and menu direction.

[0831] "Price information for residential area" refers to price information for food and ingredients in the area where the user lives.

[0832] A "menu" refers to a plan of suitable meals for a specific period of time based on the user's basic information.

[0833] The "ingredient list" is a list of ingredients required based on the generated menu.

[0834] The "projected budget" is the total amount of purchase costs predicted based on the ingredient list.

[0835] "Online Links" are links to online stores that allow users to easily purchase the ingredients they need.

[0836] "Ingredient information" is detailed information about foods and ingredients currently owned by the user.

[0837] "Seasoning preference" refers to a user's preference regarding the type of seasoning or flavor they desire for a particular dish.

[0838] "Real-time advice" refers to instructions and advice on cooking methods and next steps that users receive immediately while cooking.

[0839] "Partners" are companies and service providers that work together to supply ingredients and food products and provide them to users.

[0840] A "delivery request" is an arrangement for a delivery service to deliver ordered ingredients or food to a location specified by the user.

[0841] This invention is a system that allows users to create optimal menus that take into consideration their own health conditions, allergies, budget, etc., and to efficiently procure the necessary ingredients. A specific embodiment of this system will be described in detail below.

[0842] System configuration

[0843] This system works in conjunction with user terminals, a server, and partner companies. The user terminal accepts user input and communicates with the server. The server analyzes the information received from the user and is responsible for creating menus, suggesting recipes, and managing ingredients. Partner companies also provide and deliver the necessary ingredients.

[0844] Entering basic user information

[0845] Users input their health condition, allergies, food preferences, family composition, desired budget, preparation period, and menu direction into a user terminal, which then transmits this information to the server.

[0846] Data analysis and menu generation by server

[0847] The server receives the basic information and obtains price information for the user's area. Based on this information, it analyzes and generates a menu for the specified period. The generated menu is optimized taking into account the user's health condition, family composition, and budget.

[0848] Present the ingredients list and estimated budget

[0849] The server calculates the necessary ingredients and estimated budget based on the generated menu and sends them to the user's device. This allows the user to understand the ingredients they need to purchase and their budget. An online link to purchase ingredients is also provided, making it easy for the user to purchase ingredients.

[0850] Food ordering and delivery

[0851] 1. The server lists the ingredients needed based on the generated menu.

[0852] 2. The listed ingredients are ordered from partner companies.

[0853] 3. Once the order details are confirmed, the server will arrange for delivery and notify the user of the delivery request.

[0854] Hardware and software used

[0855] The hardware used includes the smartphone operated by the user, servers, and network communication devices. The software used includes Python programs, REST APIs, and data analysis algorithms. Specific technologies include using generative AI models to generate menus and suggest recipes.

[0856] Examples of concrete examples and prompts

[0857] Examples:

[0858] When users open the app and enter their basic information, the app displays the optimal meal plan based on their health status and budget. It also provides a list of ingredients and links to online shopping sites. By clicking on the link, users can easily purchase the ingredients they need. Users can also enter information about the ingredients in their refrigerator and receive real-time recipe suggestions.

[0859] Example prompt sentence:

[0860] "Create an app that generates optimal meals based on the user's health, allergies, budget, etc., and provides the necessary ingredients and online purchasing links."

[0861] In this way, the present invention allows users to easily lead a healthy diet and efficiently source ingredients. Furthermore, the system provides real-time cooking advice, making cooking easy even for beginners.

[0862] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0863] Step 1:

[0864] The user uses a terminal to input basic information such as health status, allergies, food preferences, family composition, budget, and menu direction. The input basic information is sent from the terminal to the server.

[0865] Input: User's health condition, allergies, disliked ingredients, family composition, budget, menu direction

[0866] Output: User basic information sent to the server

[0867] Specific operation: The user enters information into the input form on the device and presses the "Submit" button.

[0868] Step 2:

[0869] The server retrieves price information for the user's area. The server uses the Internet or existing databases to retrieve the latest food price information for the area the user enters.

[0870] Input: User's residential area

[0871] Output: Price information for your area

[0872] What happens: The server sends requests to APIs and third-party databases to retrieve price information.

[0873] Step 3:

[0874] The server analyzes basic information and price information to generate the optimal menu for the specified period. Using a generative AI model, the server generates a menu based on food price information, taking into account the user's health condition, budget, and allergies.

[0875] Input: User basic information, price information

[0876] Output: Generated menu data

[0877] How it works: The generative AI model processes the input data and generates the optimal menu.

[0878] Step 4:

[0879] The server calculates the necessary ingredients and estimated budget based on the generated menu and sends it to the user's device. This allows the user to understand the ingredients they need to purchase and their budget. An online purchase link is also provided.

[0880] Input: Generated menu data

[0881] Output: Ingredients list, estimated budget, online purchase link

[0882] What it does: The server analyzes the menu data, calculates the ingredients list and estimated budget, and generates online purchasing links for each ingredient.

[0883] Step 5:

[0884] The user confirms the order details using the terminal and presses the confirm button. Once the order details are confirmed, the server sends an order request to the partner company, and the order is officially confirmed.

[0885] Inputs: Ingredient list, estimated budget, online purchasing links

[0886] Output: Order request to partner

[0887] Specific operation: When the user confirms the order details on the terminal and presses the confirmation button, the server sends the order request to the partner company.

[0888] Step 6:

[0889] After the order is confirmed, the server arranges for delivery and notifies the user of the delivery request. After receiving confirmation from the partner company, the server requests delivery from the delivery company and notifies the user of the estimated delivery time.

[0890] Input: Order confirmation information

[0891] Output: Notification of delivery request details

[0892] Specific operation: The server sends a delivery request to the delivery company and provides notification information to the user.

[0893] Step 7:

[0894] The user inputs the ingredient information they have stored into the server from their device. The ingredient information can be input as an image, voice, or text. The user's seasoning preferences are also input at the same time.

[0895] Input: User's food information, seasoning preference

[0896] Output: Ingredient information and desired seasoning data sent to the server

[0897] Specific operation: The user takes a photo of the ingredients in the refrigerator using the device and enters their desired seasoning.

[0898] Step 8:

[0899] The server analyzes the inputted ingredient information and seasoning preferences and proposes appropriate recipes. The server uses a generative AI model to analyze the inputted data and generate recipes that match the user's preferences.

[0900] Input: Ingredient information, desired seasoning

[0901] Output: Suggested recipe

[0902] How it works: The generative AI model analyzes ingredient information and seasoning preferences to generate an appropriate recipe.

[0903] Step 9:

[0904] While the user is cooking, the cooking status is sent to the server via camera or voice input from the device, and the server analyzes the received cooking status and provides appropriate advice in real time.

[0905] Input: Cooking status data (images, audio)

[0906] Output: Real-time cooking advice

[0907] Specific operation: While cooking, the user takes a photo or voice-over of the situation on the device, and the server analyzes it and advises the next step.

[0908] In this way, at each processing step, the user, server, and terminal work together to input, analyze, and output data, enabling users to easily create healthy menus and efficiently procure the ingredients needed to make them.

[0909] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0910] The present invention combines a system that allows users to easily create personalized menus and efficiently source the necessary ingredients with an emotion engine. This system analyzes the user's emotions and provides recipes and cooking advice accordingly, creating a more personalized experience. Below, we will explain in detail how to implement this system.

[0911] System configuration

[0912] This system operates by combining user devices, a server, collaboration with partner companies, and an emotion engine. The user device accepts user input, communicates with the server, and acquires emotion data through the emotion engine. The server analyzes the information received from the user and is responsible for generating menus, suggesting recipes, and managing ingredients. Partner companies also provide and deliver the necessary ingredients.

[0913] Creating a menu and incorporating emotions

[0914] 1. Enter basic user information

[0915] Users input their health condition, allergies, disliked ingredients, family composition, desired budget, preparation period, and desired menu direction. In addition, the user can also input their current emotional state. This information is sent to the server from the user's device.

[0916] Example: "Health condition: good, Allergies: peanuts, Dislikes: tomatoes, Family size: 4 people, Desired budget: 5,000 yen per week, Menu preference: healthy, Emotional state: tired"

[0917] 2. Data analysis and menu generation by the server

[0918] The server receives the basic information and emotional state sent, and obtains price information for the user's area via an API. Based on this information, it analyzes and generates a menu for the specified period. This also includes menu suggestions to soothe the user's emotions based on their emotional state.

[0919] For example, if you are tired, a simple yet balanced meal will be suggested: "Breakfast: Oatmeal, Lunch: Simple Salad, Dinner: Stir-fried Chicken and Vegetables."

[0920] 3. Present the ingredients list and estimated budget

[0921] The server calculates the necessary ingredients and estimated budget based on the generated menu, and sends them to the user's device, allowing the user to understand the ingredients they need to purchase and their budget.

[0922] Example: Based on the estimated budget calculated by the server, the user is presented with a list of ingredients needed: oatmeal, lettuce, chicken, carrots, along with the total budget.

[0923] Recipe suggestions and cooking assistance

[0924] 1. Enter ingredient information

[0925] Users can input information about the ingredients in their refrigerator using images, voice, or text, as well as their desired seasoning and current emotional state.

[0926] Example: Take a photo of the ingredients in the refrigerator and enter the text "I want it to have a Japanese flavor" and "Emotional state: Calm."

[0927] 2. Analysis of food information and emotional state by the server

[0928] The server analyzes the transmitted ingredient information and emotional state, and suggests dishes that can be made based on the available ingredients and desired seasonings. It also suggests recipes that will help you relax depending on your emotional state.

[0929] Example: Based on the ingredients identified through photo analysis: chicken, cabbage, and carrots, the system suggests "teriyaki chicken."

[0930] 3. Real-time cooking advice

[0931] While cooking, users can use their device to send cooking status information to the server using a camera or voice input. The server then analyzes the received cooking status and the user's emotional state and provides appropriate advice in real time.

[0932] For example, if you say, "The chicken is browned," the server will advise you, "Next, add the sauce and cook over medium heat for 5 minutes," and also send a relaxing message depending on your emotional state.

[0933] Food ordering and delivery

[0934] 1. List and order ingredients

[0935] The server automatically generates a list of ingredients required based on the menu, which then becomes the basis for ordering from partner companies.

[0936] For example, "Oatmeal, lettuce, chicken, and carrots" is listed and an order request is sent to the partner company.

[0937] 2. Confirm and confirm your order

[0938] The user confirms the order details and presses the confirmation button to complete the order. The confirmed order is then officially sent to the partner company via the server.

[0939] Example: When a user checks the order details and presses the confirm button, the order is confirmed and sent to the partner company.

[0940] 3. Delivery request and notification

[0941] After the order is confirmed, the server arranges for delivery and sets the delivery request details. The user is notified of the estimated delivery time, etc.

[0942] Example: The server requests delivery from a courier company and sends a notification to the user saying, "Estimated delivery time: tomorrow at 2:00 PM."

[0943] This system allows users to eat healthy and hassle-free, and to efficiently source ingredients. Furthermore, an emotion engine is incorporated into the recipe suggestions and real-time cooking advice, providing support tailored to the user's emotional state. This increases the enjoyment and satisfaction of cooking, and is expected to be even more practical.

[0944] The processing flow will be explained below.

[0945] Creating a menu and incorporating emotions

[0946] Step 1:

[0947] Using a device, users enter their health condition, allergies, food dislikes, family composition, desired budget, preparation period, desired menu direction, and current emotional state into a dedicated form.

[0948] Specifically, the user enters the following information: "Health condition: good, Allergies: peanuts, Dislikes: tomatoes, Family size: 4 people, Desired budget: 5,000 yen per week, Menu direction: healthy, Emotional state: tired."

[0949] Step 2:

[0950] The device sends the user's input data to the server.

[0951] Specifically, the device uses a REST API to send the user's input data in JSON format to the server, along with their emotional state.

[0952] Step 3:

[0953] The server analyzes the user's basic information and emotional state and obtains price information for the user's area via an API.

[0954] The server identifies the area where the user lives and obtains the latest price information from supermarkets and markets in that area.

[0955] Step 4:

[0956] The server generates a menu for the specified period based on the basic information, emotional state, and price information analyzed by the server.

[0957] Specifically, the menu generation algorithm selects recipes that suit a user's health status, family composition, and budget, and suggests easy-to-prepare dishes for tired users, such as "Breakfast: oatmeal, Lunch: simple salad, Dinner: stir-fried chicken with vegetables."

[0958] Step 5:

[0959] Based on the generated menu, the server calculates the list of ingredients needed and the estimated budget, and sends it to the user's terminal.

[0960] Based on the ingredient list and estimated budget calculated by the server, a list of "necessary ingredients: oatmeal, lettuce, chicken, carrots" and the total budget are created and sent to the terminal.

[0961] Recipe suggestions and cooking assistance

[0962] Step 1:

[0963] The user uses the terminal to input information about the ingredients in the refrigerator (either images, voice, or text), desired seasoning, and their current emotional state.

[0964] As a specific example, a user takes a photo of the ingredients in the refrigerator and enters text such as "I want it to have a Japanese flavor" and "Emotional state: calm."

[0965] Step 2:

[0966] The terminal transmits the inputted information on ingredients, desired seasoning, and emotional state to the server.

[0967] Using the REST API, image data and seasoning requests are sent in JSON or XML format.

[0968] Step 3:

[0969] The server analyzes the received ingredient information, emotional state, and seasoning preferences, and suggests dishes that match the available ingredients and the user's emotions.

[0970] An image analysis algorithm identifies the type of ingredient and suggests the most suitable dish, such as "Teriyaki Chicken," from recipes in the database according to "Emotional State: Calm."

[0971] Step 4:

[0972] The server sends the suggested recipe information to the user's terminal and displays it.

[0973] A list of suggested dishes, cooking instructions, and any additional ingredients needed are sent in JSON format and displayed on the device.

[0974] Step 5:

[0975] While the user is cooking, the cooking status is sent to the server in real time using the device's camera or voice input.

[0976] Specifically, the cooking status is input by voice, such as "the chicken is browned."

[0977] Step 6:

[0978] The server analyzes the cooking status and emotional state received and sends appropriate real-time cooking advice to the user's device.

[0979] It will check the cooking steps, advise you on how to cook the food, and then give you advice such as "Next, add the sauce and cook over medium heat for another 5 minutes," and also send you relaxing messages based on your emotional state.

[0980] Food ordering and delivery

[0981] Step 1:

[0982] The server lists the missing ingredients based on the generated menu.

[0983] It compares the menu with the user's list of ingredients on hand and lists any ingredients that are lacking, such as "oatmeal, lettuce, chicken, and carrots."

[0984] Step 2:

[0985] The server generates an order request to the partner company based on the listed ingredient information.

[0986] Specifically, order information is generated in JSON format and sent to partner companies via API.

[0987] Step 3:

[0988] The server sends an order request to the user's terminal, which the user confirms.

[0989] The user receives an email or app notification confirming their order.

[0990] Step 4:

[0991] The user checks the order details and presses the confirmation button on the terminal.

[0992] When the confirm button is clicked, the confirmation information is sent to the server.

[0993] Step 5:

[0994] The server finally transmits the confirmed order information to the partner company and confirms the order.

[0995] Order confirmation information is sent to partner companies via API.

[0996] Step 6:

[0997] After the order is confirmed, the server sets up a delivery request and arranges an appropriate delivery time.

[0998] Use the delivery company's API to set up delivery and set the expected delivery time for the user.

[0999] Step 7:

[1000] The server notifies the user of the delivery request and provides delivery information.

[1001] Information such as "Estimated delivery time: tomorrow 2pm" is sent to users via email or app notifications.

[1002] Example 2

[1003] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1004] Conventional menu generation systems only suggest menus based on the user's basic information and price information, making it impossible to realize personalized menu suggestions that take the user's emotional state into account. Furthermore, there was no system that properly analyzed the user's ingredient information and cooking status to provide real-time cooking advice, resulting in low user convenience and satisfaction. Furthermore, there was a lack of efficient ways to order the necessary ingredients and process delivery procedures, making it difficult for users to procure ingredients without hassle.

[1005] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for inputting the user's basic information and emotional state; means for acquiring price information for the user's residential area; means for analyzing the input basic information, emotional state, and acquired price information to generate an optimal menu; means for generating a necessary ingredient list and estimated budget based on the generated menu; and means for presenting the ingredient list and estimated budget to the user. This enables personalized menu suggestions that take the user's emotional state into consideration. The server also includes means for inputting the user's ingredient information and emotional state; means for suggesting dishes that can be made by analyzing the input ingredient information, emotional state, and seasoning preferences; and means for analyzing the cooking status and emotional state during cooking to provide cooking advice in real time. This allows the user to receive appropriate cooking advice in real time. The server also includes means for listing necessary ingredients based on the generated menu; means for ordering the listed ingredients from partner companies; means for setting a delivery request after the order is confirmed; and means for notifying the user of the delivery request. This allows the user to efficiently procure the necessary ingredients without any hassle.

[1006] "Basic user information" refers to information regarding the user's health condition, allergies, food preferences, family composition, desired budget, and creation period.

[1007] "Emotional state" is information that represents the user's current mood or mental state.

[1008] "Price information" is information about the prices of common goods and services in the user's area.

[1009] "Menu" means a plan of meals that a User intends to consume within a specified time period.

[1010] The "ingredient list" is a list of ingredients required based on the generated menu.

[1011] The "projected budget" is the estimated cost required to purchase ingredients calculated based on the ingredient list.

[1012] "Ingredient information" refers to information about the type, quantity, and condition of ingredients currently owned by the user.

[1013] "Seasoning preference" refers to the user's preference regarding the taste and cooking method of the food.

[1014] "Cooking status" is information about the progress and process of cooking.

[1015] "Real-time cooking advice" refers to specific cooking instructions and advice provided instantly based on the user's cooking status while cooking.

[1016] "Partner companies" refer to companies that provide and deliver ingredients.

[1017] "Ordering" refers to the act of formally requesting ingredients needed by a user from a partner company.

[1018] A "delivery request" is a request to arrange for the delivery of the ingredients to the user after the order is confirmed.

[1019] "Notification" is a means of informing users of delivery requests and other important information.

[1020] This invention is a system that allows users to easily create menus that suit their needs and efficiently procure the necessary ingredients, and it also combines an emotion engine. This system analyzes the user's emotions and provides recipes and cooking advice accordingly, realizing a more personalized experience.

[1021] System configuration

[1022] This system operates by combining a user device, a server, collaboration with partner companies, and an emotion engine. The user device accepts input of the user's basic information and emotional state, communicates with the server, and transmits the data to the server. The server analyzes the information received from the user and is responsible for generating menus, suggesting recipes, and managing ingredients. Partner companies also provide and deliver the necessary ingredients.

[1023] Entering the user's basic information and emotional state

[1024] Users can use devices such as smartphones or tablets to input their health status, allergies, food preferences, family composition, desired budget, preparation period, and desired menu direction. In addition, users can also input their current emotional state. This information is sent from the device to the server.

[1025] Examples:

[1026] "Health: Good, Allergies: Peanuts, Dislikes: Tomatoes, Family size: 4, Desired budget: 5,000 yen per week, Menu preference: Healthy, Emotional state: Tired"

[1027] Data analysis and menu generation by server

[1028] The server receives the basic information and emotional state sent, and obtains price information for the user's area via an API. Based on this information, it analyzes and generates a menu for the specified period. This also includes menu suggestions to soothe the user's emotions based on their emotional state.

[1029] Examples:

[1030] If you're tired, the app suggests easy-to-prepare, nutritionally balanced meals: "Breakfast: Oatmeal, Lunch: Simple Salad, Dinner: Stir-fried Chicken and Vegetables."

[1031] Present the ingredients list and estimated budget

[1032] The server calculates the necessary ingredients and estimated budget based on the generated menu and sends this to the user's terminal, allowing the user to understand the ingredients they need to purchase and their budget.

[1033] Examples:

[1034] The user is given a list of "Ingredients Needed: Oatmeal, Lettuce, Chicken, Carrots" along with a budget total.

[1035] Enter ingredients and emotional state

[1036] Users can input information about the ingredients in their refrigerator using images, voice, or text, as well as their seasoning preferences and current emotional state.

[1037] Examples:

[1038] Take a photo of the ingredients in your refrigerator and enter the text "I want it to have a Japanese flavor" and "Emotional state: calm."

[1039] Server-based analysis of ingredients and emotional state

[1040] The server analyzes the transmitted ingredient information and emotional state, and suggests dishes that can be made based on the available ingredients and desired seasonings. It also suggests recipes that will help you relax depending on your emotional state.

[1041] Examples:

[1042] Based on the ingredients identified through photo analysis: chicken, cabbage, and carrots, we suggest "chicken teriyaki."

[1043] Real-time cooking advice

[1044] While cooking, the user uses the device to send the cooking status to the server using the camera or voice input, and the server analyzes this information and the user's emotional state to provide appropriate advice in real time.

[1045] Examples:

[1046] If you say, "The chicken is browned," the server will advise you, "Next, add the sauce and cook over medium heat for 5 minutes," and it will also send you a relaxing message based on your emotional state.

[1047] Ordering and delivery of necessary ingredients

[1048] The server lists the ingredients needed based on the menu and sends an order request to the partner company. The user confirms the order on the device and completes the order by pressing the confirmation button. After confirmation, the server arranges for delivery and notifies the user of the delivery request.

[1049] Examples:

[1050] "Oatmeal, lettuce, chicken, carrots" is listed and an order request is sent to the partner company. When the user confirms the order and presses the confirm button, the server requests delivery from the delivery company and notifies the user that "Estimated delivery time: tomorrow 2:00 PM."

[1051] Prompt Sentence Examples

[1052] Based on the following information, it generates a weekly menu and ingredient list, and suggests recipes that take into account your emotional state.

[1053] "Health: Good, Allergies: Peanuts, Disliked Food: Tomatoes, Family Size: 4, Budget: 5,000 yen per week, Menu Direction: Healthy, Emotional State: Tired"

[1054] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1055] Step 1:

[1056] Entering the user's basic information and emotional state

[1057] Users input their health condition, allergies, food preferences, family composition, desired budget, preparation period, and desired menu direction into the device, along with their emotional state. Input is done through a form in the app.

[1058] Input: health condition, allergies, disliked ingredients, family composition, desired budget, preparation period, desired menu direction, emotional state

[1059] Output: Basic information and emotional state data

[1060] Specific actions: The user opens the app on their smartphone, enters the required information into the form, and presses the submit button.

[1061] Step 2:

[1062] Sending basic information and emotional state to the server

[1063] The terminal transmits the input basic information and emotional state to the server.

[1064] Input: Basic information and emotional state data

[1065] Output: Data sent to the server

[1066] Specific operation: When the user presses the send button, the device sends the data to the server via the Internet.

[1067] Step 3:

[1068] Data analysis and menu generation by server

[1069] The server analyzes the received basic information and emotional state, obtains price information for the user's area via an API, and runs a program to generate a menu based on this information.

[1070] Input: Basic information, emotional state, price information

[1071] Data processing / calculation: Match basic information with emotional state and generate a menu based on price information in the area of ​​residence.

[1072] Output: Generated menu

[1073] Specific operation: The server calls the API to obtain price information and then generates a menu using an algorithm.

[1074] Step 4:

[1075] Present the ingredients list and estimated budget

[1076] The server calculates the list of ingredients needed and the estimated budget based on the generated menu, and transmits this to the user terminal.

[1077] Input: Generated menu

[1078] Data processing / calculation: Calculate the ingredients list and budget based on the menu.

[1079] Output: Ingredient list, estimated budget

[1080] What it does: The server sends the calculation results to the device, and the app displays them.

[1081] Step 5:

[1082] Enter ingredients and emotional state

[1083] The user inputs information about the ingredients in the refrigerator in the form of images, voice, or text, as well as their desired seasoning and current emotional state.

[1084] Input: Ingredient information, seasoning preference, emotional state

[1085] Output: Data sent to the server

[1086] Specific operation: The user takes a photo of the ingredients in the refrigerator and sends it by text inputting their seasoning preference and emotional state.

[1087] Step 6:

[1088] Server-based analysis of ingredients and emotional state

[1089] The server analyzes the transmitted ingredient information and emotional state, and suggests dishes that can be made based on the available ingredients and desired seasonings.

[1090] Input: Ingredient information, emotional state, desired seasoning

[1091] Data processing / computation: Identifying ingredients using image recognition, using sentiment analysis algorithms

[1092] Output: Suggested dishes

[1093] Specific operation: The server analyzes the image, searches for recipes in the database, and suggests the most suitable dish.

[1094] Step 7:

[1095] Real-time cooking advice

[1096] While cooking, the user uses the device to send cooking information to the server using a camera or voice input, and the server analyzes the information and provides appropriate advice in real time.

[1097] Input: Cooking status, emotional state

[1098] Data processing / calculation: Analyzing cooking status and emotional state

[1099] Output: Real-time cooking advice

[1100] Specific behavior: The server analyzes the voice input and suggests the next cooking steps.

[1101] Step 8:

[1102] List and order the ingredients you need

[1103] The server lists the ingredients needed based on the menu and sends an order request to partner companies.

[1104] Input: Menu

[1105] Data processing / calculation: Creating an ingredient list

[1106] Output: Order request

[1107] What happens: The server creates a list and sends it to partner companies through the ordering system.

[1108] Step 9:

[1109] Confirm and confirm your order

[1110] The user checks the order details on the terminal and presses the confirmation button to complete the order. The server then officially sends the confirmed order to the partner company.

[1111] Input: Ingredients list

[1112] Output: Order confirmation notification

[1113] Specific operation: The user checks the order details on the confirmation screen and presses the confirm button.

[1114] Step 10:

[1115] Delivery requests and notifications

[1116] After the order is confirmed, the server arranges for delivery and sets the delivery request details. The user is notified of the estimated delivery time, etc.

[1117] Input: Order Information

[1118] Data processing / calculation: Delivery schedule setting

[1119] Output: Delivery notification

[1120] Specific operation: The server requests the delivery company and sends a notification to the user.

[1121] (Application example 2)

[1122] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1123] In recent years, it has become increasingly important for users to easily create menus that suit their own health and emotional state and to efficiently source ingredients. However, conventional systems do not provide menu suggestions or cooking advice that take the user's emotions into account, making it difficult to provide a personalized experience. In addition, sourcing ingredients and requesting delivery is time-consuming for users, which is also an issue that lacks efficiency.

[1124] The specification processing by the specification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for inputting basic information about the user, means for acquiring price information for the residential area, means for analyzing the input basic information and the acquired price information to generate an optimal menu, means for generating a list of necessary ingredients and a projected budget based on the generated menu, means for analyzing the user's emotions, means for suggesting a personalized menu based on the analyzed emotions, and means for presenting the ingredient list and the projected budget to the user. This enables personalized menu suggestions and efficient ingredient procurement according to the user's emotional state.

[1125] "Means for inputting basic user information" refers to the means by which users input their health condition, allergies, food preferences, family composition, desired budget, preparation period, desired menu direction, etc. into the system.

[1126] "Means for obtaining price information in the residential area" refers to means for obtaining price information in the area where the user lives from the Internet or an external database.

[1127] "Means for analyzing the input basic information and the acquired price information to generate the optimal menu" refers to means for automatically generating the optimal menu for the user based on the basic information input by the user and the acquired price information.

[1128] The "means for generating a list of necessary ingredients and a projected budget based on the generated menu" refers to a means for calculating and generating a list of necessary ingredients and a projected budget based on the generated menu.

[1129] "Means for analyzing user emotions" refers to means for analyzing the user's facial image and voice data to detect their emotional state at that time.

[1130] The "means for making personalized menu suggestions based on analyzed emotions" is a means for suggesting a menu that is suitable for each individual user according to the user's emotional state.

[1131] The "means for presenting the ingredient list and estimated budget to the user" refers to a means for displaying the generated ingredient list and estimated budget on the user's terminal.

[1132] "Means for inputting information about ingredients owned by the user" refers to means for inputting information about ingredients owned by the user using images, voice, text, etc.

[1133] "Means for suggesting dishes that can be made by analyzing the inputted ingredient information and desired seasoning" refers to means for suggesting dishes that can be made based on the ingredient information and desired seasoning that the user inputs.

[1134] The "means for analyzing the cooking situation during cooking and providing cooking advice in real time" is a means for analyzing the cooking situation using images and sounds and providing cooking advice in real time.

[1135] The "means for generating cooking advice based on the emotional state of the user" is a means for analyzing the emotional state of the user and generating cooking advice that matches that emotion.

[1136] The "means for providing feedback according to the cooking situation and emotional state" is a means for providing appropriate feedback to the user based on the cooking situation and the user's emotional state.

[1137] The "means for listing necessary ingredients based on the generated menu" is a means for listing necessary ingredients based on the generated menu.

[1138] The "means for ordering the listed ingredients from partner companies" is a means for automatically ordering the listed ingredients from external partner companies.

[1139] "Means for requesting delivery after confirming an order" refers to a means for a user to arrange delivery after confirming and confirming the order details.

[1140] "Means for notifying the user of the details of the delivery request" refers to means for notifying the user of information such as the order details and the estimated delivery time.

[1141] The "means for proposing an optimal delivery time based on the emotional state of the user" is a means for proposing an optimal delivery time by taking into account the emotional state of the user.

[1142] To implement this invention, a system is required that can efficiently acquire a user's basic information and emotional state, and then use that information to create menus, procure ingredients, and provide cooking advice. This system operates primarily in cooperation with the user's terminal, a server, and partner companies. Specific embodiments are described below.

[1143] Hardware and Software Configuration

[1144] Hardware:

[1145] User device: A smartphone or tablet with a camera and internet connection.

[1146] Server: Responsible for data processing and analysis. A server with internet access.

[1147] software:

[1148] OpenCV: A library used for face recognition and emotion detection.

[1149] Emotion Detection API: An external API that performs emotion analysis.

[1150] Flask: A Python framework for building server-side recipe generation engines.

[1151] Delivery Service API: An API for handling food orders with partner companies.

[1152] System Functions and Processing

[1153] 1. Sentiment analysis and menu suggestions

[1154] The server first analyzes the user's emotional state. It captures a facial image using the user's device's camera and sends the image data to the Emotion Detection API. The API analyzes the image and returns the user's emotional state. Based on this emotional state, it suggests an appropriate menu from a database stored on the Flask server.

[1155] Example: "Since we detected the emotion 'tired' from your facial image, we will suggest a simple yet nutritionally balanced meal: 'Stir-fried chicken with vegetables.'"

[1156] 2. List and order ingredients

[1157] The server automatically generates a list of ingredients based on the proposed menu and calculates the estimated budget. This is then displayed on the user's device, and once the user confirms and confirms, the order information is sent to the partner company via the Delivery Service API.

[1158] For example: "I ordered lettuce, chicken, and carrots. The total cost is 1000 yen."

[1159] 3. Analysis of cooking status and provision of advice

[1160] The server receives and analyzes camera images and voice input from the user while cooking, providing real-time advice at the appropriate time based on the cooking situation. By taking the user's emotional state into consideration, the system provides feedback aimed at maximizing relaxation.

[1161] For example: "I've determined that the chicken is browned. Now add the sauce and cook over medium heat for 5 minutes."

[1162] Example prompt:

[1163] Below are some example prompts that users can use to suggest meals based on their emotional state:

[1164] Perform "face photo capture," "detect emotional state," and suggest "emotion-based menus."

[1165] summary

[1166] This invention allows users to receive menu suggestions tailored to their emotional state, efficiently source the ingredients they need, and receive real-time advice based on the cooking situation, providing a more personalized cooking experience.

[1167] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1168] Program processing steps

[1169] Step 1: Enter basic information

[1170] The user inputs their health condition, allergies, food preferences, family composition, desired budget, preparation period, and desired menu direction. This information is sent from the user terminal.

[1171] Input: Personal information entered by the user

[1172] Output: Basic information sent from the user device to the server

[1173] Step 2: Sentiment Analysis

[1174] Users take a picture of their face with their smartphone camera and send the image data to the Emotion Detection API, which analyzes the image and returns the user's emotional state.

[1175] Input: User's face image

[1176] Output: Sentiment analysis result (e.g., tired, relaxed)

[1177] Step 3: Obtaining price information

[1178] The server obtains price information from the Internet or external databases based on the user's residential area information.

[1179] Input: User's residential area information

[1180] Output: Price information data

[1181] Step 4: Create a menu

[1182] The server generates the optimal menu based on the input basic information, emotional state, and acquired price information. It uses the recipe database on the Flask server to suggest personalized menus.

[1183] Input: Basic information, emotional state, price information

[1184] Output: Optimal menu data

[1185] Step 5: Generate a list of ingredients needed and a projected budget

[1186] The server calculates and generates a list of necessary ingredients and a projected budget based on the generated menu.

[1187] Input: Optimal menu data

[1188] Output: Ingredient list, estimated budget

[1189] Step 6: Present to the user

[1190] The server sends the generated ingredient list and estimated budget to the user's terminal, where the user can check it on the screen.

[1191] Input: Ingredient list, estimated budget

[1192] Output: Information displayed on the user's terminal

[1193] Step 7: Order ingredients

[1194] The user confirms the ingredients list and budget and confirms the order. The server then sends this order information to the partner company via the Delivery Service API.

[1195] Input: Ingredient list, estimated budget, user order confirmation

[1196] Output: Order data for partner companies

[1197] Step 8: Arrange delivery and notify

[1198] Once the order is confirmed, the server sends a delivery request to the delivery company and notifies the user of the estimated time. The server also suggests the optimal delivery time, taking into account the user's emotional state.

[1199] Input: Order data, user emotional state

[1200] Output: Request to courier, notification to user

[1201] Step 9: Cooking Tips

[1202] While cooking, users take pictures of their cooking progress with their smartphone camera and report the progress by voice. The server analyzes the information and sends cooking advice in real time. The server also analyzes the user's emotional state and sends corresponding relaxing messages.

[1203] Input: Images and audio of cooking status

[1204] Output: Real-time cooking advice, emotion-based messages

[1205] Through this series of processes, users can receive personalized menu suggestions, efficiently source the necessary ingredients, and receive accurate advice while cooking.

[1206] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1207] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1208] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1209] [Third embodiment]

[1210] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1211] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1212] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1213] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1214] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1215] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1216] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1217] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1218] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1219] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1220] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1221] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1222] The present invention provides a system that allows users to easily create menus that suit their needs and efficiently procure the necessary ingredients. Hereinafter, an embodiment of the system will be described in detail.

[1223] System configuration

[1224] This system operates in cooperation with user terminals, a server, and partner companies. The user terminal accepts user input and communicates with the server. The server analyzes the information received from the user and is responsible for creating menus, suggesting recipes, and managing ingredients. Partner companies also provide and deliver the necessary ingredients.

[1225] Creating a menu

[1226] 1. Enter basic user information

[1227] Users input their health condition, allergies, food preferences, family composition, desired budget, preparation period, and menu direction. This information is sent to the server from the user's terminal.

[1228] Example: "Health condition: good, Allergies: peanuts, Dislikes: tomatoes, Family size: 4, Desired budget: 5,000 yen per week, Menu preference: healthy"

[1229] 2. Data analysis and menu generation by the server

[1230] The server receives the basic information and obtains price information for the user's area. Based on this information, it analyzes and generates a menu for the specified period. The generated menu is optimized taking into account the user's health condition, family composition, and budget.

[1231] Example: The server compares the price information it has obtained and generates a menu such as "Breakfast: oatmeal, Lunch: salad, Dinner: stir-fried chicken and vegetables."

[1232] 3. Present the ingredients list and estimated budget

[1233] The server calculates the necessary ingredients and estimated budget based on the generated menu, and sends them to the user's device, allowing the user to understand the ingredients they need to purchase and their budget.

[1234] Example: Based on the estimated budget calculated by the server, the user is presented with a list of ingredients needed: oatmeal, lettuce, chicken, carrots, along with the total budget.

[1235] Recipe suggestions and cooking assistance

[1236] 1. Enter ingredient information

[1237] Users input information about the ingredients in their refrigerator. This information can be entered using images, voice, or text. They can also input their desired seasonings at the same time.

[1238] Example: Take a photo of the ingredients in your refrigerator and enter the text "I want it to have a Japanese flavor."

[1239] 2. Server analyzes ingredient information and suggests recipes

[1240] The server analyzes the submitted ingredient information and seasoning preferences to identify available ingredients, and then suggests recipes that match the user's desired seasonings based on these ingredients.

[1241] Example: Based on the ingredients identified through photo analysis: chicken, cabbage, and carrots, the system suggests "teriyaki chicken."

[1242] 3. Real-time cooking advice

[1243] While cooking, users can use their device to send cooking information to the server using a camera or voice input, and the server analyzes the received information and provides appropriate advice in real time.

[1244] For example, say, "The chicken is browned," and the server will advise, "Next, add the sauce and cook over medium heat for 5 minutes."

[1245] Food ordering and delivery

[1246] 1. List and order ingredients

[1247] The server automatically generates a list of ingredients required based on the menu, which then becomes the basis for ordering from partner companies.

[1248] For example, "Oatmeal, lettuce, chicken, and carrots" is listed and an order request is sent to the partner company.

[1249] 2. Confirm and confirm your order

[1250] The user confirms the order details and presses the confirmation button to complete the order. The confirmed order is then officially sent to the partner company via the server.

[1251] Example: When a user checks the order details and presses the confirm button, the order is confirmed and sent to the partner company.

[1252] 3. Delivery request and notification

[1253] After the order is confirmed, the server arranges for delivery and sets the delivery request details. The user is notified of the estimated delivery time, etc.

[1254] Example: The server requests delivery from a courier company and sends a notification to the user saying, "Estimated delivery time: tomorrow at 2:00 PM."

[1255] This system allows users to lead a healthy diet without hassle and to efficiently source ingredients. In addition, it provides recipe suggestions and real-time cooking advice, so even beginners can cook with confidence.

[1256] The processing flow will be explained below.

[1257] Creating a menu

[1258] Step 1:

[1259] Users use a device to enter information about their health condition, allergies, food preferences, family composition, desired budget, preparation period, and desired menu direction into a dedicated form.

[1260] Specifically, the user enters the following information: "health condition: good, allergies: peanuts, dislikes: tomatoes, family size: 4 people, desired budget: 5,000 yen per week, menu preference: healthy."

[1261] Step 2:

[1262] The device sends the user's input data to the server.

[1263] Specifically, the terminal uses a REST API to send the user's input data to the server in JSON format.

[1264] Step 3:

[1265] The server analyzes the basic information received from the user and obtains price information for the user's area of ​​residence via an API.

[1266] The server identifies the area where the user lives and obtains the latest price information from supermarkets and markets in that area.

[1267] Step 4:

[1268] The server generates a menu for the specified period based on the price information obtained and the user's basic information.

[1269] The menu generation algorithm creates a list of recipes that suit your health status, family composition, and budget, and selects the optimal menu.

[1270] Step 5:

[1271] The server calculates the required ingredients list and estimated budget based on the generated menu and sends this information to the user's terminal.

[1272] Based on the estimated budget calculated by the server, a list of "necessary ingredients: oatmeal, lettuce, chicken, carrots" and the total budget are created and sent to the terminal.

[1273] Recipe suggestions and cooking assistance

[1274] Step 1:

[1275] The user uses the terminal to input information about the ingredients in the refrigerator (either images, voice, or text) and desired seasonings.

[1276] As a specific example, a user takes a photo of the ingredients in the refrigerator and enters the text, "I want it to have a Japanese flavor."

[1277] Step 2:

[1278] The terminal transmits the inputted ingredient information and desired seasoning to the server.

[1279] Using the REST API, image data and seasoning requests are sent to the server in JSON or XML format.

[1280] Step 3:

[1281] The server analyzes the received ingredient information and suggests dishes that can be made based on the available ingredients and desired seasonings.

[1282] Image analysis algorithms identify the type of ingredient and select the most suitable dish from a database of recipes.

[1283] Step 4:

[1284] The server sends the suggested recipe information to the user's terminal and displays it.

[1285] A list of suggested dishes, cooking instructions, and any additional ingredients needed are sent in JSON format, which is then displayed on the device.

[1286] Step 5:

[1287] While the user is cooking, the cooking status is sent to the server in real time using the device's camera or voice input.

[1288] Specifically, the cooking status is input by voice, such as "The chicken is browned."

[1289] Step 6:

[1290] The server analyzes the cooking status received and sends appropriate real-time cooking advice to the user's device.

[1291] Check cooking instructions and send advice such as, "Next, add the sauce and cook over medium heat for another 5 minutes."

[1292] Food ordering and delivery

[1293] Step 1:

[1294] The server lists the missing ingredients based on the generated menu.

[1295] It compares the menu with the user's list of ingredients on hand and lists any ingredients that are lacking, such as "oatmeal, lettuce, chicken, and carrots."

[1296] Step 2:

[1297] The server generates an order request to the partner company based on the listed ingredient information.

[1298] Specifically, order information is generated in JSON format and sent to partner companies via API.

[1299] Step 3:

[1300] The server sends an order request to the user's terminal, which the user confirms.

[1301] The user receives an email or app notification confirming their order.

[1302] Step 4:

[1303] The user checks the order details and presses the confirmation button on the terminal.

[1304] When the confirm button is clicked, the confirmation information is sent to the server.

[1305] Step 5:

[1306] The server finally transmits the confirmed order information to the partner company and confirms the order.

[1307] Order confirmation information is sent to partner companies via API.

[1308] Step 6:

[1309] After the order is confirmed, the server sets up a delivery request and arranges an appropriate delivery time.

[1310] Use the delivery company's API to set up delivery and set the expected delivery time for the user.

[1311] Step 7:

[1312] The server notifies the user of the delivery request and provides delivery information.

[1313] Information such as "Estimated delivery time: tomorrow 2pm" is sent to users via email or app notifications.

[1314] Example 1

[1315] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1316] In today's busy lifestyles, it is extremely difficult for users to plan healthy, balanced meals and efficiently source the necessary ingredients. Creating meals that take into account specific health conditions or allergies, and receiving real-time cooking advice, are even more challenging. There is a need for a system that can solve these challenges and enable users to effortlessly create appropriate meals and efficiently obtain the necessary ingredients.

[1317] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1318] In this invention, the server includes means for inputting basic information about the user, means for acquiring price information for the user's area of ​​residence, means for analyzing the input basic information and the acquired price information to generate an optimal menu, means for generating a list of necessary ingredients and a projected budget based on the generated menu, means for presenting the ingredient list and the projected budget to the user, means for analyzing the input ingredient information, photo data, and audio data to identify available ingredients, means for suggesting appropriate recipes based on the user's desired seasonings, and means for analyzing the cooking status during cooking and providing cooking advice in real time. This allows users to effortlessly plan healthy menus, procure ingredients, and efficiently manage the cooking process.

[1319] "User" refers to an individual or family member who uses the system to create menus, source ingredients, and receive cooking advice.

[1320] "Basic information" refers to information necessary for creating a menu and selecting ingredients, such as the user's health condition, allergies, food dislikes, family composition, desired budget, and menu direction.

[1321] "Price information for residential area" refers to price information for specific items in the area where the user lives, and is used for planning meals and calculating budgets.

[1322] The "ingredient list" refers to a list of specific ingredients that should be purchased based on the generated menu.

[1323] "Projected Budget" refers to an estimate of the budget required to purchase the necessary ingredients within a specified period of time.

[1324] "Photo data" refers to image information of ingredients and cooking processes that users take and upload to the system.

[1325] "Voice data" refers to voice instructions, questions, and cooking status reports provided by the user to the system.

[1326] A "recipe" refers to a method of making a dish based on specific ingredients and seasonings.

[1327] "Cooking status" refers to information about the progress and current state of cooking, and is reported to the system in real time.

[1328] "Real-time advice" refers to advice or instructions provided to the user immediately during the cooking process.

[1329] The present invention provides a system that allows users to easily create menus that suit their needs and efficiently procure the necessary ingredients. This system operates in cooperation with a user terminal, a server, and partner companies. Detailed embodiments of the system are described below.

[1330] System configuration

[1331] User Device

[1332] The user terminal is a device such as a smartphone or PC that accepts user input and communicates with the server. The user uses the terminal to input basic information about themselves, ingredients, cooking status, etc.

[1333] server

[1334] The server analyzes the information received from the user and is responsible for generating menus, suggesting recipes, and managing ingredients. It uses a cloud computing platform (such as Google Cloud) to obtain price information for the user's area. It also performs data analysis using Python and R, and generates menus and recipes using machine learning models such as TensorFlow and PyTorch.

[1335] Overview of program processing

[1336] Enter basic user information

[1337] Users input information such as their health condition, allergies, food preferences, family composition, desired budget, preparation period, menu direction, etc. via their smartphone or PC, and this information is sent from the device to the server.

[1338] Example: "Health condition: good, Allergies: peanuts, Dislikes: tomatoes, Family size: 4, Desired budget: 5,000 yen per week, Menu preference: healthy"

[1339] Data analysis and menu generation by server

[1340] The server receives the basic information sent and retrieves price information for the area from a Google Cloud database. Based on this information, it performs data analysis and generates a menu for the specified period.

[1341] Example: The server generates a menu such as "Breakfast: oatmeal, Lunch: salad, Dinner: stir-fried chicken and vegetables" based on price information and basic user information.

[1342] Present the ingredients list and estimated budget

[1343] The server calculates the necessary ingredients and estimated budget based on the generated menu, and sends this to the user's device, allowing the user to understand the ingredients they need to purchase and their budget.

[1344] Example: Provide the user with a list of ingredients needed: oatmeal, lettuce, chicken, carrots, along with a budget total.

[1345] Entering ingredient information and suggesting recipes

[1346] Entering ingredient information

[1347] Users can input information about the ingredients in their refrigerator using images, voice, or text, and can also input their desired seasonings for cooking.

[1348] Example: Take a photo of the ingredients in your refrigerator and enter the text "I want to make it Japanese flavor."

[1349] Server analyzes ingredients and suggests recipes

[1350] The server analyzes the submitted ingredient information and seasoning preferences, identifies available ingredients, and then suggests recipes using those ingredients. Image analysis uses technologies such as OpenCV and YOLOv3.

[1351] Example: Suggest "Teriyaki chicken" based on "Ingredients identified through photo analysis: chicken, cabbage, carrots."

[1352] Real-time cooking advice

[1353] While cooking, users can use their device to send cooking status information to the server using a camera or voice input. The server then analyzes the received information and provides appropriate advice in real time.

[1354] For example, say, "The chicken is browned," and the server will advise, "Next, add the sauce and cook over medium heat for 5 minutes."

[1355] Food ordering and delivery

[1356] Listing and ordering ingredients

[1357] The server automatically generates a list of ingredients required based on the menu and sends an order request to the partner company.

[1358] For example, "Oatmeal, lettuce, chicken, and carrots" is listed and an order request is sent to the partner company.

[1359] Confirm and confirm your order

[1360] The user checks the order details in the app and presses the confirmation button to complete the order. The confirmed order is then sent to the partner company via the server.

[1361] Example: When the user confirms the order details and presses the confirm button, the order is confirmed with the partner company.

[1362] Delivery requests and notifications

[1363] After the order is confirmed, the server arranges delivery with a delivery company and notifies the user of the estimated delivery time.

[1364] Example: The server requests delivery from a courier company and the user is notified that "Estimated delivery time: tomorrow 2pm."

[1365] Prompt Sentence Examples

[1366] Please enter your health condition, allergy information, family composition, desired budget, creation period, and menu direction.

[1367] "Please enter a photo of the ingredients in your refrigerator and your desired seasoning."

[1368] Please check your order details and press the confirm button.

[1369] In this way, the system helps users effortlessly lead a healthy and balanced diet. Users can easily create menus, efficiently source ingredients, and receive recipe suggestions and real-time cooking advice.

[1370] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1371] Step 1:

[1372] Users enter basic information such as their health condition, allergies, food preferences, family composition, desired budget, preparation period, and menu direction. Using a smartphone or computer, they enter the information into a dedicated app or web form and press the send button. The server receives the basic information entered and stores it in a database.

[1373] Input: health condition, allergy information, disliked ingredients, family composition, desired budget, preparation period, menu direction.

[1374] Output: Notification that basic information has been saved to the database.

[1375] Specific action: The user opens the app on their smartphone, enters the required information into the input form, and taps the submit button.

[1376] Step 2:

[1377] The server retrieves price information from a Google Cloud database based on the user's residential area information. It analyzes the price information and generates the optimal menu for the period specified by the user. It performs data analysis using Python and R, and generates the menu using machine learning models in TensorFlow and PyTorch.

[1378] Input: User's basic information, price information for the area where you live.

[1379] Output: The generated menu information.

[1380] Specific operation: The server calls the Google Cloud API to obtain price data and runs a Python script to generate a menu.

[1381] Step 3:

[1382] The server calculates the necessary ingredients and estimated budget based on the generated menu information. The calculation results are converted into HTML format and sent to the user's device, allowing the user to understand the ingredients they need to purchase and their budget.

[1383] Input: Generated menu information.

[1384] Output: A list of ingredients needed and a projected budget.

[1385] Specific operation: The server runs the program to calculate the required ingredient list and budget, and sends the results as an HTML file to the user's terminal.

[1386] Step 4:

[1387] Users input information about ingredients in their refrigerator using images, voice, or text. They take photos of the ingredients with their smartphone camera and enter their desired seasonings in text. The device then sends this information to the server.

[1388] Input: Information about ingredients in the refrigerator (images, audio, text), desired seasoning.

[1389] Output: Ingredient information and seasoning requests sent to the server.

[1390] What it does: The user takes a photo of the contents of their refrigerator with their smartphone and sends the photo and a text message about the seasoning through the app.

[1391] Step 5:

[1392] The server analyzes the received ingredient information and seasoning preferences. It identifies ingredients using image recognition technologies such as OpenCV and YOLOv3, and understands the seasoning preferences using natural language processing. Based on the results, it suggests an appropriate recipe.

[1393] Input: Submitted ingredient information and seasoning preference.

[1394] Output: A suggested recipe.

[1395] Specific operation: The server launches an image analysis engine to identify ingredients, analyzes desired seasonings using a natural language processing engine, and searches for and suggests appropriate recipes from a database.

[1396] Step 6:

[1397] While cooking, the user sends the cooking status to the server using a camera or voice input, and the server analyzes the received cooking status in real time and provides appropriate cooking advice.

[1398] Input: Image and audio data of the cooking process.

[1399] Output: Real-time cooking advice.

[1400] How it works: The user records the cooking process in real time using the camera and microphone on their smartphone and sends the footage to the server, which analyzes it and provides instant cooking advice.

[1401] Step 7:

[1402] The server automatically generates a list of ingredients based on the menu and sends an order request to partner companies via an API.

[1403] Input: A list of ingredients needed.

[1404] Output: An order request to the partner company.

[1405] Specific operation: The server references the MySQL database to obtain the necessary ingredient information, then calls the partner company's API to send an order request.

[1406] Step 8:

[1407] The user checks the order details on the app and presses the confirmation button to complete the order. The server then sends the confirmed order information to the partner company.

[1408] Input: User confirms and confirms order details.

[1409] Output: Confirmed order submission.

[1410] Specific operation: The user opens the order confirmation screen of the app, checks the order details, and taps the confirm button. The server receives this information and sends a notification of order confirmation to the partner company.

[1411] Step 9:

[1412] After the order is confirmed, the server arranges delivery with a courier and notifies the user of the estimated delivery time.

[1413] Input: Confirmed order information.

[1414] Output: Notification of delivery request details and estimated delivery time.

[1415] Specific operation: The server calls the delivery company's API to request delivery and sends the estimated delivery time to the user via push notification.

[1416] (Application example 1)

[1417] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1418] In today's world, it is extremely time-consuming for users to plan appropriate menus and efficiently procure the necessary ingredients while taking into account their own health condition, allergies, and budget. It is also difficult to receive appropriate recipe suggestions and cooking advice in real time. Furthermore, the lack of online links to purchase ingredients persists, making food procurement a complex process. Given this background, there is a demand for a system that allows users to easily lead a healthy diet.

[1419] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1420] In this invention, the server includes means for inputting basic information such as the user's health condition, allergies, budget, etc., means for acquiring price information for the user's area of ​​residence, means for analyzing the input basic information and the acquired price information to generate an optimal menu, means for generating a list of necessary ingredients and a projected budget based on the generated menu, means for providing an online link for purchasing ingredients based on the generated menu, and means for presenting the ingredient list and projected budget to the user. This enables users to easily generate optimal menus based on their own health condition, allergies, and budget, efficiently procure the necessary ingredients, and easily purchase them via the online link.

[1421] - "Basic user information" refers to personal information such as the user's health condition, allergies, disliked ingredients, family composition, desired budget, preparation period, and menu direction.

[1422] "Price information for residential area" refers to price information for food and ingredients in the area where the user lives.

[1423] A "menu" refers to a plan of suitable meals for a specific period of time based on the user's basic information.

[1424] The "ingredient list" is a list of ingredients required based on the generated menu.

[1425] The "projected budget" is the total amount of purchase costs predicted based on the ingredient list.

[1426] "Online Links" are links to online stores that allow users to easily purchase the ingredients they need.

[1427] "Ingredient information" is detailed information about foods and ingredients currently owned by the user.

[1428] "Seasoning preference" refers to a user's preference regarding the type of seasoning or flavor they desire for a particular dish.

[1429] "Real-time advice" refers to instructions and advice on cooking methods and next steps that users receive immediately while cooking.

[1430] "Partners" are companies and service providers that work together to supply ingredients and food products and provide them to users.

[1431] A "delivery request" is an arrangement for a delivery service to deliver ordered ingredients or food to a location specified by the user.

[1432] This invention is a system that allows users to create optimal menus that take into consideration their own health conditions, allergies, budget, etc., and to efficiently procure the necessary ingredients. A specific embodiment of this system will be described in detail below.

[1433] System configuration

[1434] This system works in conjunction with user terminals, a server, and partner companies. The user terminal accepts user input and communicates with the server. The server analyzes the information received from the user and is responsible for creating menus, suggesting recipes, and managing ingredients. Partner companies also provide and deliver the necessary ingredients.

[1435] Entering basic user information

[1436] Users input their health condition, allergies, food preferences, family composition, desired budget, preparation period, and menu direction into a user terminal, which then transmits this information to the server.

[1437] Data analysis and menu generation by server

[1438] The server receives the basic information and obtains price information for the user's area. Based on this information, it analyzes and generates a menu for the specified period. The generated menu is optimized taking into account the user's health condition, family composition, and budget.

[1439] Present the ingredients list and estimated budget

[1440] The server calculates the necessary ingredients and estimated budget based on the generated menu and sends them to the user's device. This allows the user to understand the ingredients they need to purchase and their budget. An online link to purchase ingredients is also provided, making it easy for the user to purchase ingredients.

[1441] Food ordering and delivery

[1442] 1. The server lists the ingredients needed based on the generated menu.

[1443] 2. The listed ingredients are ordered from partner companies.

[1444] 3. Once the order details are confirmed, the server will arrange for delivery and notify the user of the delivery request.

[1445] Hardware and software used

[1446] The hardware used includes the smartphone operated by the user, servers, and network communication devices. The software used includes Python programs, REST APIs, and data analysis algorithms. Specific technologies include using generative AI models to generate menus and suggest recipes.

[1447] Examples of concrete examples and prompts

[1448] Examples:

[1449] When users open the app and enter their basic information, the app displays the optimal meal plan based on their health status and budget. It also provides a list of ingredients and links to online shopping sites. By clicking on the link, users can easily purchase the ingredients they need. Users can also enter information about the ingredients in their refrigerator and receive real-time recipe suggestions.

[1450] Example prompt sentence:

[1451] "Create an app that generates optimal meals based on the user's health, allergies, budget, etc., and provides the necessary ingredients and online purchasing links."

[1452] In this way, the present invention allows users to easily lead a healthy diet and efficiently source ingredients. Furthermore, the system provides real-time cooking advice, making cooking easy even for beginners.

[1453] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1454] Step 1:

[1455] The user uses a terminal to input basic information such as health status, allergies, food preferences, family composition, budget, and menu direction. The input basic information is sent from the terminal to the server.

[1456] Input: User's health condition, allergies, disliked ingredients, family composition, budget, menu direction

[1457] Output: User basic information sent to the server

[1458] Specific operation: The user enters information into the input form on the device and presses the "Submit" button.

[1459] Step 2:

[1460] The server retrieves price information for the user's area. The server uses the Internet or existing databases to retrieve the latest food price information for the area the user enters.

[1461] Input: User's residential area

[1462] Output: Price information for your area

[1463] What happens: The server sends requests to APIs and third-party databases to retrieve price information.

[1464] Step 3:

[1465] The server analyzes basic information and price information to generate the optimal menu for the specified period. Using a generative AI model, the server generates a menu based on food price information, taking into account the user's health condition, budget, and allergies.

[1466] Input: User basic information, price information

[1467] Output: Generated menu data

[1468] How it works: The generative AI model processes the input data and generates the optimal menu.

[1469] Step 4:

[1470] The server calculates the necessary ingredients and estimated budget based on the generated menu and sends it to the user's device. This allows the user to understand the ingredients they need to purchase and their budget. An online purchase link is also provided.

[1471] Input: Generated menu data

[1472] Output: Ingredients list, estimated budget, online purchase link

[1473] What it does: The server analyzes the menu data, calculates the ingredients list and estimated budget, and generates online purchasing links for each ingredient.

[1474] Step 5:

[1475] The user confirms the order details using the terminal and presses the confirm button. Once the order details are confirmed, the server sends an order request to the partner company, and the order is officially confirmed.

[1476] Inputs: Ingredient list, estimated budget, online purchasing links

[1477] Output: Order request to partner

[1478] Specific operation: When the user confirms the order details on the terminal and presses the confirmation button, the server sends the order request to the partner company.

[1479] Step 6:

[1480] After the order is confirmed, the server arranges for delivery and notifies the user of the delivery request. After receiving confirmation from the partner company, the server requests delivery from the delivery company and notifies the user of the estimated delivery time.

[1481] Input: Order confirmation information

[1482] Output: Notification of delivery request details

[1483] Specific operation: The server sends a delivery request to the delivery company and provides notification information to the user.

[1484] Step 7:

[1485] The user inputs the ingredient information they have stored into the server from their device. The ingredient information can be input as an image, voice, or text. The user's seasoning preferences are also input at the same time.

[1486] Input: User's food information, seasoning preference

[1487] Output: Ingredient information and desired seasoning data sent to the server

[1488] Specific operation: The user takes a photo of the ingredients in the refrigerator using the device and enters their desired seasoning.

[1489] Step 8:

[1490] The server analyzes the inputted ingredient information and seasoning preferences and proposes appropriate recipes. The server uses a generative AI model to analyze the inputted data and generate recipes that match the user's preferences.

[1491] Input: Ingredient information, desired seasoning

[1492] Output: Suggested recipe

[1493] How it works: The generative AI model analyzes ingredient information and seasoning preferences to generate an appropriate recipe.

[1494] Step 9:

[1495] While the user is cooking, the cooking status is sent to the server via camera or voice input from the device, and the server analyzes the received cooking status and provides appropriate advice in real time.

[1496] Input: Cooking status data (images, audio)

[1497] Output: Real-time cooking advice

[1498] Specific operation: While cooking, the user takes a photo or voice-over of the situation on the device, and the server analyzes it and advises the next step.

[1499] In this way, at each processing step, the user, server, and terminal work together to input, analyze, and output data, enabling users to easily create healthy menus and efficiently procure the ingredients needed to make them.

[1500] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1501] The present invention combines a system that allows users to easily create personalized menus and efficiently source the necessary ingredients with an emotion engine. This system analyzes the user's emotions and provides recipes and cooking advice accordingly, creating a more personalized experience. Below, we will explain in detail how to implement this system.

[1502] System configuration

[1503] This system operates by combining user devices, a server, collaboration with partner companies, and an emotion engine. The user device accepts user input, communicates with the server, and acquires emotion data through the emotion engine. The server analyzes the information received from the user and is responsible for generating menus, suggesting recipes, and managing ingredients. Partner companies also provide and deliver the necessary ingredients.

[1504] Creating a menu and incorporating emotions

[1505] 1. Enter basic user information

[1506] Users input their health condition, allergies, disliked ingredients, family composition, desired budget, preparation period, and desired menu direction. In addition, the user can also input their current emotional state. This information is sent to the server from the user's device.

[1507] Example: "Health condition: good, Allergies: peanuts, Dislikes: tomatoes, Family size: 4 people, Desired budget: 5,000 yen per week, Menu preference: healthy, Emotional state: tired"

[1508] 2. Data analysis and menu generation by the server

[1509] The server receives the basic information and emotional state sent, and obtains price information for the user's area via an API. Based on this information, it analyzes and generates a menu for the specified period. This also includes menu suggestions to soothe the user's emotions based on their emotional state.

[1510] For example, if you are tired, a simple yet balanced meal will be suggested: "Breakfast: Oatmeal, Lunch: Simple Salad, Dinner: Stir-fried Chicken and Vegetables."

[1511] 3. Present the ingredients list and estimated budget

[1512] The server calculates the necessary ingredients and estimated budget based on the generated menu, and sends them to the user's device, allowing the user to understand the ingredients they need to purchase and their budget.

[1513] Example: Based on the estimated budget calculated by the server, the user is presented with a list of ingredients needed: oatmeal, lettuce, chicken, carrots, along with the total budget.

[1514] Recipe suggestions and cooking assistance

[1515] 1. Enter ingredient information

[1516] Users can input information about the ingredients in their refrigerator using images, voice, or text, as well as their desired seasoning and current emotional state.

[1517] Example: Take a photo of the ingredients in the refrigerator and enter the text "I want it to have a Japanese flavor" and "Emotional state: Calm."

[1518] 2. Analysis of food information and emotional state by the server

[1519] The server analyzes the transmitted ingredient information and emotional state, and suggests dishes that can be made based on the available ingredients and desired seasonings. It also suggests recipes that will help you relax depending on your emotional state.

[1520] Example: Based on the ingredients identified through photo analysis: chicken, cabbage, and carrots, the system suggests "teriyaki chicken."

[1521] 3. Real-time cooking advice

[1522] While cooking, users can use their device to send cooking status information to the server using a camera or voice input. The server then analyzes the received cooking status and the user's emotional state and provides appropriate advice in real time.

[1523] For example, if you say, "The chicken is browned," the server will advise you, "Next, add the sauce and cook over medium heat for 5 minutes," and also send a relaxing message depending on your emotional state.

[1524] Food ordering and delivery

[1525] 1. List and order ingredients

[1526] The server automatically generates a list of ingredients required based on the menu, which then becomes the basis for ordering from partner companies.

[1527] For example, "Oatmeal, lettuce, chicken, and carrots" is listed and an order request is sent to the partner company.

[1528] 2. Confirm and confirm your order

[1529] The user confirms the order details and presses the confirmation button to complete the order. The confirmed order is then officially sent to the partner company via the server.

[1530] Example: When a user checks the order details and presses the confirm button, the order is confirmed and sent to the partner company.

[1531] 3. Delivery request and notification

[1532] After the order is confirmed, the server arranges for delivery and sets the delivery request details. The user is notified of the estimated delivery time, etc.

[1533] Example: The server requests delivery from a courier company and sends a notification to the user saying, "Estimated delivery time: tomorrow at 2:00 PM."

[1534] This system allows users to eat healthy and hassle-free, and to efficiently source ingredients. Furthermore, an emotion engine is incorporated into the recipe suggestions and real-time cooking advice, providing support tailored to the user's emotional state. This increases the enjoyment and satisfaction of cooking, and is expected to be even more practical.

[1535] The processing flow will be explained below.

[1536] Creating a menu and incorporating emotions

[1537] Step 1:

[1538] Using a device, users enter their health condition, allergies, food dislikes, family composition, desired budget, preparation period, desired menu direction, and current emotional state into a dedicated form.

[1539] Specifically, the user enters the following information: "Health condition: good, Allergies: peanuts, Dislikes: tomatoes, Family size: 4 people, Desired budget: 5,000 yen per week, Menu direction: healthy, Emotional state: tired."

[1540] Step 2:

[1541] The device sends the user's input data to the server.

[1542] Specifically, the device uses a REST API to send the user's input data in JSON format to the server, along with their emotional state.

[1543] Step 3:

[1544] The server analyzes the user's basic information and emotional state and obtains price information for the user's area via an API.

[1545] The server identifies the area where the user lives and obtains the latest price information from supermarkets and markets in that area.

[1546] Step 4:

[1547] The server generates a menu for the specified period based on the basic information, emotional state, and price information analyzed by the server.

[1548] Specifically, the menu generation algorithm selects recipes that suit a user's health status, family composition, and budget, and suggests easy-to-prepare dishes for tired users, such as "Breakfast: oatmeal, Lunch: simple salad, Dinner: stir-fried chicken with vegetables."

[1549] Step 5:

[1550] Based on the generated menu, the server calculates the list of ingredients needed and the estimated budget, and sends it to the user's terminal.

[1551] Based on the ingredient list and estimated budget calculated by the server, a list of "necessary ingredients: oatmeal, lettuce, chicken, carrots" and the total budget are created and sent to the terminal.

[1552] Recipe suggestions and cooking assistance

[1553] Step 1:

[1554] The user uses the terminal to input information about the ingredients in the refrigerator (either images, voice, or text), desired seasoning, and their current emotional state.

[1555] As a specific example, a user takes a photo of the ingredients in the refrigerator and enters text such as "I want it to have a Japanese flavor" and "Emotional state: calm."

[1556] Step 2:

[1557] The terminal transmits the inputted information on ingredients, desired seasoning, and emotional state to the server.

[1558] Using the REST API, image data and seasoning requests are sent in JSON or XML format.

[1559] Step 3:

[1560] The server analyzes the received ingredient information, emotional state, and seasoning preferences, and suggests dishes that match the available ingredients and the user's emotions.

[1561] An image analysis algorithm identifies the type of ingredient and suggests the most suitable dish, such as "Teriyaki Chicken," from recipes in the database according to "Emotional State: Calm."

[1562] Step 4:

[1563] The server sends the suggested recipe information to the user's terminal and displays it.

[1564] A list of suggested dishes, cooking instructions, and any additional ingredients needed are sent in JSON format and displayed on the device.

[1565] Step 5:

[1566] While the user is cooking, the cooking status is sent to the server in real time using the device's camera or voice input.

[1567] Specifically, the cooking status is input by voice, such as "the chicken is browned."

[1568] Step 6:

[1569] The server analyzes the cooking status and emotional state received and sends appropriate real-time cooking advice to the user's device.

[1570] It will check the cooking steps, advise you on how to cook the food, and then give you advice such as "Next, add the sauce and cook over medium heat for another 5 minutes," and also send you relaxing messages based on your emotional state.

[1571] Food ordering and delivery

[1572] Step 1:

[1573] The server lists the missing ingredients based on the generated menu.

[1574] It compares the menu with the user's list of ingredients on hand and lists any ingredients that are lacking, such as "oatmeal, lettuce, chicken, and carrots."

[1575] Step 2:

[1576] The server generates an order request to the partner company based on the listed ingredient information.

[1577] Specifically, order information is generated in JSON format and sent to partner companies via API.

[1578] Step 3:

[1579] The server sends an order request to the user's terminal, which the user confirms.

[1580] The user receives an email or app notification confirming their order.

[1581] Step 4:

[1582] The user checks the order details and presses the confirmation button on the terminal.

[1583] When the confirm button is clicked, the confirmation information is sent to the server.

[1584] Step 5:

[1585] The server finally transmits the confirmed order information to the partner company and confirms the order.

[1586] Order confirmation information is sent to partner companies via API.

[1587] Step 6:

[1588] After the order is confirmed, the server sets up a delivery request and arranges an appropriate delivery time.

[1589] Use the delivery company's API to set up delivery and set the expected delivery time for the user.

[1590] Step 7:

[1591] The server notifies the user of the delivery request and provides delivery information.

[1592] Information such as "Estimated delivery time: tomorrow 2pm" is sent to users via email or app notifications.

[1593] Example 2

[1594] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1595] Conventional menu generation systems only suggest menus based on the user's basic information and price information, making it impossible to realize personalized menu suggestions that take the user's emotional state into account. Furthermore, there was no system that properly analyzed the user's ingredient information and cooking status to provide real-time cooking advice, resulting in low user convenience and satisfaction. Furthermore, there was a lack of efficient ways to order the necessary ingredients and process delivery procedures, making it difficult for users to procure ingredients without hassle.

[1596] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for inputting the user's basic information and emotional state; means for acquiring price information for the user's residential area; means for analyzing the input basic information, emotional state, and acquired price information to generate an optimal menu; means for generating a necessary ingredient list and estimated budget based on the generated menu; and means for presenting the ingredient list and estimated budget to the user. This enables personalized menu suggestions that take the user's emotional state into consideration. The server also includes means for inputting the user's ingredient information and emotional state; means for suggesting dishes that can be made by analyzing the input ingredient information, emotional state, and seasoning preferences; and means for analyzing the cooking status and emotional state during cooking to provide cooking advice in real time. This allows the user to receive appropriate cooking advice in real time. The server also includes means for listing necessary ingredients based on the generated menu; means for ordering the listed ingredients from partner companies; means for setting a delivery request after the order is confirmed; and means for notifying the user of the delivery request. This allows the user to efficiently procure the necessary ingredients without any hassle.

[1597] "Basic user information" refers to information regarding the user's health condition, allergies, food preferences, family composition, desired budget, and creation period.

[1598] "Emotional state" is information that represents the user's current mood or mental state.

[1599] "Price information" is information about the prices of common goods and services in the user's area.

[1600] "Menu" means a plan of meals that a User intends to consume within a specified time period.

[1601] The "ingredient list" is a list of ingredients required based on the generated menu.

[1602] The "projected budget" is the estimated cost required to purchase ingredients calculated based on the ingredient list.

[1603] "Ingredient information" refers to information about the type, quantity, and condition of ingredients currently owned by the user.

[1604] "Seasoning preference" refers to the user's preference regarding the taste and cooking method of the food.

[1605] "Cooking status" is information about the progress and process of cooking.

[1606] "Real-time cooking advice" refers to specific cooking instructions and advice provided instantly based on the user's cooking status while cooking.

[1607] "Partner companies" refer to companies that provide and deliver ingredients.

[1608] "Ordering" refers to the act of formally requesting ingredients needed by a user from a partner company.

[1609] A "delivery request" is a request to arrange for the delivery of the ingredients to the user after the order is confirmed.

[1610] "Notification" is a means of informing users of delivery requests and other important information.

[1611] This invention is a system that allows users to easily create menus that suit their needs and efficiently procure the necessary ingredients, and it also combines an emotion engine. This system analyzes the user's emotions and provides recipes and cooking advice accordingly, realizing a more personalized experience.

[1612] System configuration

[1613] This system operates by combining a user device, a server, collaboration with partner companies, and an emotion engine. The user device accepts input of the user's basic information and emotional state, communicates with the server, and transmits the data to the server. The server analyzes the information received from the user and is responsible for generating menus, suggesting recipes, and managing ingredients. Partner companies also provide and deliver the necessary ingredients.

[1614] Entering the user's basic information and emotional state

[1615] Users can use devices such as smartphones or tablets to input their health status, allergies, food preferences, family composition, desired budget, preparation period, and desired menu direction. In addition, users can also input their current emotional state. This information is sent from the device to the server.

[1616] Examples:

[1617] "Health: Good, Allergies: Peanuts, Dislikes: Tomatoes, Family size: 4, Desired budget: 5,000 yen per week, Menu preference: Healthy, Emotional state: Tired"

[1618] Data analysis and menu generation by server

[1619] The server receives the basic information and emotional state sent, and obtains price information for the user's area via an API. Based on this information, it analyzes and generates a menu for the specified period. This also includes menu suggestions to soothe the user's emotions based on their emotional state.

[1620] Examples:

[1621] If you're tired, the app suggests easy-to-prepare, nutritionally balanced meals: "Breakfast: Oatmeal, Lunch: Simple Salad, Dinner: Stir-fried Chicken and Vegetables."

[1622] Present the ingredients list and estimated budget

[1623] The server calculates the necessary ingredients and estimated budget based on the generated menu and sends this to the user's terminal, allowing the user to understand the ingredients they need to purchase and their budget.

[1624] Examples:

[1625] The user is given a list of "Ingredients Needed: Oatmeal, Lettuce, Chicken, Carrots" along with a budget total.

[1626] Enter ingredients and emotional state

[1627] Users can input information about the ingredients in their refrigerator using images, voice, or text, as well as their seasoning preferences and current emotional state.

[1628] Examples:

[1629] Take a photo of the ingredients in your refrigerator and enter the text "I want it to have a Japanese flavor" and "Emotional state: calm."

[1630] Server-based analysis of ingredients and emotional state

[1631] The server analyzes the transmitted ingredient information and emotional state, and suggests dishes that can be made based on the available ingredients and desired seasonings. It also suggests recipes that will help you relax depending on your emotional state.

[1632] Examples:

[1633] Based on the ingredients identified through photo analysis: chicken, cabbage, and carrots, we suggest "chicken teriyaki."

[1634] Real-time cooking advice

[1635] While cooking, the user uses the device to send the cooking status to the server using the camera or voice input, and the server analyzes this information and the user's emotional state to provide appropriate advice in real time.

[1636] Examples:

[1637] If you say, "The chicken is browned," the server will advise you, "Next, add the sauce and cook over medium heat for 5 minutes," and it will also send you a relaxing message based on your emotional state.

[1638] Ordering and delivery of necessary ingredients

[1639] The server lists the ingredients needed based on the menu and sends an order request to the partner company. The user confirms the order on the device and completes the order by pressing the confirmation button. After confirmation, the server arranges for delivery and notifies the user of the delivery request.

[1640] Examples:

[1641] "Oatmeal, lettuce, chicken, carrots" is listed and an order request is sent to the partner company. When the user confirms the order and presses the confirm button, the server requests delivery from the delivery company and notifies the user that "Estimated delivery time: tomorrow 2:00 PM."

[1642] Prompt Sentence Examples

[1643] Based on the following information, it generates a weekly menu and ingredient list, and suggests recipes that take into account your emotional state.

[1644] "Health: Good, Allergies: Peanuts, Disliked Food: Tomatoes, Family Size: 4, Budget: 5,000 yen per week, Menu Direction: Healthy, Emotional State: Tired"

[1645] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1646] Step 1:

[1647] Entering the user's basic information and emotional state

[1648] Users input their health condition, allergies, food preferences, family composition, desired budget, preparation period, and desired menu direction into the device, along with their emotional state. Input is done through a form in the app.

[1649] Input: health condition, allergies, disliked ingredients, family composition, desired budget, preparation period, desired menu direction, emotional state

[1650] Output: Basic information and emotional state data

[1651] Specific actions: The user opens the app on their smartphone, enters the required information into the form, and presses the submit button.

[1652] Step 2:

[1653] Sending basic information and emotional state to the server

[1654] The terminal transmits the input basic information and emotional state to the server.

[1655] Input: Basic information and emotional state data

[1656] Output: Data sent to the server

[1657] Specific operation: When the user presses the send button, the device sends the data to the server via the Internet.

[1658] Step 3:

[1659] Data analysis and menu generation by server

[1660] The server analyzes the received basic information and emotional state, obtains price information for the user's area via an API, and runs a program to generate a menu based on this information.

[1661] Input: Basic information, emotional state, price information

[1662] Data processing / calculation: Match basic information with emotional state and generate a menu based on price information in the area of ​​residence.

[1663] Output: Generated menu

[1664] Specific operation: The server calls the API to obtain price information and then generates a menu using an algorithm.

[1665] Step 4:

[1666] Present the ingredients list and estimated budget

[1667] The server calculates the list of ingredients needed and the estimated budget based on the generated menu, and transmits this to the user terminal.

[1668] Input: Generated menu

[1669] Data processing / calculation: Calculate the ingredients list and budget based on the menu.

[1670] Output: Ingredient list, estimated budget

[1671] What it does: The server sends the calculation results to the device, and the app displays them.

[1672] Step 5:

[1673] Enter ingredients and emotional state

[1674] The user inputs information about the ingredients in the refrigerator in the form of images, voice, or text, as well as their desired seasoning and current emotional state.

[1675] Input: Ingredient information, seasoning preference, emotional state

[1676] Output: Data sent to the server

[1677] Specific operation: The user takes a photo of the ingredients in the refrigerator and sends it by text inputting their seasoning preference and emotional state.

[1678] Step 6:

[1679] Server-based analysis of ingredients and emotional state

[1680] The server analyzes the transmitted ingredient information and emotional state, and suggests dishes that can be made based on the available ingredients and desired seasonings.

[1681] Input: Ingredient information, emotional state, desired seasoning

[1682] Data processing / computation: Identifying ingredients using image recognition, using sentiment analysis algorithms

[1683] Output: Suggested dishes

[1684] Specific operation: The server analyzes the image, searches for recipes in the database, and suggests the most suitable dish.

[1685] Step 7:

[1686] Real-time cooking advice

[1687] While cooking, the user uses the device to send cooking information to the server using a camera or voice input, and the server analyzes the information and provides appropriate advice in real time.

[1688] Input: Cooking status, emotional state

[1689] Data processing / calculation: Analyzing cooking status and emotional state

[1690] Output: Real-time cooking advice

[1691] Specific behavior: The server analyzes the voice input and suggests the next cooking steps.

[1692] Step 8:

[1693] List and order the ingredients you need

[1694] The server lists the ingredients needed based on the menu and sends an order request to partner companies.

[1695] Input: Menu

[1696] Data processing / calculation: Creating an ingredient list

[1697] Output: Order request

[1698] What happens: The server creates a list and sends it to partner companies through the ordering system.

[1699] Step 9:

[1700] Confirm and confirm your order

[1701] The user checks the order details on the terminal and presses the confirmation button to complete the order. The server then officially sends the confirmed order to the partner company.

[1702] Input: Ingredients list

[1703] Output: Order confirmation notification

[1704] Specific operation: The user checks the order details on the confirmation screen and presses the confirm button.

[1705] Step 10:

[1706] Delivery requests and notifications

[1707] After the order is confirmed, the server arranges for delivery and sets the delivery request details. The user is notified of the estimated delivery time, etc.

[1708] Input: Order Information

[1709] Data processing / calculation: Delivery schedule setting

[1710] Output: Delivery notification

[1711] Specific operation: The server requests the delivery company and sends a notification to the user.

[1712] (Application example 2)

[1713] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1714] In recent years, it has become increasingly important for users to easily create menus that suit their own health and emotional state and to efficiently source ingredients. However, conventional systems do not provide menu suggestions or cooking advice that take the user's emotions into account, making it difficult to provide a personalized experience. In addition, sourcing ingredients and requesting delivery is time-consuming for users, which is also an issue that lacks efficiency.

[1715] The specification processing by the specification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for inputting basic information about the user, means for acquiring price information for the residential area, means for analyzing the input basic information and the acquired price information to generate an optimal menu, means for generating a list of necessary ingredients and a projected budget based on the generated menu, means for analyzing the user's emotions, means for suggesting a personalized menu based on the analyzed emotions, and means for presenting the ingredient list and the projected budget to the user. This enables personalized menu suggestions and efficient ingredient procurement according to the user's emotional state.

[1716] "Means for inputting basic user information" refers to the means by which users input their health condition, allergies, food preferences, family composition, desired budget, preparation period, desired menu direction, etc. into the system.

[1717] "Means for obtaining price information in the residential area" refers to means for obtaining price information in the area where the user lives from the Internet or an external database.

[1718] "Means for analyzing the input basic information and the acquired price information to generate the optimal menu" refers to means for automatically generating the optimal menu for the user based on the basic information input by the user and the acquired price information.

[1719] The "means for generating a list of necessary ingredients and a projected budget based on the generated menu" refers to a means for calculating and generating a list of necessary ingredients and a projected budget based on the generated menu.

[1720] "Means for analyzing user emotions" refers to means for analyzing the user's facial image and voice data to detect their emotional state at that time.

[1721] The "means for making personalized menu suggestions based on analyzed emotions" is a means for suggesting a menu that is suitable for each individual user according to the user's emotional state.

[1722] The "means for presenting the ingredient list and estimated budget to the user" refers to a means for displaying the generated ingredient list and estimated budget on the user's terminal.

[1723] "Means for inputting information about ingredients owned by the user" refers to means for inputting information about ingredients owned by the user using images, voice, text, etc.

[1724] "Means for suggesting dishes that can be made by analyzing the inputted ingredient information and desired seasoning" refers to means for suggesting dishes that can be made based on the ingredient information and desired seasoning that the user inputs.

[1725] The "means for analyzing the cooking situation during cooking and providing cooking advice in real time" is a means for analyzing the cooking situation using images and sounds and providing cooking advice in real time.

[1726] The "means for generating cooking advice based on the emotional state of the user" is a means for analyzing the emotional state of the user and generating cooking advice that matches that emotion.

[1727] The "means for providing feedback according to the cooking situation and emotional state" is a means for providing appropriate feedback to the user based on the cooking situation and the user's emotional state.

[1728] The "means for listing necessary ingredients based on the generated menu" is a means for listing necessary ingredients based on the generated menu.

[1729] The "means for ordering the listed ingredients from partner companies" is a means for automatically ordering the listed ingredients from external partner companies.

[1730] "Means for requesting delivery after confirming an order" refers to a means for a user to arrange delivery after confirming and confirming the order details.

[1731] "Means for notifying the user of the details of the delivery request" refers to means for notifying the user of information such as the order details and the estimated delivery time.

[1732] The "means for proposing an optimal delivery time based on the emotional state of the user" is a means for proposing an optimal delivery time by taking into account the emotional state of the user.

[1733] To implement this invention, a system is required that can efficiently acquire a user's basic information and emotional state, and then use that information to create menus, procure ingredients, and provide cooking advice. This system operates primarily in cooperation with the user's terminal, a server, and partner companies. Specific embodiments are described below.

[1734] Hardware and Software Configuration

[1735] Hardware:

[1736] User device: A smartphone or tablet with a camera and internet connection.

[1737] Server: Responsible for data processing and analysis. A server with internet access.

[1738] software:

[1739] OpenCV: A library used for face recognition and emotion detection.

[1740] Emotion Detection API: An external API that performs emotion analysis.

[1741] Flask: A Python framework for building server-side recipe generation engines.

[1742] Delivery Service API: An API for handling food orders with partner companies.

[1743] System Functions and Processing

[1744] 1. Sentiment analysis and menu suggestions

[1745] The server first analyzes the user's emotional state. It captures a facial image using the user's device's camera and sends the image data to the Emotion Detection API. The API analyzes the image and returns the user's emotional state. Based on this emotional state, it suggests an appropriate menu from a database stored on the Flask server.

[1746] Example: "Since we detected the emotion 'tired' from your facial image, we will suggest a simple yet nutritionally balanced meal: 'Stir-fried chicken with vegetables.'"

[1747] 2. List and order ingredients

[1748] The server automatically generates a list of ingredients based on the proposed menu and calculates the estimated budget. This is then displayed on the user's device, and once the user confirms and confirms, the order information is sent to the partner company via the Delivery Service API.

[1749] For example: "I ordered lettuce, chicken, and carrots. The total cost is 1000 yen."

[1750] 3. Analysis of cooking status and provision of advice

[1751] The server receives and analyzes camera images and voice input from the user while cooking, providing real-time advice at the appropriate time based on the cooking situation. By taking the user's emotional state into consideration, the system provides feedback aimed at maximizing relaxation.

[1752] For example: "I've determined that the chicken is browned. Now add the sauce and cook over medium heat for 5 minutes."

[1753] Example prompt:

[1754] Below are some example prompts that users can use to suggest meals based on their emotional state:

[1755] Perform "face photo capture," "detect emotional state," and suggest "emotion-based menus."

[1756] summary

[1757] This invention allows users to receive menu suggestions tailored to their emotional state, efficiently source the ingredients they need, and receive real-time advice based on the cooking situation, providing a more personalized cooking experience.

[1758] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1759] Program processing steps

[1760] Step 1: Enter basic information

[1761] The user inputs their health condition, allergies, food preferences, family composition, desired budget, preparation period, and desired menu direction. This information is sent from the user terminal.

[1762] Input: Personal information entered by the user

[1763] Output: Basic information sent from the user device to the server

[1764] Step 2: Sentiment Analysis

[1765] Users take a picture of their face with their smartphone camera and send the image data to the Emotion Detection API, which analyzes the image and returns the user's emotional state.

[1766] Input: User's face image

[1767] Output: Sentiment analysis result (e.g., tired, relaxed)

[1768] Step 3: Obtaining price information

[1769] The server obtains price information from the Internet or external databases based on the user's residential area information.

[1770] Input: User's residential area information

[1771] Output: Price information data

[1772] Step 4: Create a menu

[1773] The server generates the optimal menu based on the input basic information, emotional state, and acquired price information. It uses the recipe database on the Flask server to suggest personalized menus.

[1774] Input: Basic information, emotional state, price information

[1775] Output: Optimal menu data

[1776] Step 5: Generate a list of ingredients needed and a projected budget

[1777] The server calculates and generates a list of necessary ingredients and a projected budget based on the generated menu.

[1778] Input: Optimal menu data

[1779] Output: Ingredient list, estimated budget

[1780] Step 6: Present to the user

[1781] The server sends the generated ingredient list and estimated budget to the user's terminal, where the user can check it on the screen.

[1782] Input: Ingredient list, estimated budget

[1783] Output: Information displayed on the user's terminal

[1784] Step 7: Order ingredients

[1785] The user confirms the ingredients list and budget and confirms the order. The server then sends this order information to the partner company via the Delivery Service API.

[1786] Input: Ingredient list, estimated budget, user order confirmation

[1787] Output: Order data for partner companies

[1788] Step 8: Arrange delivery and notify

[1789] Once the order is confirmed, the server sends a delivery request to the delivery company and notifies the user of the estimated time. The server also suggests the optimal delivery time, taking into account the user's emotional state.

[1790] Input: Order data, user emotional state

[1791] Output: Request to courier, notification to user

[1792] Step 9: Cooking Tips

[1793] While cooking, users take pictures of their cooking progress with their smartphone camera and report the progress by voice. The server analyzes the information and sends cooking advice in real time. The server also analyzes the user's emotional state and sends corresponding relaxing messages.

[1794] Input: Images and audio of cooking status

[1795] Output: Real-time cooking advice, emotion-based messages

[1796] Through this series of processes, users can receive personalized menu suggestions, efficiently source the necessary ingredients, and receive accurate advice while cooking.

[1797] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1798] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1799] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1800] [Fourth embodiment]

[1801] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1802] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1803] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1804] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1805] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1806] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1807] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1808] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1809] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1810] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1811] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1812] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1813] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1814] The present invention provides a system that allows users to easily create menus that suit their needs and efficiently procure the necessary ingredients. Hereinafter, an embodiment of the system will be described in detail.

[1815] System configuration

[1816] This system operates in cooperation with user terminals, a server, and partner companies. The user terminal accepts user input and communicates with the server. The server analyzes the information received from the user and is responsible for creating menus, suggesting recipes, and managing ingredients. Partner companies also provide and deliver the necessary ingredients.

[1817] Creating a menu

[1818] 1. Enter basic user information

[1819] Users input their health condition, allergies, food preferences, family composition, desired budget, preparation period, and menu direction. This information is sent to the server from the user's terminal.

[1820] Example: "Health condition: good, Allergies: peanuts, Dislikes: tomatoes, Family size: 4, Desired budget: 5,000 yen per week, Menu preference: healthy"

[1821] 2. Data analysis and menu generation by the server

[1822] The server receives the basic information and obtains price information for the user's area. Based on this information, it analyzes and generates a menu for the specified period. The generated menu is optimized taking into account the user's health condition, family composition, and budget.

[1823] Example: The server compares the price information it has obtained and generates a menu such as "Breakfast: oatmeal, Lunch: salad, Dinner: stir-fried chicken and vegetables."

[1824] 3. Present the ingredients list and estimated budget

[1825] The server calculates the necessary ingredients and estimated budget based on the generated menu, and sends them to the user's device, allowing the user to understand the ingredients they need to purchase and their budget.

[1826] Example: Based on the estimated budget calculated by the server, the user is presented with a list of ingredients needed: oatmeal, lettuce, chicken, carrots, along with the total budget.

[1827] Recipe suggestions and cooking assistance

[1828] 1. Enter ingredient information

[1829] Users input information about the ingredients in their refrigerator. This information can be entered using images, voice, or text. They can also input their desired seasonings at the same time.

[1830] Example: Take a photo of the ingredients in your refrigerator and enter the text "I want it to have a Japanese flavor."

[1831] 2. Server analyzes ingredient information and suggests recipes

[1832] The server analyzes the submitted ingredient information and seasoning preferences to identify available ingredients, and then suggests recipes that match the user's desired seasonings based on these ingredients.

[1833] Example: Based on the ingredients identified through photo analysis: chicken, cabbage, and carrots, the system suggests "teriyaki chicken."

[1834] 3. Real-time cooking advice

[1835] While cooking, users can use their device to send cooking information to the server using a camera or voice input, and the server analyzes the received information and provides appropriate advice in real time.

[1836] For example, say, "The chicken is browned," and the server will advise, "Next, add the sauce and cook over medium heat for 5 minutes."

[1837] Food ordering and delivery

[1838] 1. List and order ingredients

[1839] The server automatically generates a list of ingredients required based on the menu, which then becomes the basis for ordering from partner companies.

[1840] For example, "Oatmeal, lettuce, chicken, and carrots" is listed and an order request is sent to the partner company.

[1841] 2. Confirm and confirm your order

[1842] The user confirms the order details and presses the confirmation button to complete the order. The confirmed order is then officially sent to the partner company via the server.

[1843] Example: When a user checks the order details and presses the confirm button, the order is confirmed and sent to the partner company.

[1844] 3. Delivery request and notification

[1845] After the order is confirmed, the server arranges for delivery and sets the delivery request details. The user is notified of the estimated delivery time, etc.

[1846] Example: The server requests delivery from a courier company and sends a notification to the user saying, "Estimated delivery time: tomorrow at 2:00 PM."

[1847] This system allows users to lead a healthy diet without hassle and to efficiently source ingredients. In addition, it provides recipe suggestions and real-time cooking advice, so even beginners can cook with confidence.

[1848] The processing flow will be explained below.

[1849] Creating a menu

[1850] Step 1:

[1851] Users use a device to enter information about their health condition, allergies, food preferences, family composition, desired budget, preparation period, and desired menu direction into a dedicated form.

[1852] Specifically, the user enters the following information: "health condition: good, allergies: peanuts, dislikes: tomatoes, family size: 4 people, desired budget: 5,000 yen per week, menu preference: healthy."

[1853] Step 2:

[1854] The device sends the user's input data to the server.

[1855] Specifically, the terminal uses a REST API to send the user's input data to the server in JSON format.

[1856] Step 3:

[1857] The server analyzes the basic information received from the user and obtains price information for the user's area of ​​residence via an API.

[1858] The server identifies the area where the user lives and obtains the latest price information from supermarkets and markets in that area.

[1859] Step 4:

[1860] The server generates a menu for the specified period based on the price information obtained and the user's basic information.

[1861] The menu generation algorithm creates a list of recipes that suit your health status, family composition, and budget, and selects the optimal menu.

[1862] Step 5:

[1863] The server calculates the required ingredients list and estimated budget based on the generated menu and sends this information to the user's terminal.

[1864] Based on the estimated budget calculated by the server, a list of "necessary ingredients: oatmeal, lettuce, chicken, carrots" and the total budget are created and sent to the terminal.

[1865] Recipe suggestions and cooking assistance

[1866] Step 1:

[1867] The user uses the terminal to input information about the ingredients in the refrigerator (either images, voice, or text) and desired seasonings.

[1868] As a specific example, a user takes a photo of the ingredients in the refrigerator and enters the text, "I want it to have a Japanese flavor."

[1869] Step 2:

[1870] The terminal transmits the inputted ingredient information and desired seasoning to the server.

[1871] Using the REST API, image data and seasoning requests are sent to the server in JSON or XML format.

[1872] Step 3:

[1873] The server analyzes the received ingredient information and suggests dishes that can be made based on the available ingredients and desired seasonings.

[1874] Image analysis algorithms identify the type of ingredient and select the most suitable dish from a database of recipes.

[1875] Step 4:

[1876] The server sends the suggested recipe information to the user's terminal and displays it.

[1877] A list of suggested dishes, cooking instructions, and any additional ingredients needed are sent in JSON format, which is then displayed on the device.

[1878] Step 5:

[1879] While the user is cooking, the cooking status is sent to the server in real time using the device's camera or voice input.

[1880] Specifically, the cooking status is input by voice, such as "The chicken is browned."

[1881] Step 6:

[1882] The server analyzes the cooking status received and sends appropriate real-time cooking advice to the user's device.

[1883] Check cooking instructions and send advice such as, "Next, add the sauce and cook over medium heat for another 5 minutes."

[1884] Food ordering and delivery

[1885] Step 1:

[1886] The server lists the missing ingredients based on the generated menu.

[1887] It compares the menu with the user's list of ingredients on hand and lists any ingredients that are lacking, such as "oatmeal, lettuce, chicken, and carrots."

[1888] Step 2:

[1889] The server generates an order request to the partner company based on the listed ingredient information.

[1890] Specifically, order information is generated in JSON format and sent to partner companies via API.

[1891] Step 3:

[1892] The server sends an order request to the user's terminal, which the user confirms.

[1893] The user receives an email or app notification confirming their order.

[1894] Step 4:

[1895] The user checks the order details and presses the confirmation button on the terminal.

[1896] When the confirm button is clicked, the confirmation information is sent to the server.

[1897] Step 5:

[1898] The server finally transmits the confirmed order information to the partner company and confirms the order.

[1899] Order confirmation information is sent to partner companies via API.

[1900] Step 6:

[1901] After the order is confirmed, the server sets up a delivery request and arranges an appropriate delivery time.

[1902] Use the delivery company's API to set up delivery and set the expected delivery time for the user.

[1903] Step 7:

[1904] The server notifies the user of the delivery request and provides delivery information.

[1905] Information such as "Estimated delivery time: tomorrow 2pm" is sent to users via email or app notifications.

[1906] Example 1

[1907] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1908] In today's busy lifestyles, it is extremely difficult for users to plan healthy, balanced meals and efficiently source the necessary ingredients. Creating meals that take into account specific health conditions or allergies, and receiving real-time cooking advice, are even more challenging. There is a need for a system that can solve these challenges and enable users to effortlessly create appropriate meals and efficiently obtain the necessary ingredients.

[1909] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1910] In this invention, the server includes means for inputting basic information about the user, means for acquiring price information for the user's area of ​​residence, means for analyzing the input basic information and the acquired price information to generate an optimal menu, means for generating a list of necessary ingredients and a projected budget based on the generated menu, means for presenting the ingredient list and the projected budget to the user, means for analyzing the input ingredient information, photo data, and audio data to identify available ingredients, means for suggesting appropriate recipes based on the user's desired seasonings, and means for analyzing the cooking status during cooking and providing cooking advice in real time. This allows users to effortlessly plan healthy menus, procure ingredients, and efficiently manage the cooking process.

[1911] "User" refers to an individual or family member who uses the system to create menus, source ingredients, and receive cooking advice.

[1912] "Basic information" refers to information necessary for creating a menu and selecting ingredients, such as the user's health condition, allergies, food dislikes, family composition, desired budget, and menu direction.

[1913] "Price information for residential area" refers to price information for specific items in the area where the user lives, and is used for planning meals and calculating budgets.

[1914] The "ingredient list" refers to a list of specific ingredients that should be purchased based on the generated menu.

[1915] "Projected Budget" refers to an estimate of the budget required to purchase the necessary ingredients within a specified period of time.

[1916] "Photo data" refers to image information of ingredients and cooking processes that users take and upload to the system.

[1917] "Voice data" refers to voice instructions, questions, and cooking status reports provided by the user to the system.

[1918] A "recipe" refers to a method of making a dish based on specific ingredients and seasonings.

[1919] "Cooking status" refers to information about the progress and current state of cooking, and is reported to the system in real time.

[1920] "Real-time advice" refers to advice or instructions provided to the user immediately during the cooking process.

[1921] The present invention provides a system that allows users to easily create menus that suit their needs and efficiently procure the necessary ingredients. This system operates in cooperation with a user terminal, a server, and partner companies. Detailed embodiments of the system are described below.

[1922] System configuration

[1923] User Device

[1924] The user terminal is a device such as a smartphone or PC that accepts user input and communicates with the server. The user uses the terminal to input basic information about themselves, ingredients, cooking status, etc.

[1925] server

[1926] The server analyzes the information received from the user and is responsible for generating menus, suggesting recipes, and managing ingredients. It uses a cloud computing platform (such as Google Cloud) to obtain price information for the user's area. It also performs data analysis using Python and R, and generates menus and recipes using machine learning models such as TensorFlow and PyTorch.

[1927] Overview of program processing

[1928] Enter basic user information

[1929] Users input information such as their health condition, allergies, food preferences, family composition, desired budget, preparation period, menu direction, etc. via their smartphone or PC, and this information is sent from the device to the server.

[1930] Example: "Health condition: good, Allergies: peanuts, Dislikes: tomatoes, Family size: 4, Desired budget: 5,000 yen per week, Menu preference: healthy"

[1931] Data analysis and menu generation by server

[1932] The server receives the basic information sent and retrieves price information for the area from a Google Cloud database. Based on this information, it performs data analysis and generates a menu for the specified period.

[1933] Example: The server generates a menu such as "Breakfast: oatmeal, Lunch: salad, Dinner: stir-fried chicken and vegetables" based on price information and basic user information.

[1934] Present the ingredients list and estimated budget

[1935] The server calculates the necessary ingredients and estimated budget based on the generated menu, and sends this to the user's device, allowing the user to understand the ingredients they need to purchase and their budget.

[1936] Example: Provide the user with a list of ingredients needed: oatmeal, lettuce, chicken, carrots, along with a budget total.

[1937] Entering ingredient information and suggesting recipes

[1938] Entering ingredient information

[1939] Users can input information about the ingredients in their refrigerator using images, voice, or text, and can also input their desired seasonings for cooking.

[1940] Example: Take a photo of the ingredients in your refrigerator and enter the text "I want to make it Japanese flavor."

[1941] Server analyzes ingredients and suggests recipes

[1942] The server analyzes the submitted ingredient information and seasoning preferences, identifies available ingredients, and then suggests recipes using those ingredients. Image analysis uses technologies such as OpenCV and YOLOv3.

[1943] Example: Suggest "Teriyaki chicken" based on "Ingredients identified through photo analysis: chicken, cabbage, carrots."

[1944] Real-time cooking advice

[1945] While cooking, users can use their device to send cooking status information to the server using a camera or voice input. The server then analyzes the received information and provides appropriate advice in real time.

[1946] For example, say, "The chicken is browned," and the server will advise, "Next, add the sauce and cook over medium heat for 5 minutes."

[1947] Food ordering and delivery

[1948] Listing and ordering ingredients

[1949] The server automatically generates a list of ingredients required based on the menu and sends an order request to the partner company.

[1950] For example, "Oatmeal, lettuce, chicken, and carrots" is listed and an order request is sent to the partner company.

[1951] Confirm and confirm your order

[1952] The user checks the order details in the app and presses the confirmation button to complete the order. The confirmed order is then sent to the partner company via the server.

[1953] Example: When the user confirms the order details and presses the confirm button, the order is confirmed with the partner company.

[1954] Delivery requests and notifications

[1955] After the order is confirmed, the server arranges delivery with a delivery company and notifies the user of the estimated delivery time.

[1956] Example: The server requests delivery from a courier company and the user is notified that "Estimated delivery time: tomorrow 2pm."

[1957] Prompt Sentence Examples

[1958] Please enter your health condition, allergy information, family composition, desired budget, creation period, and menu direction.

[1959] "Please enter a photo of the ingredients in your refrigerator and your desired seasoning."

[1960] Please check your order details and press the confirm button.

[1961] In this way, the system helps users effortlessly lead a healthy and balanced diet. Users can easily create menus, efficiently source ingredients, and receive recipe suggestions and real-time cooking advice.

[1962] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1963] Step 1:

[1964] Users enter basic information such as their health condition, allergies, food preferences, family composition, desired budget, preparation period, and menu direction. Using a smartphone or computer, they enter the information into a dedicated app or web form and press the send button. The server receives the basic information entered and stores it in a database.

[1965] Input: health condition, allergy information, disliked ingredients, family composition, desired budget, preparation period, menu direction.

[1966] Output: Notification that basic information has been saved to the database.

[1967] Specific action: The user opens the app on their smartphone, enters the required information into the input form, and taps the submit button.

[1968] Step 2:

[1969] The server retrieves price information from a Google Cloud database based on the user's residential area information. It analyzes the price information and generates the optimal menu for the period specified by the user. It performs data analysis using Python and R, and generates the menu using machine learning models in TensorFlow and PyTorch.

[1970] Input: User's basic information, price information for the area where you live.

[1971] Output: The generated menu information.

[1972] Specific operation: The server calls the Google Cloud API to obtain price data and runs a Python script to generate a menu.

[1973] Step 3:

[1974] The server calculates the necessary ingredients and estimated budget based on the generated menu information. The calculation results are converted into HTML format and sent to the user's device, allowing the user to understand the ingredients they need to purchase and their budget.

[1975] Input: Generated menu information.

[1976] Output: A list of ingredients needed and a projected budget.

[1977] Specific operation: The server runs the program to calculate the required ingredient list and budget, and sends the results as an HTML file to the user's terminal.

[1978] Step 4:

[1979] Users input information about ingredients in their refrigerator using images, voice, or text. They take photos of the ingredients with their smartphone camera and enter their desired seasonings in text. The device then sends this information to the server.

[1980] Input: Information about ingredients in the refrigerator (images, audio, text), desired seasoning.

[1981] Output: Ingredient information and seasoning requests sent to the server.

[1982] What it does: The user takes a photo of the contents of their refrigerator with their smartphone and sends the photo and a text message about the seasoning through the app.

[1983] Step 5:

[1984] The server analyzes the received ingredient information and seasoning preferences. It identifies ingredients using image recognition technologies such as OpenCV and YOLOv3, and understands the seasoning preferences using natural language processing. Based on the results, it suggests an appropriate recipe.

[1985] Input: Submitted ingredient information and seasoning preference.

[1986] Output: A suggested recipe.

[1987] Specific operation: The server launches an image analysis engine to identify ingredients, analyzes desired seasonings using a natural language processing engine, and searches for and suggests appropriate recipes from a database.

[1988] Step 6:

[1989] While cooking, the user sends the cooking status to the server using a camera or voice input, and the server analyzes the received cooking status in real time and provides appropriate cooking advice.

[1990] Input: Image and audio data of the cooking process.

[1991] Output: Real-time cooking advice.

[1992] How it works: The user records the cooking process in real time using the camera and microphone on their smartphone and sends the footage to the server, which analyzes it and provides instant cooking advice.

[1993] Step 7:

[1994] The server automatically generates a list of ingredients based on the menu and sends an order request to partner companies via an API.

[1995] Input: A list of ingredients needed.

[1996] Output: An order request to the partner company.

[1997] Specific operation: The server references the MySQL database to obtain the necessary ingredient information, then calls the partner company's API to send an order request.

[1998] Step 8:

[1999] The user checks the order details on the app and presses the confirmation button to complete the order. The server then sends the confirmed order information to the partner company.

[2000] Input: User confirms and confirms order details.

[2001] Output: Confirmed order submission.

[2002] Specific operation: The user opens the order confirmation screen of the app, checks the order details, and taps the confirm button. The server receives this information and sends a notification of order confirmation to the partner company.

[2003] Step 9:

[2004] After the order is confirmed, the server arranges delivery with a courier and notifies the user of the estimated delivery time.

[2005] Input: Confirmed order information.

[2006] Output: Notification of delivery request details and estimated delivery time.

[2007] Specific operation: The server calls the delivery company's API to request delivery and sends the estimated delivery time to the user via push notification.

[2008] (Application example 1)

[2009] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2010] In today's world, it is extremely time-consuming for users to plan appropriate menus and efficiently procure the necessary ingredients while taking into account their own health condition, allergies, and budget. It is also difficult to receive appropriate recipe suggestions and cooking advice in real time. Furthermore, the lack of online links to purchase ingredients persists, making food procurement a complex process. Given this background, there is a demand for a system that allows users to easily lead a healthy diet.

[2011] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[2012] In this invention, the server includes means for inputting basic information such as the user's health condition, allergies, budget, etc., means for acquiring price information for the user's area of ​​residence, means for analyzing the input basic information and the acquired price information to generate an optimal menu, means for generating a list of necessary ingredients and a projected budget based on the generated menu, means for providing an online link for purchasing ingredients based on the generated menu, and means for presenting the ingredient list and projected budget to the user. This enables users to easily generate optimal menus based on their own health condition, allergies, and budget, efficiently procure the necessary ingredients, and easily purchase them via the online link.

[2013] - "Basic user information" refers to personal information such as the user's health condition, allergies, disliked ingredients, family composition, desired budget, preparation period, and menu direction.

[2014] "Price information for residential area" refers to price information for food and ingredients in the area where the user lives.

[2015] A "menu" refers to a plan of suitable meals for a specific period of time based on the user's basic information.

[2016] The "ingredient list" is a list of ingredients required based on the generated menu.

[2017] The "projected budget" is the total amount of purchase costs predicted based on the ingredient list.

[2018] "Online Links" are links to online stores that allow users to easily purchase the ingredients they need.

[2019] "Ingredient information" is detailed information about foods and ingredients currently owned by the user.

[2020] "Seasoning preference" refers to a user's preference regarding the type of seasoning or flavor they desire for a particular dish.

[2021] "Real-time advice" refers to instructions and advice on cooking methods and next steps that users receive immediately while cooking.

[2022] "Partners" are companies and service providers that work together to supply ingredients and food products and provide them to users.

[2023] A "delivery request" is an arrangement for a delivery service to deliver ordered ingredients or food to a location specified by the user.

[2024] This invention is a system that allows users to create optimal menus that take into consideration their own health conditions, allergies, budget, etc., and to efficiently procure the necessary ingredients. A specific embodiment of this system will be described in detail below.

[2025] System configuration

[2026] This system works in conjunction with user terminals, a server, and partner companies. The user terminal accepts user input and communicates with the server. The server analyzes the information received from the user and is responsible for creating menus, suggesting recipes, and managing ingredients. Partner companies also provide and deliver the necessary ingredients.

[2027] Entering basic user information

[2028] Users input their health condition, allergies, food preferences, family composition, desired budget, preparation period, and menu direction into a user terminal, which then transmits this information to the server.

[2029] Data analysis and menu generation by server

[2030] The server receives the basic information and obtains price information for the user's area. Based on this information, it analyzes and generates a menu for the specified period. The generated menu is optimized taking into account the user's health condition, family composition, and budget.

[2031] Present the ingredients list and estimated budget

[2032] The server calculates the necessary ingredients and estimated budget based on the generated menu and sends them to the user's device. This allows the user to understand the ingredients they need to purchase and their budget. An online link to purchase ingredients is also provided, making it easy for the user to purchase ingredients.

[2033] Food ordering and delivery

[2034] 1. The server lists the ingredients needed based on the generated menu.

[2035] 2. The listed ingredients are ordered from partner companies.

[2036] 3. Once the order details are confirmed, the server will arrange for delivery and notify the user of the delivery request.

[2037] Hardware and software used

[2038] The hardware used includes the smartphone operated by the user, servers, and network communication devices. The software used includes Python programs, REST APIs, and data analysis algorithms. Specific technologies include using generative AI models to generate menus and suggest recipes.

[2039] Examples of concrete examples and prompts

[2040] Examples:

[2041] When users open the app and enter their basic information, the app displays the optimal meal plan based on their health status and budget. It also provides a list of ingredients and links to online shopping sites. By clicking on the link, users can easily purchase the ingredients they need. Users can also enter information about the ingredients in their refrigerator and receive real-time recipe suggestions.

[2042] Example prompt sentence:

[2043] "Create an app that generates optimal meals based on the user's health, allergies, budget, etc., and provides the necessary ingredients and online purchasing links."

[2044] In this way, the present invention allows users to easily lead a healthy diet and efficiently source ingredients. Furthermore, the system provides real-time cooking advice, making cooking easy even for beginners.

[2045] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[2046] Step 1:

[2047] The user uses a terminal to input basic information such as health status, allergies, food preferences, family composition, budget, and menu direction. The input basic information is sent from the terminal to the server.

[2048] Input: User's health condition, allergies, disliked ingredients, family composition, budget, menu direction

[2049] Output: User basic information sent to the server

[2050] Specific operation: The user enters information into the input form on the device and presses the "Submit" button.

[2051] Step 2:

[2052] The server retrieves price information for the user's area. The server uses the Internet or existing databases to retrieve the latest food price information for the area the user enters.

[2053] Input: User's residential area

[2054] Output: Price information for your area

[2055] What happens: The server sends requests to APIs and third-party databases to retrieve price information.

[2056] Step 3:

[2057] The server analyzes basic information and price information to generate the optimal menu for the specified period. Using a generative AI model, the server generates a menu based on food price information, taking into account the user's health condition, budget, and allergies.

[2058] Input: User basic information, price information

[2059] Output: Generated menu data

[2060] How it works: The generative AI model processes the input data and generates the optimal menu.

[2061] Step 4:

[2062] The server calculates the necessary ingredients and estimated budget based on the generated menu and sends it to the user's device. This allows the user to understand the ingredients they need to purchase and their budget. An online purchase link is also provided.

[2063] Input: Generated menu data

[2064] Output: Ingredients list, estimated budget, online purchase link

[2065] What it does: The server analyzes the menu data, calculates the ingredients list and estimated budget, and generates online purchasing links for each ingredient.

[2066] Step 5:

[2067] The user confirms the order details using the terminal and presses the confirm button. Once the order details are confirmed, the server sends an order request to the partner company, and the order is officially confirmed.

[2068] Inputs: Ingredient list, estimated budget, online purchasing links

[2069] Output: Order request to partner

[2070] Specific operation: When the user confirms the order details on the terminal and presses the confirmation button, the server sends the order request to the partner company.

[2071] Step 6:

[2072] After the order is confirmed, the server arranges for delivery and notifies the user of the delivery request. After receiving confirmation from the partner company, the server requests delivery from the delivery company and notifies the user of the estimated delivery time.

[2073] Input: Order confirmation information

[2074] Output: Notification of delivery request details

[2075] Specific operation: The server sends a delivery request to the delivery company and provides notification information to the user.

[2076] Step 7:

[2077] The user inputs the ingredient information they have stored into the server from their device. The ingredient information can be input as an image, voice, or text. The user's seasoning preferences are also input at the same time.

[2078] Input: User's food information, seasoning preference

[2079] Output: Ingredient information and desired seasoning data sent to the server

[2080] Specific operation: The user takes a photo of the ingredients in the refrigerator using the device and enters their desired seasoning.

[2081] Step 8:

[2082] The server analyzes the inputted ingredient information and seasoning preferences and proposes appropriate recipes. The server uses a generative AI model to analyze the inputted data and generate recipes that match the user's preferences.

[2083] Input: Ingredient information, desired seasoning

[2084] Output: Suggested recipe

[2085] How it works: The generative AI model analyzes ingredient information and seasoning preferences to generate an appropriate recipe.

[2086] Step 9:

[2087] While the user is cooking, the cooking status is sent to the server via camera or voice input from the device, and the server analyzes the received cooking status and provides appropriate advice in real time.

[2088] Input: Cooking status data (images, audio)

[2089] Output: Real-time cooking advice

[2090] Specific operation: While cooking, the user takes a photo or voice-over of the situation on the device, and the server analyzes it and advises the next step.

[2091] In this way, at each processing step, the user, server, and terminal work together to input, analyze, and output data, enabling users to easily create healthy menus and efficiently procure the ingredients needed to make them.

[2092] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[2093] The present invention combines a system that allows users to easily create personalized menus and efficiently source the necessary ingredients with an emotion engine. This system analyzes the user's emotions and provides recipes and cooking advice accordingly, creating a more personalized experience. Below, we will explain in detail how to implement this system.

[2094] System configuration

[2095] This system operates by combining user devices, a server, collaboration with partner companies, and an emotion engine. The user device accepts user input, communicates with the server, and acquires emotion data through the emotion engine. The server analyzes the information received from the user and is responsible for generating menus, suggesting recipes, and managing ingredients. Partner companies also provide and deliver the necessary ingredients.

[2096] Creating a menu and incorporating emotions

[2097] 1. Enter basic user information

[2098] Users input their health condition, allergies, disliked ingredients, family composition, desired budget, preparation period, and desired menu direction. In addition, the user can also input their current emotional state. This information is sent to the server from the user's device.

[2099] Example: "Health condition: good, Allergies: peanuts, Dislikes: tomatoes, Family size: 4 people, Desired budget: 5,000 yen per week, Menu preference: healthy, Emotional state: tired"

[2100] 2. Data analysis and menu generation by the server

[2101] The server receives the basic information and emotional state sent, and obtains price information for the user's area via an API. Based on this information, it analyzes and generates a menu for the specified period. This also includes menu suggestions to soothe the user's emotions based on their emotional state.

[2102] For example, if you are tired, a simple yet balanced meal will be suggested: "Breakfast: Oatmeal, Lunch: Simple Salad, Dinner: Stir-fried Chicken and Vegetables."

[2103] 3. Present the ingredients list and estimated budget

[2104] The server calculates the necessary ingredients and estimated budget based on the generated menu, and sends them to the user's device, allowing the user to understand the ingredients they need to purchase and their budget.

[2105] Example: Based on the estimated budget calculated by the server, the user is presented with a list of ingredients needed: oatmeal, lettuce, chicken, carrots, along with the total budget.

[2106] Recipe suggestions and cooking assistance

[2107] 1. Enter ingredient information

[2108] Users can input information about the ingredients in their refrigerator using images, voice, or text, as well as their desired seasoning and current emotional state.

[2109] Example: Take a photo of the ingredients in the refrigerator and enter the text "I want it to have a Japanese flavor" and "Emotional state: Calm."

[2110] 2. Analysis of food information and emotional state by the server

[2111] The server analyzes the transmitted ingredient information and emotional state, and suggests dishes that can be made based on the available ingredients and desired seasonings. It also suggests recipes that will help you relax depending on your emotional state.

[2112] Example: Based on the ingredients identified through photo analysis: chicken, cabbage, and carrots, the system suggests "teriyaki chicken."

[2113] 3. Real-time cooking advice

[2114] While cooking, users can use their device to send cooking status information to the server using a camera or voice input. The server then analyzes the received cooking status and the user's emotional state and provides appropriate advice in real time.

[2115] For example, if you say, "The chicken is browned," the server will advise you, "Next, add the sauce and cook over medium heat for 5 minutes," and also send a relaxing message depending on your emotional state.

[2116] Food ordering and delivery

[2117] 1. List and order ingredients

[2118] The server automatically generates a list of ingredients required based on the menu, which then becomes the basis for ordering from partne...

Claims

1. A means of entering basic user information; A means of obtaining price information for the area in which you live; A means for analyzing the input basic information and the acquired price information to generate an optimal menu; A means for generating a list of ingredients required and a projected budget based on the generated menu; A means of presenting the user with an ingredient list and a projected budget; A system including:

2. A means for users to input information about ingredients they have, A method to suggest dishes that can be made by analyzing the inputted ingredient information and seasoning preferences, A means for analyzing the cooking status during cooking and providing cooking advice in real time; The system of claim 1 further comprising:

3. A means for listing ingredients required based on the generated menu; A way to order the listed ingredients from partner companies, A way to set up a delivery request after placing an order, A means for notifying the user of the delivery request; The system of claim 1 further comprising:

Images