system

Abstract

We provide the system. [Solution] A means for inputting information on multiple food items present in the refrigerator, A means for inputting the expiration date of the food ingredient, A means of searching a database for multiple recipes using the ingredient in question based on the information entered, A means for determining priority based on the expiration date of the ingredients and sorting each recipe according to priority, A means of suggesting the highest priority recipe to the user and displaying the instructions for making that recipe, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In modern busy lives, there is a problem that it is difficult to effectively use the ingredients in the refrigerator. Due to insufficient management of ingredients, ingredients are often wasted, and it is often difficult to select appropriate dishes. There is also a need to reduce food loss by preferentially using ingredients whose expiration date is approaching. Thus, there is a need for a system that maximally utilizes the potential of ingredients and helps plan daily meals.

Means for Solving the Problems

[0005] This invention is a system that takes information about multiple food items in a refrigerator and their expiration dates as input and suggests the most suitable recipe based on this information. Specifically, the user inputs the food items in the refrigerator and their expiration dates via a terminal and sends this data to a server. The server analyzes this data and searches a database for multiple recipes using the food items. Next, it determines the priority of the recipes based on the expiration date of each food item and finally suggests the highest priority recipe to the user. Furthermore, by selecting the most suitable recipe based on the user's past selection history and preferences, user satisfaction is improved. In this way, it becomes possible to use the food items in the refrigerator effectively and efficiently and reduce food waste.

[0006] "Ingredients" refers to the materials used for cooking or eating food.

[0007] "Best before date" refers to the date that indicates the period during which food can be safely consumed while maintaining its quality.

[0008] "Input" refers to the act of providing data to a system or device.

[0009] A "database" refers to a collection of data organized for the efficient management, retrieval, and acquisition of specific data.

[0010] A "server" is a dedicated computer system that provides services to other computers and devices over a network.

[0011] A "terminal" refers to a computer or device that a user uses to input data or interact with a system.

[0012] "Priority" refers to the order in which certain items are processed or executed with higher priority than others.

[0013] A "recipe" refers to a document that lists the steps and ingredients needed to prepare a dish.

[0014] "Proposal" refers to the act of presenting specific options or solutions.

[0015] "User" refers to a person who operates using a system or device.

[0016] "Recipe" refers to the specific steps or methods for completing a dish.

[0017] "Food loss" refers to food that is discarded but could be eaten.

Brief Explanation of Drawings

[0018] [Figure 1] It is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] It is a conceptual diagram showing an example of the main functions of a data processing device and a smart device according to the first embodiment. [Figure 3] It is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] It is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which multiple emotions are mapped. [Figure 10] It shows an emotion map to which multiple emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12]It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Embodiment 2 when combined with an emotion engine. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when combined with an emotion engine.

Mode for Carrying Out the Invention

[0019] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0020] First, the language used in the following description will be explained.

[0021] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be one arithmetic unit or a combination of a plurality of arithmetic units. Also, the processor may be one type of arithmetic unit or a combination of a plurality of types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0022] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0023] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0024] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0025] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0026] [First Embodiment]

[0027] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0028] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0029] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0030] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0031] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0032] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0033] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0034] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0035] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0036] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0037] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0038] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0039] This invention relates to a system for managing leftover ingredients in a refrigerator and their expiration dates, and for suggesting optimal menus and their preparation methods. The following describes specific embodiments for carrying out the invention.

[0040] overview

[0041] This system allows users to input each ingredient in their refrigerator and its expiration date, and the server then searches for and suggests the most suitable recipe based on that data. Users can then cook based on these suggestions, efficiently utilizing the ingredients in their refrigerator.

[0042] System Configuration

[0043] 1. User terminal:

[0044] The user terminal is a device used to input information about the food items in the refrigerator and their expiration dates. This could be a smartphone, tablet, or PC.

[0045] 2. Server:

[0046] The server is the central component that receives ingredient information sent from the user's terminal and searches for and suggests appropriate recipes based on that information. The server accesses the database to retrieve appropriate recipes and provides data according to the user's request.

[0047] 3. Database:

[0048] The database is a system for storing ingredients, recipes, and related information. It manages details of each recipe, the required ingredients, and their corresponding expiration dates.

[0049] Specific processing flow

[0050] 1. Data entry phase:

[0051] Users input information about the food items in their refrigerator and their expiration dates using a smartphone or other device. For example, they might enter "Pork, expiration date 2023-10-20" and "Cabbage, expiration date 2023-10-18" into the app's input form.

[0052] The terminal converts the input data into JSON format and sends it to the server as an HTTP POST request.

[0053] 2. Data Processing Phase:

[0054] The server receives and parses data sent by the user. For example, it receives data in the format {"materials": [{"name": "Pork", "expiry_date": "2023-10-20"}, {"name": "Cabbage", "expiry_date": "2023-10-18"}]}.

[0055] The server searches the database for multiple recipes corresponding to the ingredients. For example, it retrieves relevant recipes such as "stir-fried vegetables" or "curry" from the database.

[0056] The server prioritizes recipes based on the expiration dates of the ingredients. Recipes that use ingredients with shorter expiration dates are placed higher in the list.

[0057] The server takes into account the user's past selection history and preferences to ultimately select the most suitable recipe.

[0058] 3. Data display phase:

[0059] The server generates detailed information about the selected optimal recipe in JSON format and sends it to the terminal. For example, it sends data in the format {"menu": {"name": "Stir-fried pork and cabbage", "instructions": "Chop the ingredients and stir-fry them in a frying pan"}}.

[0060] The terminal analyzes the received data and displays a suggested menu and its preparation method to the user. It displays "Today's suggested menu: Stir-fried pork and cabbage, Preparation method: Cut the ingredients and stir-fry them in a frying pan."

[0061] The user reviews the displayed information and prepares the meal based on the suggested menu.

[0062] Specific example

[0063] For example, if a user inputs that they have "pork," "cabbage," and "carrots" in their refrigerator, the server searches its database for recipes using these ingredients and prioritizes them based on their expiration dates. As a result, it provides the user with a suggested menu item, "stir-fried pork and cabbage," along with detailed instructions on how to make it. By following these instructions, the user can efficiently utilize the ingredients in their refrigerator.

[0064] This system enables effective management and efficient use of food in the refrigerator, significantly reducing food waste. Furthermore, it reduces the effort required for daily cooking, allowing users to achieve a healthier and more planned diet.

[0065] The following describes the processing flow.

[0066] Step 1:

[0067] The user collects information on multiple food items in their refrigerator, along with their expiration dates. Using a smartphone or other device, the user enters information such as "Pork, expiration date 2023-10-20" and "Cabbage, expiration date 2023-10-18" into the application's form.

[0068] Step 2:

[0069] The terminal converts the entered ingredient information into JSON format data. For example, it will be in the format {"materials": [{"name": "Pork", "expiry_date": "2023-10-20"}, {"name": "Cabbage", "expiry_date": "2023-10-18"}]}.

[0070] Step 3:

[0071] The terminal sends data to the server. The transmission method uses an HTTP POST request. For example, the request body contains the JSON data mentioned earlier.

[0072] Step 4:

[0073] The server receives data sent from the terminal. It parses the HTTP request, obtains JSON data, and then parses it. For example, it might obtain information about each ingredient in array format as the parsed data.

[0074] Step 5:

[0075] The server accesses the database and searches for relevant recipes based on the received ingredient information. The database stores detailed information and lists of required ingredients for each recipe. For example, it retrieves recipes such as "stir-fried vegetables" and "curry" that include "pork" and "cabbage."

[0076] Step 6:

[0077] The server prioritizes the retrieved recipes based on the expiration dates of each ingredient. For example, it compares the expiration dates of ingredients and sorts recipes prioritizing those with ingredients that need to be consumed the fastest.

[0078] Step 7:

[0079] The server selects the most suitable recipe by considering the user's past selection history and preferences. This enables personalized recipe suggestions for each user. For example, if "stir-fried vegetables" is determined to be the most suitable, that recipe will be selected.

[0080] Step 8:

[0081] The server generates detailed information about the selected recipe in JSON format. For example, it creates data in the format {"menu": {"name": "Stir-fried pork and cabbage", "instructions": "Chop the ingredients and stir-fry them in a frying pan"}}.

[0082] Step 9:

[0083] The server sends the generated JSON data to the terminal as an HTTP response. The response contains the selected recipe and its details.

[0084] Step 10:

[0085] The terminal analyzes the response data received from the server. It parses the JSON data to obtain the suggested menu and how to make it.

[0086] Step 11:

[0087] The device displays suggested menus and instructions for preparation to the user based on the analyzed data. For example, it might display something like, "Today's suggested menu: Stir-fried pork and cabbage, Instructions: Cut the ingredients and stir-fry them in a pan."

[0088] Step 12:

[0089] The user reviews the displayed information and prepares the dish based on the suggested menu. For example, they might cut pork and cabbage, then stir-fry them in a pan to complete the dish.

[0090] (Example 1)

[0091] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0092] Because the management and efficient use of ingredients in refrigerators are unclear, there is a need for a system that eliminates food waste and suggests the optimal cooking method based on the expiration date of the ingredients. Furthermore, conventional systems have not been able to provide suggestions that take into account the user's past selection history and preferences.

[0093] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0094] In this invention, the server includes means for inputting ingredient information, means for inputting expiration date information of the ingredient, means for retrieving multiple cooking methods using the ingredient from a data storage means based on the input information, means for determining priority based on the expiration date information of the ingredient and arranging each cooking method according to priority, and means for proposing the highest priority cooking method to the user and displaying how to prepare the cooking method. This makes it possible to efficiently utilize ingredients in the refrigerator, reduce food waste, and propose appropriate cooking methods that take into account the user's preferences.

[0095] "Means for inputting food information" refers to a device or interface for users to input the types and names of food items inside a refrigerator.

[0096] "Means for inputting expiration date information" refers to a device or interface for users to input the best-before date or expiration date for each food item.

[0097] "Data storage means" refers to a database or storage device for storing and managing information on ingredients, expiration dates, and multiple cooking methods based on these.

[0098] "Means for determining and arranging priorities" refers to a device or program that has the function of setting the priority of each cooking method based on the input expiration date information and listing them in that order.

[0099] "Means for proposing and displaying cooking methods" refers to a device or interface that presents the user with the determined optimal cooking method and displays the detailed preparation procedure.

[0100] "Communication means" refers to a network or communication interface used to transmit data input from a terminal to the main control unit.

[0101] The "main control unit" is a central control unit or server that receives and analyzes data and selects the optimal cooking method.

[0102] "Means of selecting based on the user's past selection history and preferences" refers to a device or program that has the function of recording the cooking methods and preferences previously chosen by the user and selecting the optimal cooking method taking that into consideration.

[0103] This invention is a system that manages information about food items in a refrigerator and their expiration dates, and suggests the optimal cooking method. The following describes specific embodiments for carrying out this invention.

[0104] This system allows users to input each ingredient in their refrigerator and its expiration date, and the server then searches for and suggests the most suitable cooking method based on that data. This enables users to efficiently utilize the ingredients in their refrigerator and reduce food waste.

[0105] System Configuration

[0106] 1. User terminal:

[0107] This is a device used by users to input food ingredient information and expiration date information. This includes smartphones, tablets, or PCs. Users launch a dedicated app and enter data through input forms such as text boxes.

[0108] 2. Server:

[0109] The server is the central component that receives data sent from the user terminal and searches for and suggests appropriate cooking methods based on that data. The server accesses the database, retrieves the relevant cooking methods, and provides data according to the user's request.

[0110] 3. Database:

[0111] The database is a system for storing ingredients, cooking methods, and related information. It manages details of each cooking method, required ingredients, and their corresponding expiration dates.

[0112] Specific example

[0113] For example, if a user inputs that they have "pork," "cabbage," and "carrots" in their refrigerator, the server searches its database for cooking methods using these ingredients and prioritizes them based on their expiration dates. As a result, it provides the user with a suggested cooking method, such as "stir-fried pork and cabbage," along with a detailed explanation of how to make it. By cooking based on this suggestion, the user can efficiently utilize the ingredients in their refrigerator.

[0114] Example of a prompt

[0115] Please enter the information about the ingredients in your refrigerator and their expiration dates. For example, enter it in the format of 'Pork, Expiration Date 2023-10-20', 'Cabbage, Expiration Date 2023-10-18', etc. We will then suggest the best cooking method.

[0116] This prompt allows users to easily operate the system and input ingredient and expiration date information. This enables the system to provide appropriate cooking instructions and reduce food waste.

[0117] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0118] Step 1: User enters ingredient information.

[0119] Users open their smartphones or tablets and launch the dedicated app. They then enter food information and expiration dates into the app's input form, such as "Pork, Expiration Date 2023-10-20" or "Cabbage, Expiration Date 2023-10-18". Specifically, users type the information into the text box and press the submit button to complete the input.

[0120] Input: Ingredient information (e.g., "Pork, Cabbage") and its expiration date (e.g., "2023-10-20, 2023-10-18")

[0121] Output: User-input text data

[0122] Step 2: Data conversion and transmission by the terminal

[0123] The device converts the ingredient information entered by the user into JSON format. For example, it formats it into the following format: {"materials": [{"name": "Pork", "expiry_date": "2023-10-20"}, {"name": "Cabbage", "expiry_date": "2023-10-18"}]}. Next, it sends this data to the server as an HTTP POST request. Specifically, the device sends the data to the API endpoint over the internet.

[0124] Input: User-inputted text data

[0125] Output: Data in JSON format and an HTTP POST request

[0126] Step 3: Server receives and analyzes data.

[0127] The server receives an HTTP request from the terminal and extracts and parses the data sent from the payload. Specifically, it parses JSON data in the format {"materials": [{"name": "Pork", "expiry_date": "2023-10-20"}, {"name": "Cabbage", "expiry_date": "2023-10-18"}]} to obtain the name of each ingredient and its expiration date.

[0128] Input: Data in JSON format

[0129] Output: Information on the names of ingredients and their corresponding expiration dates.

[0130] Step 4: Search for cooking methods using the server

[0131] The server uses the analyzed ingredient names to query the database and search for corresponding cooking methods. For example, to retrieve "recipes using pork and cabbage" from the database, it sends an SQL query. Specifically, it executes the query and returns a list of multiple cooking methods.

[0132] Input: Name of ingredients (e.g., "Pork, cabbage")

[0133] Output: List of suggested cooking methods

[0134] Step 5: Server prioritizes cooking method

[0135] The server evaluates the ingredient list and expiration dates for each cooking method and sets priorities. Specifically, it prioritizes cooking methods that use ingredients with shorter expiration dates. Based on the calculated priorities, it sorts the list of cooking methods.

[0136] Input: A list of suggested cooking methods and information on ingredients and their expiration dates.

[0137] Output: List of cooking methods sorted by priority

[0138] Step 6: Server selects the optimal cooking method

[0139] The server selects the optimal cooking method from among the high-priority options, taking into account the user's past selection history and preferences. For example, if the user previously preferred "stir-fry," the server will select "stir-fried pork and cabbage."

[0140] Input: A list of cooking methods sorted by priority, user selection history, and preference data.

[0141] Output: Optimal cooking method

[0142] Step 7: Server generates and transmits recipe information.

[0143] The server generates detailed information in JSON format about the selected optimal cooking method. For example, it might generate it in the format {"menu": {"name": "Stir-fried pork and cabbage", "instructions": "Cut the ingredients and stir-fry them in a frying pan"}} and send this to the terminal as an HTTP response.

[0144] Input: Optimal cooking method

[0145] Output: Detailed cooking instructions (in JSON format) and HTTP response

[0146] Step 8: Displaying the recipe on the device

[0147] The device parses the JSON data received from the server and displays the suggested cooking method and instructions to the user. Specifically, it displays "Today's suggested menu: Stir-fried pork and cabbage, Instructions: Cut the ingredients and stir-fry in a frying pan."

[0148] Input: Detailed cooking instructions (JSON format)

[0149] Output: Display of cooking instructions and preparation methods to the user.

[0150] Step 9: User performs cooking

[0151] The user checks the displayed cooking method and instructions, and then prepares the dish based on the ingredients in their refrigerator. Specifically, they prepare the ingredients according to the provided steps and complete the dish.

[0152] Input: Detailed cooking instructions and ingredients in the refrigerator

[0153] Output: Finished dish

[0154] (Application Example 1)

[0155] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0156] In modern times, there is a need to efficiently manage the food in refrigerators and reduce the amount of food that goes past its expiration date. However, it is difficult for users to keep track of the food in their refrigerators and use it appropriately in their daily lives, and food waste often occurs. Also, if necessary ingredients are missing, people have to go shopping each time, which is time-consuming and troublesome. To solve this problem, a system is needed that efficiently manages information about the food in the refrigerator and has an automatic ordering function for missing ingredients.

[0157] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0158] In this invention, the server includes means for inputting information on multiple food items present in the refrigerator, means for inputting the expiration dates of the food items, means for searching a database for multiple recipes using the food items based on the input information, means for determining priority based on the expiration dates of the food items and sorting each recipe according to priority, means for suggesting the highest-priority recipe to the user and displaying the instructions for making that recipe, and means for automatically ordering any missing ingredients based on the information on the food items. This enables centralized management of food item information in the refrigerator, allowing for optimal recipe suggestions to the user and automatic ordering of necessary ingredients.

[0159] "Means for inputting information on multiple food items present in a refrigerator" refers to devices or applications that allow users to input information such as the names, quantities, and locations of various food items stored in a refrigerator.

[0160] "Means for inputting the expiration date of the food item" refers to a device or application for the user to input the expiration date of each food item in the refrigerator.

[0161] "Means for searching a database for multiple recipes using the ingredients in question based on the input information" refers to a server or software that searches a database for multiple recipes using the ingredients entered by the user.

[0162] "Means for determining priority based on the expiration date of the ingredient and sorting each recipe according to priority" refers to a server or algorithm that determines whether an ingredient should be used preferentially based on the expiration date information of the input ingredient, and then sorts the corresponding recipes in order of priority.

[0163] "A means of suggesting a top-priority recipe to a user and displaying how to make that recipe" refers to a device or application that presents the user with the highest priority recipe and displays the user's detailed instructions for making that recipe.

[0164] "Means for automatically ordering missing ingredients based on information about the ingredients in question" refers to a server or software that automatically determines the missing ingredients needed for a dish based on ingredient information entered by the user and orders them online.

[0165] "A means of communication for sending data entered from a terminal to a server" refers to a communication interface, protocol, or system for sending information about ingredients and expiration dates from a terminal used by a user to a server.

[0166] "A means of selecting the optimal recipe from multiple recipes based on the user's past selection history and preferences" refers to an algorithm or software that analyzes the user's past selection history and preference information and suggests the most suitable recipe based on that information.

[0167] This invention relates to a system that manages the remaining ingredients in a refrigerator and their expiration dates, suggests the optimal menu and how to prepare it, and can automatically order any missing ingredients from an online shopping site.

[0168] System Configuration

[0169] 1. Method for entering information about the ingredients in the refrigerator:

[0170] Users use devices such as smartphones or tablets to input information about the food items in their refrigerator and their expiration dates. For example, they might enter "Pork, expiration date 2023-10-20" and "Cabbage, expiration date 2023-10-18" into the app's input form.

[0171] 2. Means of communication:

[0172] The aforementioned terminal sends the input data to the server via HTTP communication. It uses JSON format data for communication and sends an HTTP POST request.

[0173] 3. Means of searching for recipes based on input information:

[0174] The server analyzes the received data and searches the database for a suitable recipe based on the information about the ingredients in the refrigerator. For example, the server analyzes the received JSON data and retrieves the corresponding recipe from the database.

[0175] 4. Means for determining and sorting recipe priority:

[0176] The server prioritizes ingredients that should be consumed as soon as possible, based on their expiration dates. It then sorts the searched recipes according to this priority.

[0177] 5. Means for suggesting the highest priority recipe to the user and displaying the instructions:

[0178] The server selects the highest-priority recipe from the sorted list and sends its detailed instructions to the terminal. For example, the selected recipe for "Stir-fried Pork and Cabbage" and its instructions will be displayed.

[0179] 6. A means of automatically ordering missing materials:

[0180] The server matches the entered ingredient information with the suggested recipe and automatically orders any necessary ingredients that are not in the refrigerator from an online shopping site. For example, if the customer only has pork and cabbage and does not have carrots, which are needed for "stir-fried pork and cabbage," the server will automatically add carrots to the shopping cart from the online store.

[0181] Program Processing Description

[0182] The server uses a Python program to suggest recipes based on user input and automatically order any missing ingredients. The server uses the Python Requests library for communication. SQL is used to search for recipes in the database. Algorithms are implemented for prioritizing and sorting recipes. Furthermore, a machine learning model is used to select the optimal recipe based on the user's past selection history and preferences.

[0183] For example, if a user enters "pork, cabbage" along with expiration date information through a smartphone app, that information is sent to the server. The server then searches for a suitable recipe and suggests "stir-fried pork and cabbage," prioritizing ingredients with the earliest expiration dates. If carrots are missing in the recipe, they are automatically ordered from an online shopping site.

[0184] Example of a prompt

[0185] Please enter the ingredients in your refrigerator (e.g., "pork," "cabbage") and their expiration dates. Based on this information, we will automatically order the necessary ingredients from an online shopping site and suggest the best recipes.

[0186] As a result, this invention allows for efficient management of ingredients in the refrigerator, reduces food waste, and makes meal preparation easier for the user.

[0187] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0188] Step 1:

[0189] Users input information about the food items in their refrigerator and their expiration dates using a smartphone, tablet, or other device.

[0190] Specific examples of the information to be entered include "Pork, Expiration Date 2023-10-20" and "Cabbage, Expiration Date 2023-10-18". This will be the input data.

[0191] Step 2:

[0192] The terminal converts the entered ingredient information into JSON format and sends it to the server as an HTTP POST request.

[0193] This communication method utilizes the JSON format and the HTTP protocol. The data is sent in the format {"materials": [{"name": "Pork", "expiry_date": "2023-10-20"}, {"name": "Cabbage", "expiry_date": "2023-10-18"}]}. This serves as the input data, and the server performs the analysis.

[0194] Step 3:

[0195] The server receives an HTTP POST request and parses the received data.

[0196] The analyzed data extracts the expiration dates of each food item in the refrigerator. Based on the input data, a query is generated for the database to search for the corresponding recipe. For example, it searches for recipes using "pork" and "cabbage".

[0197] Step 4:

[0198] The server receives multiple recipes as search results from the database, determines a priority based on the expiration dates of the ingredients in each recipe, and sorts the recipes.

[0199] This sorting is performed using an algorithm that prioritizes recipes that use ingredients nearing their expiration date. For example, "Stir-fried pork and cabbage" would be one such recipe.

[0200] Step 5:

[0201] The server selects the most suitable recipe from several options based on the user's past selection history and preferences.

[0202] This process utilizes machine learning models. Based on past selection history data and preference data, it selects the most suitable recipe and prioritizes it to determine the best option.

[0203] Step 6:

[0204] The server generates detailed information about the selected top-priority recipe in JSON format and sends it to the terminal.

[0205] For example, send the following in the format: {"menu": {"name": "Stir-fried pork and cabbage", "instructions": "Cut the ingredients and stir-fry them in a frying pan"}}. This will be the output data.

[0206] Step 7:

[0207] The device analyzes the received recipe information and displays suggested menus and their preparation methods to the user.

[0208] Specifically, it displays: "Today's suggested menu: Stir-fried pork and cabbage, Instructions: Cut the ingredients and stir-fry them in a frying pan."

[0209] Step 8:

[0210] The server automatically orders any missing ingredients from online shopping sites based on the suggested recipe.

[0211] The server generates a list of all the ingredients needed for the suggested recipe and determines which ingredients are missing from the refrigerator. It then automatically orders the missing ingredients through the online shopping site's API. For example, it adds "carrots" to the online shopping site's shopping cart.

[0212] Step 9:

[0213] The user prepares the meal according to the suggested menu and instructions. They also confirm that any missing ingredients are automatically ordered from an online store, and the purchasing process proceeds. Thus, the user's meal preparation is complete.

[0214] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0215] This invention combines an emotional engine with a system that manages remaining ingredients in a refrigerator and their expiration dates, and suggests the optimal menu and preparation method. The following describes specific embodiments for carrying out the invention.

[0216] overview

[0217] This system allows users to input each ingredient in their refrigerator and its expiration date, and the server then searches for and suggests the most suitable recipe based on that data. Furthermore, it can use an emotion engine to recognize the user's emotional state and provide recipes tailored to that state.

[0218] System Configuration

[0219] 1. User terminal:

[0220] The user terminal is a device used to input information about the food items in the refrigerator and their expiration dates. It also includes voice and image analysis functions to input the user's emotional state. Examples include smartphones, tablets, and PCs.

[0221] 2. Server:

[0222] The server is the central component that receives ingredient information and emotional states sent from the user's terminal and searches for and suggests appropriate recipes based on that information. The server accesses the database to retrieve appropriate recipes and provides data according to the user's request.

[0223] 3. Database:

[0224] The database is a system for storing ingredients, recipes, and related information. It manages details of each recipe, required ingredients and their corresponding expiration dates, as well as data on the user's past selection history and emotional state.

[0225] 4. Emotional Engine:

[0226] The emotion engine is a system for recognizing a user's emotional state. It includes voice analysis and image analysis methods to analyze the user's emotions from their voice and facial expressions and transmit that data to a server.

[0227] Specific processing flow

[0228] Data entry phase

[0229] 1. The user enters information about the food items in their refrigerator and their expiration dates using a smartphone or other device. For example, they might enter information such as "Pork, expiration date 2023-10-20" or "Cabbage, expiration date 2023-10-18" into the application's input form.

[0230] 2. The user inputs their emotional state through the emotion engine. The emotional state is analyzed by capturing facial expressions using voice input or a camera.

[0231] Data transmission phase

[0232] 3. The terminal converts the entered food information and emotional state into JSON data. For example, {"materials": [{"name": "Pork", "expiry_date": "2023-10-20"}, {"name": "Cabbage", "expiry_date": "2023-10-18"}], "emotion": {"mood": "happy"}}.

[0233] 4. The terminal sends data to the server. The transmission method is an HTTP POST request.

[0234] Data Processing Phase

[0235] 5. The server receives data sent from the terminal. It parses the HTTP request, obtains the JSON data, and then parses it.

[0236] 6. The server accesses the database and searches for a suitable recipe based on the received ingredient information and emotional state. For example, it retrieves recipes such as "stir-fried vegetables" or "curry" that contain "pork" and "cabbage" from the database.

[0237] 7. The server determines the priority of the retrieved recipes based on the expiration date of each ingredient. For example, it compares the expiration dates of the ingredients and sorts the recipes with ingredients that need to be consumed the fastest to the top.

[0238] 8. The server selects the optimal recipe by considering the user's past selection history and preferences, as well as their emotional state recognized by the emotion engine. For example, it will suggest more enjoyable dishes for a "happy" emotional state.

[0239] Proposed menu generation phase

[0240] 9. The server generates detailed information about the selected recipe in JSON format. For example, it creates data in the format {"menu": {"name": "Stir-fried pork and cabbage", "instructions": "Chop the ingredients and stir-fry them in a pan"}}.

[0241] 10. The server sends the generated JSON data to the terminal as an HTTP response. The response contains the selected recipe and its details.

[0242] Data display phase

[0243] 11. The terminal analyzes the response data received from the server. It parses the JSON data to obtain the suggested menu and how to make it.

[0244] 12. The terminal displays suggested menus and instructions for preparation to the user based on the analyzed data. For example, it might display "Today's suggested menu: Stir-fried pork and cabbage, Instructions: Cut the ingredients and stir-fry them in a pan."

[0245] 13. The user reviews the displayed information and prepares the dish based on the suggested menu. For example, they might cut pork and cabbage, then stir-fry them in a pan to complete the dish.

[0246] Specific example

[0247] For example, if a user inputs that they have "pork" and "cabbage" in their refrigerator, and the emotion engine recognizes a "happy" emotional state, the server searches its database for recipes using these ingredients, prioritizing them based on their expiration dates. As a result, it provides the user with a suggested menu item, "stir-fried pork and cabbage," which is appropriate for their happy emotional state, and explains how to make it in detail. By following these instructions, the user can efficiently utilize the ingredients in their refrigerator.

[0248] This system not only allows for the effective management and use of ingredients in the refrigerator, but also enables personalized recipe suggestions tailored to the user's emotional state. This makes the daily cooking experience more enjoyable and satisfying, contributing to a reduction in food waste.

[0249] The following describes the processing flow.

[0250] Step 1:

[0251] The user collects information on multiple food items in their refrigerator, along with their expiration dates. Using a smartphone or other device, the user enters information such as "Pork, expiration date 2023-10-20" and "Cabbage, expiration date 2023-10-18" into the application's form.

[0252] Step 2:

[0253] Users record their emotional state through an emotion engine. For example, their emotional state is analyzed by capturing facial expressions using voice input or a camera.

[0254] Step 3:

[0255] The terminal converts the entered food information and emotional state into JSON data. For example, the format will be {"materials": [{"name": "Pork", "expiry_date": "2023-10-20"}, {"name": "Cabbage", "expiry_date": "2023-10-18"}], "emotion": {"mood": "happy"}}.

[0256] Step 4:

[0257] The terminal sends the converted data to the server. The transmission method uses an HTTP POST request. For example, the request body contains the JSON data mentioned earlier.

[0258] Step 5:

[0259] The server receives data sent from the terminal. The server parses the HTTP request, obtains JSON data, and parses it. For example, it might obtain information about each ingredient in array format as the parsed data.

[0260] Step 6:

[0261] The server accesses the database and searches for relevant recipes based on the received ingredient information and emotional state. The database stores detailed information and required ingredient lists for each recipe. For example, it retrieves recipes such as "stir-fried vegetables" and "curry" that contain "pork" and "cabbage."

[0262] Step 7:

[0263] The server prioritizes the retrieved recipes based on the expiration dates of each ingredient. For example, it compares the expiration dates of ingredients and sorts recipes prioritizing those with ingredients that need to be consumed the fastest.

[0264] Step 8:

[0265] The server selects the optimal recipe by considering the user's past selection history and preferences, as well as their emotional state recognized by the emotion engine. For example, it suggests more enjoyable dishes for a "happy" emotional state.

[0266] Step 9:

[0267] The server generates detailed information about the selected recipe in JSON format. For example, it creates data in the format {"menu": {"name": "Stir-fried pork and cabbage", "instructions": "Chop the ingredients and stir-fry them in a frying pan"}}.

[0268] Step 10:

[0269] The server sends the generated JSON data to the terminal as an HTTP response. The response contains the selected recipe and its details.

[0270] Step 11:

[0271] The terminal analyzes the response data received from the server. It parses the JSON data to obtain the suggested menu and how to make it.

[0272] Step 12:

[0273] The device displays suggested menus and instructions for preparation to the user based on the analyzed data. For example, it might display something like, "Today's suggested menu: Stir-fried pork and cabbage, Instructions: Cut the ingredients and stir-fry them in a pan."

[0274] Step 13:

[0275] The user reviews the displayed information and prepares the dish based on the suggested menu. For example, they might cut pork and cabbage, then stir-fry them in a pan to complete the dish.

[0276] (Example 2)

[0277] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0278] Modern households store a wide variety of ingredients in their refrigerators, but efficiently utilizing these ingredients is difficult. In particular, it is hard to plan for consuming ingredients that are nearing their expiration date, leading to food waste. Furthermore, the lack of meal suggestions tailored to the user's emotional state on any given day diminishes the enjoyment and satisfaction of daily cooking. Therefore, there is a need for a system that can suggest optimal recipes based on ingredient information and emotional state, thereby reducing food waste and providing a cooking experience that matches the user's emotions.

[0279] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for inputting information on multiple food items present in the refrigerator, means for inputting the expiration dates of the food items, and means for inputting the user's emotional state. This makes it possible to search a database for multiple recipes using the food items based on the input information and emotional state, determine priority based on the expiration dates of the food items, sort each recipe according to priority, select the optimal recipe considering the user's past selection history, preferences, and emotional state, propose the highest priority recipe to the user, and display how to make it. This makes it possible to reduce food waste and provide personalized cooking suggestions that are tailored to the user's emotions.

[0280] "A means of inputting information on multiple food items present in a refrigerator" refers to a device or software that allows a user to input the types of food items stored in the refrigerator and their respective expiration dates into a terminal.

[0281] "Means for inputting the expiration date of the food item" refers to a device or software that allows the user to input the expiration date of each food item into a terminal.

[0282] "Means for inputting the user's emotional state" refers to devices or software that analyze the user's voice and facial expressions to recognize their emotional state and input that information into the system.

[0283] The "means for retrieving a plurality of recipes using the food ingredient from a database based on the input information and emotional state" is an algorithm or software for retrieving relevant recipes from within the database using the data on the food ingredient information and emotional state received from the terminal.

[0284] The "means for determining a priority based on the expiration date of the food ingredient and sorting each recipe according to the priority" is an algorithm or software for determining the priority order of recipes based on the expiration date of the input food ingredient and rearranging the recipes based thereon.

[0285] The "means for selecting an optimal recipe in consideration of the user's past selection history, preferences, and emotional state" is an algorithm or software for judging and selecting an optimal recipe for the user using the data on the user's past recipe selection history and emotional state.

[0286] The "means for proposing the most prioritized recipe to the user and displaying the cooking method of the recipe" is a device or software for notifying the user of the recipe with a high priority and displaying the detailed cooking method thereof.

[0287] The "communication means for transmitting data input from the terminal to the server" is a network communication device or protocol for transmitting the food ingredient information and emotional state input from the user's terminal to the server.

[0288] The "emotion engine" is an algorithm or software for analyzing the user's voice and expression to identify the emotional state thereof.

[0289] The present invention relates to a system for providing an optimal recipe in consideration of the food ingredient information and expiration date in a refrigerator, and further the emotional state of the user. Hereinafter, a specific method for implementing the present invention will be described.

[0290] System Configuration

[0291] This system consists of user terminals, a server, a database, and an emotion engine. Details of each component are as follows:

[0292] User terminal

[0293] The user terminal is a device used to input information about the food items in the refrigerator and their expiration dates. It also includes voice and image analysis functions for inputting the user's emotional state. Examples include smartphones, tablets, and PCs.

[0294] server

[0295] The server is the central component that receives ingredient information and emotional states sent from the user's terminal and searches for and suggests appropriate recipes based on that information. The server accesses a database to retrieve appropriate recipes and provides data according to the user's request. Specifically, Apache® servers and Nginx servers are often used.

[0296] database

[0297] A database is a system for storing ingredients, recipes, and related information. It manages details of each recipe, required ingredients and their corresponding expiration dates, as well as data on the user's past selection history and emotional state. Specifically, relational databases such as MySQL® and PostgreSQL are used.

[0298] Emotional Engine

[0299] An emotion engine is a system for recognizing a user's emotional state. It includes voice and image analysis methods to analyze emotions from the user's voice and facial expressions and send that data to a server. Specifically, it uses tools such as OpenCV and Google's Cloud Speech-to-Text API.

[0300] Specific example

[0301] The following presents specific examples to facilitate understanding of the present invention.

[0302] For example, if a user inputs that there is "pork" and "cabbage" in the refrigerator and uses an emotion engine to recognize a "happy" emotional state, the server searches the database for recipes using these ingredients and determines priorities considering the expiration dates. As a result, a proposed menu of "pasta with pork and cabbage" adapted to the user's happy emotional state is generated. Then, the server sends the details of this recipe and the cooking method to the user terminal so that the user can start cooking.

[0303] Examples of prompt sentences

[0304] The following are specific examples of prompt sentences to be input into the generative AI model:

[0305] There is "pork" and "cabbage" in the refrigerator. Also, the current emotional state is "happy". Please propose a recipe suitable for this emotional state using these ingredients.

[0306] According to the present invention, by effectively managing the ingredients in the refrigerator and making optimal recipe proposals, it is possible to reduce food loss and realize a personalized cooking experience according to the user's emotions.

[0307] The flow of specific processing in Example 2 will be described using FIG. 13.

[0308] Step 1:

[0309] The user inputs information on the ingredients in the refrigerator and their expiration dates using a terminal such as a smartphone. For example, information such as "pork, expiration date 2023-10-20" and "cabbage, expiration date 2023-10-18" is input into the input form of the app. The input data is output as JSON-formatted data from the input form of the terminal.

[0310] Step 2:

[0311] Users input their emotional state through the emotion engine. For example, they might use the voice input function to say "I feel good today," and the emotion engine analyzes the voice and recognizes the user's emotional state as "happy." The input data is output from the emotion engine as JSON data.

[0312] Step 3:

[0313] The terminal integrates the entered food information and emotional state to generate final JSON data. For example, it formats the data as follows: {"materials": [{"name": "Pork", "expiry_date": "2023-10-20"}, {"name": "Cabbage", "expiry_date": "2023-10-18"}], "emotion": {"mood": "happy"}}. The integrated data is then sent from the terminal to the server.

[0314] Step 4:

[0315] The device sends the generated JSON data to the server using an HTTP POST request. The detailed communication protocol used is HTTP / HTTPS. The transmitted data is received by the server.

[0316] Step 5:

[0317] The server parses the received HTTP request and retrieves JSON data. Specifically, it extracts JSON data from the body of the HTTP request. The extracted data is then passed to an internal processing module.

[0318] Step 6:

[0319] The server accesses the database and searches for relevant recipes based on the received ingredient information. For example, it searches the database for recipes containing "pork" and "cabbage" using an SQL query. The search results return a list of relevant recipes.

[0320] Step 7:

[0321] The server compares the expiration dates of each ingredient in the retrieved recipes to determine priority. For example, to prioritize ingredients with approaching expiration dates, it uses the SQL ORDER BY clause to sort the recipes. A sorted list of recipes is then generated.

[0322] Step 8:

[0323] The server selects the optimal recipe by considering the user's past selection history and preferences. It also takes into account the emotional state recognized by the emotion engine. For example, it might select the best recipe from those the user has previously chosen when they were in a "happy" emotional state. One optimal recipe is then selected.

[0324] Step 9:

[0325] The server generates detailed information about the selected recipe in JSON format. For example, it creates data in the format {"menu": {"name": "Pork and cabbage pasta", "instructions": "Chop the ingredients, boil the pasta, and stir-fry in a pan"}}. The generated data is then sent to the terminal.

[0326] Step 10:

[0327] The server sends the generated JSON data to the terminal as an HTTP response. The response data is sent via the HTTP / HTTPS protocol.

[0328] Step 11:

[0329] The terminal analyzes the response data received from the server and parses the data in JSON format. For example, it analyzes data such as {"menu": {"name": "Pork and cabbage pasta", "instructions": "Cut the ingredients, boil the pasta, and stir-fry in a pan"}}. The analysis results are stored as an internal data structure.

[0330] Step 12:

[0331] The device displays suggested menus and instructions for preparation based on the analyzed data. For example, it might display: "Today's suggested menu: Pork and cabbage pasta, Instructions: Cut the ingredients, boil the pasta, and stir-fry in a pan."

[0332] Step 13:

[0333] The user cooks according to the displayed recipe. For example, they might cut pork and cabbage, boil pasta, and then stir-fry it in a pan to complete the dish.

[0334] (Application Example 2)

[0335] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0336] Traditional refrigerator management systems, while capable of managing information about ingredients and expiration dates, were unable to provide personalized recipe suggestions based on the user's emotional state. This led to decreased user satisfaction and food waste. Furthermore, the suggested recipes often didn't match the user's mood or preferences at the time, resulting in a lack of enjoyment in daily cooking.

[0337] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0338] In this invention, the server includes means for inputting information on multiple food items present in the refrigerator, means for inputting the expiration dates of the food items, means for determining priority based on the expiration dates of the food items and recognizing the user's emotional state, and means for optimizing recipes based on the emotional state. This enables the suggestion of optimal recipes according to the user's emotional state, improving the daily cooking experience and reducing food waste.

[0339] "Information on multiple food items present in the refrigerator" refers to data about various food items stored in the refrigerator, including their names, quantities, acquisition dates, and other attributes.

[0340] The "best before date" for a food item refers to the date by which the food stored in the refrigerator should be consumed appropriately, and this date varies depending on the food item.

[0341] "Means of input" refers to devices or methods for users to register information about the food in their refrigerator and its expiration date into the system, such as input devices like smartphones, tablets, or PCs.

[0342] "Methods for searching a database" refer to programs or algorithms that extract information from a database based on specific conditions and provide it to the user.

[0343] "Means for determining priority and sorting each recipe according to priority" refers to a method for determining the priority of multiple recipes based on expiration dates or other conditions, and then rearranging the recipes based on the results.

[0344] "Means for recognizing a user's emotional state" refers to devices or software that analyze a user's facial expressions, voice, etc., to evaluate their psychological state at that time.

[0345] "Methods for optimizing recipes based on emotional state" refer to methods or systems that select recipes and suggest the most suitable dishes by reflecting the recognized emotional state of the user.

[0346] "A means of suggesting the highest-priority recipe to the user and displaying how to make that recipe" refers to a function that presents the user with the most suitable recipe selected by the system and provides a detailed explanation of how to cook it.

[0347] This invention combines an emotional engine with a system that manages the remaining ingredients in a refrigerator and their expiration dates, and suggests the optimal menu and how to prepare it. The following describes specific embodiments for carrying out the invention.

[0348] System Configuration

[0349] 1. User terminal

[0350] The user terminal is a device used to input information about the food items in the refrigerator and their expiration dates. It also includes voice and image analysis functions to input the user's emotional state. Smartphones, tablets, and PCs are examples of this type of device. The terminal also has a means of communication to send the input data to the server.

[0351] 2. Server

[0352] The server is the central component that receives ingredient information and emotional states sent from the user's terminal and searches for and suggests appropriate recipes based on that information. The server accesses the database to retrieve appropriate recipes and provides data according to the user's request.

[0353] The software used includes Flask (a web server framework) and other web server frameworks.

[0354] 3. Database

[0355] The database is a system for storing ingredients, recipes, and related information. It manages details of each recipe, required ingredients and their corresponding expiration dates, as well as data on the user's past selection history and emotional state.

[0356] 4. Emotional Engine

[0357] An emotion engine is a system for recognizing a user's emotional state and includes methods for speech analysis and image analysis. Specifically, speech recognition APIs (e.g., Google Speech-to-Text API) are used for speech analysis, and libraries such as OpenCV are used for image analysis.

[0358] Specific processing flow

[0359] Users input information about the food items in their refrigerator and their expiration dates using a smartphone or other device. For example, they might enter information such as "Pork, expiration date 2023-10-20" or "Cabbage, expiration date 2023-10-18" into the application's input form. Next, they input their emotional state through an emotion engine, and their emotional state is analyzed by capturing facial expressions using voice input or a camera.

[0360] The terminal converts the entered ingredient information and emotional state into JSON data and sends it to the server. The server receives and analyzes this data, accesses the database, and searches for the corresponding recipe based on the received ingredient information and emotional state. Furthermore, it determines priorities based on the expiration date of each ingredient and selects the optimal recipe by considering the user's past selection history and preferences, as well as the emotional state recognized by the emotion engine.

[0361] Detailed information about the selected recipe is generated in JSON format and sent from the server to the terminal. The terminal parses it and displays the suggested menu and its preparation method to the user. By following these instructions, the user can efficiently utilize the ingredients in their refrigerator.

[0362] Examples of specific cases and prompt statements

[0363] Specific example

[0364] If a user inputs that they have "pork" and "cabbage" in their refrigerator and the emotion engine recognizes a "happy" emotional state, the server searches its database for recipes using these ingredients, prioritizing them based on their expiration dates. As a result, it provides the user with a suggested menu item, "Stir-fried pork and cabbage," which is appropriate for their happy emotional state, and explains how to make it in detail.

[0365] Example of a prompt

[0366] If you want to send the following data to the server, you can send data like this:

[0367] {

[0368] "materials": ["pork", "cabbage"],

[0369] "expiry_dates": ["2023-10-20", "2023-10-18"],

[0370] "image": <Image data of the user's facial expression captured by the camera>

[0371] }

[0372] In this way, users can not only effectively manage and utilize the ingredients in their refrigerator using the system, but also receive personalized recipe suggestions based on their emotional state. This makes the daily cooking experience more enjoyable and satisfying, and also contributes to reducing food waste.

[0373] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0374] Step 1:

[0375] The user enters information about the food items in their refrigerator and their expiration dates using a smartphone or other device.

[0376] Specific operation: The user launches the application and enters the name, quantity, and expiration date of the ingredients into the input form.

[0377] Input: Ingredient information (e.g., "Pork", "Cabbage") and expiration date (e.g., "2023-10-20", "2023-10-18")

[0378] Output: Input food information and expiration date data

[0379] Step 2:

[0380] Users input their emotional state through an emotion engine, and their facial expressions are captured using voice input or a camera.

[0381] Specific operation: The user uses the in-app camera function to take a picture of their face and have their emotional state recognized.

[0382] Input: User's facial expression images and audio data

[0383] Output: User's emotional state (e.g., "happy", "sad")

[0384] Step 3:

[0385] The terminal converts the entered food information and emotional state into JSON data and sends it to the server.

[0386] Specific operation: The terminal packages the ingredient information and emotional state into JSON format (for example, {"materials": ["pork", "cabbage"], "expiry_dates": ["2023-10-20", "2023-10-18"], "emotion": "happy"}) and sends it to the server using an HTTP POST request.

[0387] Input: Data in JSON format

[0388] Output: HTTP request sent to the server

[0389] Step 4:

[0390] The server receives and analyzes the data sent from the terminal.

[0391] Specific operation: The server receives an HTTP request and parses the JSON data to retrieve ingredient information, expiration date, and emotional state.

[0392] Input: JSON data sent from the device

[0393] Output: Analyzed food information, expiration date, emotional state

[0394] Step 5:

[0395] The server accesses the database and searches for the appropriate recipe based on the received ingredient information and emotional state.

[0396] Specific operation: The server executes a database query to retrieve a recipe suitable for the ingredients. The database stores recipes corresponding to the ingredients and emotional states.

[0397] Input: Analyzed food information, expiration date, emotional state

[0398] Output: List of relevant recipes

[0399] Step 6:

[0400] The server determines the priority of recipes based on the expiration date of each ingredient and sorts the recipes accordingly.

[0401] Specific operation: The server compares the expiration dates of ingredients and prioritizes recipes that include ingredients that need to be consumed the soonest.

[0402] Input: List of relevant recipes, expiration dates of ingredients

[0403] Output: High-priority recipe list

[0404] Step 7:

[0405] The server selects the optimal recipe by considering the user's past selection history, preferences, and emotional state.

[0406] Specific operation: The server retrieves the user's past selection history from the database and executes an algorithm to select the optimal recipe in conjunction with their emotional state.

[0407] Input: High-priority recipe list, user's past selection history, emotional state

[0408] Output: Optimal recipe

[0409] Step 8:

[0410] The server generates detailed information about the selected recipe in JSON format and sends it to the terminal.

[0411] Specific operation: The server formats the selected recipe and its details (e.g., ingredients, instructions) in JSON format and sends it to the terminal as an HTTP response.

[0412] Input: Optimal recipe

[0413] Output: Recipe data in JSON format

[0414] Step 9:

[0415] The terminal analyzes the response data received from the server and displays the suggested menu and its preparation method to the user.

[0416] Specific operation: The terminal parses the received JSON data and displays the recipe and cooking instructions on the screen.

[0417] Input: Recipe data in JSON format

[0418] Output: The recipe and its details displayed to the user

[0419] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0420] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0421] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0422] [Second Embodiment]

[0423] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0424] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0425] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0426] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0427] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0428] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0429] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0430] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0431] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0432] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0433] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0434] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0435] This invention relates to a system for managing leftover ingredients in a refrigerator and their expiration dates, and for suggesting optimal menus and their preparation methods. The following describes specific embodiments for carrying out the invention.

[0436] overview

[0437] This system allows users to input each ingredient in their refrigerator and its expiration date, and the server then searches for and suggests the most suitable recipe based on that data. Users can then cook based on these suggestions, efficiently utilizing the ingredients in their refrigerator.

[0438] System Configuration

[0439] 1. User terminal:

[0440] The user terminal is a device used to input information about the food items in the refrigerator and their expiration dates. This could be a smartphone, tablet, or PC.

[0441] 2. Server:

[0442] The server is the central component that receives ingredient information sent from the user's terminal and searches for and suggests appropriate recipes based on that information. The server accesses the database to retrieve appropriate recipes and provides data according to the user's request.

[0443] 3. Database:

[0444] The database is a system for storing ingredients, recipes, and related information. It manages details of each recipe, the required ingredients, and their corresponding expiration dates.

[0445] Specific processing flow

[0446] 1. Data entry phase:

[0447] Users input information about the food items in their refrigerator and their expiration dates using a smartphone or other device. For example, they might enter "Pork, expiration date 2023-10-20" and "Cabbage, expiration date 2023-10-18" into the app's input form.

[0448] The terminal converts the input data into JSON format and sends it to the server as an HTTP POST request.

[0449] 2. Data Processing Phase:

[0450] The server receives and parses data sent by the user. For example, it receives data in the format {"materials": [{"name": "Pork", "expiry_date": "2023-10-20"}, {"name": "Cabbage", "expiry_date": "2023-10-18"}]}.

[0451] The server searches the database for multiple recipes corresponding to the ingredients. For example, it retrieves relevant recipes such as "stir-fried vegetables" or "curry" from the database.

[0452] The server prioritizes recipes based on the expiration dates of the ingredients. Recipes that use ingredients with shorter expiration dates are placed higher in the list.

[0453] The server takes into account the user's past selection history and preferences to ultimately select the most suitable recipe.

[0454] 3. Data display phase:

[0455] The server generates detailed information about the selected optimal recipe in JSON format and sends it to the terminal. For example, it sends data in the format {"menu": {"name": "Stir-fried pork and cabbage", "instructions": "Chop the ingredients and stir-fry them in a frying pan"}}.

[0456] The terminal analyzes the received data and displays a suggested menu and its preparation method to the user. It displays "Today's suggested menu: Stir-fried pork and cabbage, Preparation method: Cut the ingredients and stir-fry them in a frying pan."

[0457] The user reviews the displayed information and prepares the meal based on the suggested menu.

[0458] Specific example

[0459] For example, if a user inputs that they have "pork," "cabbage," and "carrots" in their refrigerator, the server searches its database for recipes using these ingredients and prioritizes them based on their expiration dates. As a result, it provides the user with a suggested menu item, "stir-fried pork and cabbage," along with detailed instructions on how to make it. By following these instructions, the user can efficiently utilize the ingredients in their refrigerator.

[0460] This system enables effective management and efficient use of food in the refrigerator, significantly reducing food waste. Furthermore, it reduces the effort required for daily cooking, allowing users to achieve a healthier and more planned diet.

[0461] The following describes the processing flow.

[0462] Step 1:

[0463] The user collects information on multiple food items in their refrigerator, along with their expiration dates. Using a smartphone or other device, the user enters information such as "Pork, expiration date 2023-10-20" and "Cabbage, expiration date 2023-10-18" into the application's form.

[0464] Step 2:

[0465] The terminal converts the entered ingredient information into JSON format data. For example, it will be in the format {"materials": [{"name": "Pork", "expiry_date": "2023-10-20"}, {"name": "Cabbage", "expiry_date": "2023-10-18"}]}.

[0466] Step 3:

[0467] The terminal sends data to the server. The transmission method uses an HTTP POST request. For example, the request body contains the JSON data mentioned earlier.

[0468] Step 4:

[0469] The server receives data sent from the terminal. It parses the HTTP request, obtains JSON data, and then parses it. For example, it might obtain information about each ingredient in array format as the parsed data.

[0470] Step 5:

[0471] The server accesses the database and searches for relevant recipes based on the received ingredient information. The database stores detailed information and lists of required ingredients for each recipe. For example, it retrieves recipes such as "stir-fried vegetables" and "curry" that include "pork" and "cabbage."

[0472] Step 6:

[0473] The server prioritizes the retrieved recipes based on the expiration dates of each ingredient. For example, it compares the expiration dates of ingredients and sorts recipes prioritizing those with ingredients that need to be consumed the fastest.

[0474] Step 7:

[0475] The server selects the most suitable recipe by considering the user's past selection history and preferences. This enables personalized recipe suggestions for each user. For example, if "stir-fried vegetables" is determined to be the most suitable, that recipe will be selected.

[0476] Step 8:

[0477] The server generates detailed information about the selected recipe in JSON format. For example, it creates data in the format {"menu": {"name": "Stir-fried pork and cabbage", "instructions": "Chop the ingredients and stir-fry them in a frying pan"}}.

[0478] Step 9:

[0479] The server sends the generated JSON data to the terminal as an HTTP response. The response contains the selected recipe and its details.

[0480] Step 10:

[0481] The terminal analyzes the response data received from the server. It parses the JSON data to obtain the suggested menu and how to make it.

[0482] Step 11:

[0483] The device displays suggested menus and instructions for preparation to the user based on the analyzed data. For example, it might display something like, "Today's suggested menu: Stir-fried pork and cabbage, Instructions: Cut the ingredients and stir-fry them in a pan."

[0484] Step 12:

[0485] The user reviews the displayed information and prepares the dish based on the suggested menu. For example, they might cut pork and cabbage, then stir-fry them in a pan to complete the dish.

[0486] (Example 1)

[0487] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0488] Because the management and efficient use of ingredients in refrigerators are unclear, there is a need for a system that eliminates food waste and suggests the optimal cooking method based on the expiration date of the ingredients. Furthermore, conventional systems have not been able to provide suggestions that take into account the user's past selection history and preferences.

[0489] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0490] In this invention, the server includes means for inputting ingredient information, means for inputting expiration date information of the ingredient, means for retrieving multiple cooking methods using the ingredient from a data storage means based on the input information, means for determining priority based on the expiration date information of the ingredient and arranging each cooking method according to priority, and means for proposing the highest priority cooking method to the user and displaying how to prepare the cooking method. This makes it possible to efficiently utilize ingredients in the refrigerator, reduce food waste, and propose appropriate cooking methods that take into account the user's preferences.

[0491] "Means for inputting food information" refers to a device or interface for users to input the types and names of food items inside a refrigerator.

[0492] "Means for inputting expiration date information" refers to a device or interface for users to input the best-before date or expiration date for each food item.

[0493] "Data storage means" refers to a database or storage device for storing and managing information on ingredients, expiration dates, and multiple cooking methods based on these.

[0494] "Means for determining and arranging priorities" refers to a device or program that has the function of setting the priority of each cooking method based on the input expiration date information and listing them in that order.

[0495] "Means for proposing and displaying cooking methods" refers to a device or interface that presents the user with the determined optimal cooking method and displays the detailed preparation procedure.

[0496] "Communication means" refers to a network or communication interface used to transmit data input from a terminal to the main control unit.

[0497] The "main control unit" is a central control unit or server that receives and analyzes data and selects the optimal cooking method.

[0498] "Means of selecting based on the user's past selection history and preferences" refers to a device or program that has the function of recording the cooking methods and preferences previously chosen by the user and selecting the optimal cooking method taking that into consideration.

[0499] This invention is a system that manages information about food items in a refrigerator and their expiration dates, and suggests the optimal cooking method. The following describes specific embodiments for carrying out this invention.

[0500] This system allows users to input each ingredient in their refrigerator and its expiration date, and the server then searches for and suggests the most suitable cooking method based on that data. This enables users to efficiently utilize the ingredients in their refrigerator and reduce food waste.

[0501] System Configuration

[0502] 1. User terminal:

[0503] This is a device used by users to input food ingredient information and expiration date information. This includes smartphones, tablets, or PCs. Users launch a dedicated app and enter data through input forms such as text boxes.

[0504] 2. Server:

[0505] The server is the central component that receives data sent from the user terminal and searches for and suggests appropriate cooking methods based on that data. The server accesses the database, retrieves the relevant cooking methods, and provides data according to the user's request.

[0506] 3. Database:

[0507] The database is a system for storing ingredients, cooking methods, and related information. It manages details of each cooking method, required ingredients, and their corresponding expiration dates.

[0508] Specific example

[0509] For example, if a user inputs that they have "pork," "cabbage," and "carrots" in their refrigerator, the server searches its database for cooking methods using these ingredients and prioritizes them based on their expiration dates. As a result, it provides the user with a suggested cooking method, such as "stir-fried pork and cabbage," along with a detailed explanation of how to make it. By cooking based on this suggestion, the user can efficiently utilize the ingredients in their refrigerator.

[0510] Example of a prompt

[0511] Please enter the information about the ingredients in your refrigerator and their expiration dates. For example, enter it in the format of 'Pork, Expiration Date 2023-10-20', 'Cabbage, Expiration Date 2023-10-18', etc. We will then suggest the best cooking method.

[0512] This prompt allows users to easily operate the system and input ingredient and expiration date information. This enables the system to provide appropriate cooking instructions and reduce food waste.

[0513] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0514] Step 1: User enters ingredient information.

[0515] Users open their smartphones or tablets and launch the dedicated app. They then enter food information and expiration dates into the app's input form, such as "Pork, Expiration Date 2023-10-20" or "Cabbage, Expiration Date 2023-10-18". Specifically, users type the information into the text box and press the submit button to complete the input.

[0516] Input: Ingredient information (e.g., "Pork, Cabbage") and its expiration date (e.g., "2023-10-20, 2023-10-18")

[0517] Output: User-input text data

[0518] Step 2: Data conversion and transmission by the terminal

[0519] The device converts the ingredient information entered by the user into JSON format. For example, it formats it into the following format: {"materials": [{"name": "Pork", "expiry_date": "2023-10-20"}, {"name": "Cabbage", "expiry_date": "2023-10-18"}]}. Next, it sends this data to the server as an HTTP POST request. Specifically, the device sends the data to the API endpoint over the internet.

[0520] Input: User-inputted text data

[0521] Output: Data in JSON format and an HTTP POST request

[0522] Step 3: Server receives and analyzes data.

[0523] The server receives an HTTP request from the terminal and extracts and parses the data sent from the payload. Specifically, it parses JSON data in the format {"materials": [{"name": "Pork", "expiry_date": "2023-10-20"}, {"name": "Cabbage", "expiry_date": "2023-10-18"}]} to obtain the name of each ingredient and its expiration date.

[0524] Input: Data in JSON format

[0525] Output: Information on the names of ingredients and their corresponding expiration dates.

[0526] Step 4: Search for cooking methods using the server

[0527] The server uses the analyzed ingredient names to query the database and search for corresponding cooking methods. For example, to retrieve "recipes using pork and cabbage" from the database, it sends an SQL query. Specifically, it executes the query and returns a list of multiple cooking methods.

[0528] Input: Name of ingredients (e.g., "Pork, cabbage")

[0529] Output: List of suggested cooking methods

[0530] Step 5: Server prioritizes cooking method

[0531] The server evaluates the ingredient list and expiration dates for each cooking method and sets priorities. Specifically, it prioritizes cooking methods that use ingredients with shorter expiration dates. Based on the calculated priorities, it sorts the list of cooking methods.

[0532] Input: A list of suggested cooking methods and information on ingredients and their expiration dates.

[0533] Output: List of cooking methods sorted by priority

[0534] Step 6: Server selects the optimal cooking method

[0535] The server selects the optimal cooking method from among the high-priority options, taking into account the user's past selection history and preferences. For example, if the user previously preferred "stir-fry," the server will select "stir-fried pork and cabbage."

[0536] Input: A list of cooking methods sorted by priority, user selection history, and preference data.

[0537] Output: Optimal cooking method

[0538] Step 7: Server generates and transmits recipe information.

[0539] The server generates detailed information in JSON format about the selected optimal cooking method. For example, it might generate it in the format {"menu": {"name": "Stir-fried pork and cabbage", "instructions": "Cut the ingredients and stir-fry them in a frying pan"}} and send this to the terminal as an HTTP response.

[0540] Input: Optimal cooking method

[0541] Output: Detailed cooking instructions (in JSON format) and HTTP response

[0542] Step 8: Displaying the recipe on the device

[0543] The device parses the JSON data received from the server and displays the suggested cooking method and instructions to the user. Specifically, it displays "Today's suggested menu: Stir-fried pork and cabbage, Instructions: Cut the ingredients and stir-fry in a frying pan."

[0544] Input: Detailed cooking instructions (JSON format)

[0545] Output: Display of cooking instructions and preparation methods to the user.

[0546] Step 9: User performs cooking

[0547] The user checks the displayed cooking method and instructions, and then prepares the dish based on the ingredients in their refrigerator. Specifically, they prepare the ingredients according to the provided steps and complete the dish.

[0548] Input: Detailed cooking instructions and ingredients in the refrigerator

[0549] Output: Finished dish

[0550] (Application Example 1)

[0551] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0552] In modern times, there is a need to efficiently manage the food in refrigerators and reduce the amount of food that goes past its expiration date. However, it is difficult for users to keep track of the food in their refrigerators and use it appropriately in their daily lives, and food waste often occurs. Also, if necessary ingredients are missing, people have to go shopping each time, which is time-consuming and troublesome. To solve this problem, a system is needed that efficiently manages information about the food in the refrigerator and has an automatic ordering function for missing ingredients.

[0553] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0554] In this invention, the server includes means for inputting information on multiple food items present in the refrigerator, means for inputting the expiration dates of the food items, means for searching a database for multiple recipes using the food items based on the input information, means for determining priority based on the expiration dates of the food items and sorting each recipe according to priority, means for suggesting the highest-priority recipe to the user and displaying the instructions for making that recipe, and means for automatically ordering any missing ingredients based on the information on the food items. This enables centralized management of food item information in the refrigerator, allowing for optimal recipe suggestions to the user and automatic ordering of necessary ingredients.

[0555] "Means for inputting information on multiple food items present in a refrigerator" refers to devices or applications that allow users to input information such as the names, quantities, and locations of various food items stored in a refrigerator.

[0556] "Means for inputting the expiration date of the food item" refers to a device or application for the user to input the expiration date of each food item in the refrigerator.

[0557] "Means for searching a database for multiple recipes using the ingredients in question based on the input information" refers to a server or software that searches a database for multiple recipes using the ingredients entered by the user.

[0558] "Means for determining priority based on the expiration date of the ingredient and sorting each recipe according to priority" refers to a server or algorithm that determines whether an ingredient should be used preferentially based on the expiration date information of the input ingredient, and then sorts the corresponding recipes in order of priority.

[0559] "A means of suggesting a top-priority recipe to a user and displaying how to make that recipe" refers to a device or application that presents the user with the highest priority recipe and displays the user's detailed instructions for making that recipe.

[0560] "Means for automatically ordering missing ingredients based on information about the ingredients in question" refers to a server or software that automatically determines the missing ingredients needed for a dish based on ingredient information entered by the user and orders them online.

[0561] "A means of communication for sending data entered from a terminal to a server" refers to a communication interface, protocol, or system for sending information about ingredients and expiration dates from a terminal used by a user to a server.

[0562] "A means of selecting the optimal recipe from multiple recipes based on the user's past selection history and preferences" refers to an algorithm or software that analyzes the user's past selection history and preference information and suggests the most suitable recipe based on that information.

[0563] This invention relates to a system that manages the remaining ingredients in a refrigerator and their expiration dates, suggests the optimal menu and how to prepare it, and can automatically order any missing ingredients from an online shopping site.

[0564] System Configuration

[0565] 1. Method for entering information about the ingredients in the refrigerator:

[0566] Users use devices such as smartphones or tablets to input information about the food items in their refrigerator and their expiration dates. For example, they might enter "Pork, expiration date 2023-10-20" and "Cabbage, expiration date 2023-10-18" into the app's input form.

[0567] 2. Means of communication:

[0568] The aforementioned terminal sends the input data to the server via HTTP communication. It uses JSON format data for communication and sends an HTTP POST request.

[0569] 3. Means of searching for recipes based on input information:

[0570] The server analyzes the received data and searches the database for a suitable recipe based on the information about the ingredients in the refrigerator. For example, the server analyzes the received JSON data and retrieves the corresponding recipe from the database.

[0571] 4. Means for determining and sorting recipe priority:

[0572] The server prioritizes ingredients that should be consumed as soon as possible, based on their expiration dates. It then sorts the searched recipes according to this priority.

[0573] 5. Means for suggesting the highest priority recipe to the user and displaying the instructions:

[0574] The server selects the highest-priority recipe from the sorted list and sends its detailed instructions to the terminal. For example, the selected recipe for "Stir-fried Pork and Cabbage" and its instructions will be displayed.

[0575] 6. A means of automatically ordering missing materials:

[0576] The server matches the entered ingredient information with the suggested recipe and automatically orders any necessary ingredients that are not in the refrigerator from an online shopping site. For example, if the customer only has pork and cabbage and does not have carrots, which are needed for "stir-fried pork and cabbage," the server will automatically add carrots to the shopping cart from the online store.

[0577] Program Processing Description

[0578] The server uses a Python program to suggest recipes based on user input and automatically order any missing ingredients. The server uses the Python Requests library for communication. SQL is used to search for recipes in the database. Algorithms are implemented for prioritizing and sorting recipes. Furthermore, a machine learning model is used to select the optimal recipe based on the user's past selection history and preferences.

[0579] For example, if a user enters "pork, cabbage" along with expiration date information through a smartphone app, that information is sent to the server. The server then searches for a suitable recipe and suggests "stir-fried pork and cabbage," prioritizing ingredients with the earliest expiration dates. If carrots are missing in the recipe, they are automatically ordered from an online shopping site.

[0580] Example of a prompt

[0581] Please enter the ingredients in your refrigerator (e.g., "pork," "cabbage") and their expiration dates. Based on this information, we will automatically order the necessary ingredients from an online shopping site and suggest the best recipes.

[0582] As a result, this invention allows for efficient management of ingredients in the refrigerator, reduces food waste, and makes meal preparation easier for the user.

[0583] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0584] Step 1:

[0585] Users input information about the food items in their refrigerator and their expiration dates using a smartphone, tablet, or other device.

[0586] Specific examples of the information to be entered include "Pork, Expiration Date 2023-10-20" and "Cabbage, Expiration Date 2023-10-18". This will be the input data.

[0587] Step 2:

[0588] The terminal converts the entered ingredient information into JSON format and sends it to the server as an HTTP POST request.

[0589] This communication method utilizes the JSON format and the HTTP protocol. The data is sent in the format {"materials": [{"name": "Pork", "expiry_date": "2023-10-20"}, {"name": "Cabbage", "expiry_date": "2023-10-18"}]}. This serves as the input data, and the server performs the analysis.

[0590] Step 3:

[0591] The server receives an HTTP POST request and parses the received data.

[0592] The analyzed data extracts the expiration dates of each food item in the refrigerator. Based on the input data, a query is generated for the database to search for the corresponding recipe. For example, it searches for recipes using "pork" and "cabbage".

[0593] Step 4:

[0594] The server receives multiple recipes as search results from the database, determines a priority based on the expiration dates of the ingredients in each recipe, and sorts the recipes.

[0595] This sorting is performed using an algorithm that prioritizes recipes that use ingredients nearing their expiration date. For example, "Stir-fried pork and cabbage" would be one such recipe.

[0596] Step 5:

[0597] The server selects the most suitable recipe from several options based on the user's past selection history and preferences.

[0598] This process utilizes machine learning models. Based on past selection history data and preference data, it selects the most suitable recipe and prioritizes it to determine the best option.

[0599] Step 6:

[0600] The server generates detailed information about the selected top-priority recipe in JSON format and sends it to the terminal.

[0601] For example, send the following in the format: {"menu": {"name": "Stir-fried pork and cabbage", "instructions": "Cut the ingredients and stir-fry them in a frying pan"}}. This will be the output data.

[0602] Step 7:

[0603] The device analyzes the received recipe information and displays suggested menus and their preparation methods to the user.

[0604] Specifically, it displays: "Today's suggested menu: Stir-fried pork and cabbage, Instructions: Cut the ingredients and stir-fry them in a frying pan."

[0605] Step 8:

[0606] The server automatically orders any missing ingredients from online shopping sites based on the suggested recipe.

[0607] The server generates a list of all the ingredients needed for the suggested recipe and determines which ingredients are missing from the refrigerator. It then automatically orders the missing ingredients through the online shopping site's API. For example, it adds "carrots" to the online shopping site's shopping cart.

[0608] Step 9:

[0609] The user prepares the meal according to the suggested menu and instructions. They also confirm that any missing ingredients are automatically ordered from an online store, and the purchasing process proceeds. Thus, the user's meal preparation is complete.

[0610] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0611] This invention combines an emotional engine with a system that manages remaining ingredients in a refrigerator and their expiration dates, and suggests the optimal menu and preparation method. The following describes specific embodiments for carrying out the invention.

[0612] overview

[0613] This system allows users to input each ingredient in their refrigerator and its expiration date, and the server then searches for and suggests the most suitable recipe based on that data. Furthermore, it can use an emotion engine to recognize the user's emotional state and provide recipes tailored to that state.

[0614] System Configuration

[0615] 1. User terminal:

[0616] The user terminal is a device used to input information about the food items in the refrigerator and their expiration dates. It also includes voice and image analysis functions to input the user's emotional state. Examples include smartphones, tablets, and PCs.

[0617] 2. Server:

[0618] The server is the central component that receives ingredient information and emotional states sent from the user's terminal and searches for and suggests appropriate recipes based on that information. The server accesses the database to retrieve appropriate recipes and provides data according to the user's request.

[0619] 3. Database:

[0620] The database is a system for storing ingredients, recipes, and related information. It manages details of each recipe, required ingredients and their corresponding expiration dates, as well as data on the user's past selection history and emotional state.

[0621] 4. Emotional Engine:

[0622] The emotion engine is a system for recognizing a user's emotional state. It includes voice analysis and image analysis methods to analyze the user's emotions from their voice and facial expressions and transmit that data to a server.

[0623] Specific processing flow

[0624] Data entry phase

[0625] 1. The user enters information about the food items in their refrigerator and their expiration dates using a smartphone or other device. For example, they might enter information such as "Pork, expiration date 2023-10-20" or "Cabbage, expiration date 2023-10-18" into the application's input form.

[0626] 2. The user inputs their emotional state through the emotion engine. The emotional state is analyzed by capturing facial expressions using voice input or a camera.

[0627] Data transmission phase

[0628] 3. The terminal converts the entered food information and emotional state into JSON data. For example, {"materials": [{"name": "Pork", "expiry_date": "2023-10-20"}, {"name": "Cabbage", "expiry_date": "2023-10-18"}], "emotion": {"mood": "happy"}}.

[0629] 4. The terminal sends data to the server. The transmission method is an HTTP POST request.

[0630] Data Processing Phase

[0631] 5. The server receives data sent from the terminal. It parses the HTTP request, obtains the JSON data, and then parses it.

[0632] 6. The server accesses the database and searches for a suitable recipe based on the received ingredient information and emotional state. For example, it retrieves recipes such as "stir-fried vegetables" or "curry" that contain "pork" and "cabbage" from the database.

[0633] 7. The server determines the priority of the retrieved recipes based on the expiration date of each ingredient. For example, it compares the expiration dates of the ingredients and sorts the recipes with ingredients that need to be consumed the fastest to the top.

[0634] 8. The server selects the optimal recipe by considering the user's past selection history and preferences, as well as their emotional state recognized by the emotion engine. For example, it will suggest more enjoyable dishes for a "happy" emotional state.

[0635] Proposed menu generation phase

[0636] 9. The server generates detailed information about the selected recipe in JSON format. For example, it creates data in the format {"menu": {"name": "Stir-fried pork and cabbage", "instructions": "Chop the ingredients and stir-fry them in a pan"}}.

[0637] 10. The server sends the generated JSON data to the terminal as an HTTP response. The response contains the selected recipe and its details.

[0638] Data display phase

[0639] 11. The terminal analyzes the response data received from the server. It parses the JSON data to obtain the suggested menu and how to make it.

[0640] 12. The terminal displays suggested menus and instructions for preparation to the user based on the analyzed data. For example, it might display "Today's suggested menu: Stir-fried pork and cabbage, Instructions: Cut the ingredients and stir-fry them in a pan."

[0641] 13. The user reviews the displayed information and prepares the dish based on the suggested menu. For example, they might cut pork and cabbage, then stir-fry them in a pan to complete the dish.

[0642] Specific example

[0643] For example, if a user inputs that they have "pork" and "cabbage" in their refrigerator, and the emotion engine recognizes a "happy" emotional state, the server searches its database for recipes using these ingredients, prioritizing them based on their expiration dates. As a result, it provides the user with a suggested menu item, "stir-fried pork and cabbage," which is appropriate for their happy emotional state, and explains how to make it in detail. By following these instructions, the user can efficiently utilize the ingredients in their refrigerator.

[0644] This system not only allows for the effective management and use of ingredients in the refrigerator, but also enables personalized recipe suggestions tailored to the user's emotional state. This makes the daily cooking experience more enjoyable and satisfying, contributing to a reduction in food waste.

[0645] The following describes the processing flow.

[0646] Step 1:

[0647] The user collects information on multiple food items in their refrigerator, along with their expiration dates. Using a smartphone or other device, the user enters information such as "Pork, expiration date 2023-10-20" and "Cabbage, expiration date 2023-10-18" into the application's form.

[0648] Step 2:

[0649] Users record their emotional state through an emotion engine. For example, their emotional state is analyzed by capturing facial expressions using voice input or a camera.

[0650] Step 3:

[0651] The terminal converts the entered food information and emotional state into JSON data. For example, the format will be {"materials": [{"name": "Pork", "expiry_date": "2023-10-20"}, {"name": "Cabbage", "expiry_date": "2023-10-18"}], "emotion": {"mood": "happy"}}.

[0652] Step 4:

[0653] The terminal sends the converted data to the server. The transmission method uses an HTTP POST request. For example, the request body contains the JSON data mentioned earlier.

[0654] Step 5:

[0655] The server receives data sent from the terminal. The server parses the HTTP request, obtains JSON data, and parses it. For example, it might obtain information about each ingredient in array format as the parsed data.

[0656] Step 6:

[0657] The server accesses the database and searches for relevant recipes based on the received ingredient information and emotional state. The database stores detailed information and required ingredient lists for each recipe. For example, it retrieves recipes such as "stir-fried vegetables" and "curry" that contain "pork" and "cabbage."

[0658] Step 7:

[0659] The server prioritizes the retrieved recipes based on the expiration dates of each ingredient. For example, it compares the expiration dates of ingredients and sorts recipes prioritizing those with ingredients that need to be consumed the fastest.

[0660] Step 8:

[0661] The server selects the optimal recipe by considering the user's past selection history and preferences, as well as their emotional state recognized by the emotion engine. For example, it suggests more enjoyable dishes for a "happy" emotional state.

[0662] Step 9:

[0663] The server generates detailed information about the selected recipe in JSON format. For example, it creates data in the format {"menu": {"name": "Stir-fried pork and cabbage", "instructions": "Chop the ingredients and stir-fry them in a frying pan"}}.

[0664] Step 10:

[0665] The server sends the generated JSON data to the terminal as an HTTP response. The response contains the selected recipe and its details.

[0666] Step 11:

[0667] The terminal analyzes the response data received from the server. It parses the JSON data to obtain the suggested menu and how to make it.

[0668] Step 12:

[0669] The device displays suggested menus and instructions for preparation to the user based on the analyzed data. For example, it might display something like, "Today's suggested menu: Stir-fried pork and cabbage, Instructions: Cut the ingredients and stir-fry them in a pan."

[0670] Step 13:

[0671] The user reviews the displayed information and prepares the dish based on the suggested menu. For example, they might cut pork and cabbage, then stir-fry them in a pan to complete the dish.

[0672] (Example 2)

[0673] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0674] Modern households store a wide variety of ingredients in their refrigerators, but efficiently utilizing these ingredients is difficult. In particular, it is hard to plan for consuming ingredients that are nearing their expiration date, leading to food waste. Furthermore, the lack of meal suggestions tailored to the user's emotional state on any given day diminishes the enjoyment and satisfaction of daily cooking. Therefore, there is a need for a system that can suggest optimal recipes based on ingredient information and emotional state, thereby reducing food waste and providing a cooking experience that matches the user's emotions.

[0675] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for inputting information on multiple food items present in the refrigerator, means for inputting the expiration dates of the food items, and means for inputting the user's emotional state. This makes it possible to search a database for multiple recipes using the food items based on the input information and emotional state, determine priority based on the expiration dates of the food items, sort each recipe according to priority, select the optimal recipe considering the user's past selection history, preferences, and emotional state, propose the highest priority recipe to the user, and display how to make it. This makes it possible to reduce food waste and provide personalized cooking suggestions that are tailored to the user's emotions.

[0676] "A means of inputting information on multiple food items present in a refrigerator" refers to a device or software that allows a user to input the types of food items stored in the refrigerator and their respective expiration dates into a terminal.

[0677] "Means for inputting the expiration date of the food item" refers to a device or software that allows the user to input the expiration date of each food item into a terminal.

[0678] "Means for inputting the user's emotional state" refers to devices or software that analyze the user's voice and facial expressions to recognize their emotional state and input that information into the system.

[0679] "Means for searching a database for multiple recipes using the ingredient in question based on input information and emotional state" refers to an algorithm or software for searching a database for relevant recipes using ingredient information and emotional state data received from a terminal.

[0680] "Means for determining priority based on the expiration date of the ingredients and sorting each recipe according to priority" refers to an algorithm or software that determines the priority of recipes based on the expiration date of the input ingredients and rearranges the recipes accordingly.

[0681] "Means for selecting the optimal recipe by considering the user's past selection history, preferences, and emotional state" refers to algorithms or software that use data on the user's past recipe selection history and emotional state to determine and select the most suitable recipe for the user.

[0682] "Means for suggesting a top-priority recipe to the user and displaying how to make that recipe" refers to devices or software that notify the user of a high-priority recipe and display its detailed instructions.

[0683] "A means of communication for sending data entered from a terminal to a server" refers to network communication devices and protocols for sending food information and emotional states entered from a user's terminal to a server.

[0684] An "emotion engine" is an algorithm or software that analyzes a user's voice and facial expressions to identify their emotional state.

[0685] This invention relates to a system that provides an optimal recipe by taking into account information about the ingredients in the refrigerator, their expiration dates, and the user's emotional state. The specific methods for implementing this invention are described below.

[0686] System Configuration

[0687] This system consists of user terminals, a server, a database, and an emotion engine. Details of each component are as follows:

[0688] User terminal

[0689] The user terminal is a device used to input information about the food items in the refrigerator and their expiration dates. It also includes voice and image analysis functions for inputting the user's emotional state. Examples include smartphones, tablets, and PCs.

[0690] server

[0691] The server is the central component that receives ingredient information and emotional states sent from the user's terminal and searches for and suggests appropriate recipes based on that information. The server accesses a database to retrieve appropriate recipes and provides data according to the user's request. Specifically, Apache servers and Nginx servers are often used.

[0692] database

[0693] A database is a system for storing ingredients, recipes, and related information. It manages details of each recipe, required ingredients and their corresponding expiration dates, as well as data on the user's past selection history and emotional state. Specifically, relational databases such as MySQL and PostgreSQL are used.

[0694] Emotional Engine

[0695] An emotion engine is a system for recognizing a user's emotional state. It includes voice and image analysis methods to analyze emotions from the user's voice and facial expressions and send that data to a server. Specifically, it uses tools such as OpenCV and the Google Cloud Speech-to-Text API.

[0696] Specific example

[0697] The following specific examples will help to understand the present invention.

[0698] For example, if a user inputs that they have "pork" and "cabbage" in their refrigerator, and the emotion engine recognizes a "happy" emotional state, the server searches its database for recipes using these ingredients, prioritizing them based on their expiration dates. As a result, it generates a suggested menu item, such as "Pork and Cabbage Pasta," which is adapted to the user's happy emotional state. The server then sends this recipe and detailed instructions to the user's terminal, allowing the user to begin cooking.

[0699] Example of a prompt

[0700] The following are specific examples of prompts to input into a generative AI model:

[0701] The refrigerator contains pork and cabbage. The current emotional state is "happy." Please suggest a recipe using these ingredients that is appropriate for this emotional state.

[0702] This invention enables effective management of ingredients in the refrigerator and provides optimal recipe suggestions, thereby reducing food waste and realizing a personalized cooking experience that responds to the user's emotions.

[0703] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0704] Step 1:

[0705] Users input information about the food items in their refrigerator and their expiration dates using a smartphone or other device. For example, they might enter "Pork, expiration date 2023-10-20" and "Cabbage, expiration date 2023-10-18" into the app's input form. The input data is output as JSON data from the device's input form.

[0706] Step 2:

[0707] Users input their emotional state through the emotion engine. For example, they might use the voice input function to say "I feel good today," and the emotion engine analyzes the voice and recognizes the user's emotional state as "happy." The input data is output from the emotion engine as JSON data.

[0708] Step 3:

[0709] The terminal integrates the entered food information and emotional state to generate final JSON data. For example, it formats the data as follows: {"materials": [{"name": "Pork", "expiry_date": "2023-10-20"}, {"name": "Cabbage", "expiry_date": "2023-10-18"}], "emotion": {"mood": "happy"}}. The integrated data is then sent from the terminal to the server.

[0710] Step 4:

[0711] The device sends the generated JSON data to the server using an HTTP POST request. The detailed communication protocol used is HTTP / HTTPS. The transmitted data is received by the server.

[0712] Step 5:

[0713] The server parses the received HTTP request and retrieves JSON data. Specifically, it extracts JSON data from the body of the HTTP request. The extracted data is then passed to an internal processing module.

[0714] Step 6:

[0715] The server accesses the database and searches for relevant recipes based on the received ingredient information. For example, it searches the database for recipes containing "pork" and "cabbage" using an SQL query. The search results return a list of relevant recipes.

[0716] Step 7:

[0717] The server compares the expiration dates of each ingredient in the retrieved recipes to determine priority. For example, to prioritize ingredients with approaching expiration dates, it uses the SQL ORDER BY clause to sort the recipes. A sorted list of recipes is then generated.

[0718] Step 8:

[0719] The server selects the optimal recipe by considering the user's past selection history and preferences. It also takes into account the emotional state recognized by the emotion engine. For example, it might select the best recipe from those the user has previously chosen when they were in a "happy" emotional state. One optimal recipe is then selected.

[0720] Step 9:

[0721] The server generates detailed information about the selected recipe in JSON format. For example, it creates data in the format {"menu": {"name": "Pork and cabbage pasta", "instructions": "Chop the ingredients, boil the pasta, and stir-fry in a pan"}}. The generated data is then sent to the terminal.

[0722] Step 10:

[0723] The server sends the generated JSON data to the terminal as an HTTP response. The response data is sent via the HTTP / HTTPS protocol.

[0724] Step 11:

[0725] The terminal analyzes the response data received from the server and parses the data in JSON format. For example, it analyzes data such as {"menu": {"name": "Pork and cabbage pasta", "instructions": "Cut the ingredients, boil the pasta, and stir-fry in a pan"}}. The analysis results are stored as an internal data structure.

[0726] Step 12:

[0727] The device displays suggested menus and instructions for preparation based on the analyzed data. For example, it might display: "Today's suggested menu: Pork and cabbage pasta, Instructions: Cut the ingredients, boil the pasta, and stir-fry in a pan."

[0728] Step 13:

[0729] The user cooks according to the displayed recipe. For example, they might cut pork and cabbage, boil pasta, and then stir-fry it in a pan to complete the dish.

[0730] (Application Example 2)

[0731] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0732] Traditional refrigerator management systems, while capable of managing information about ingredients and expiration dates, were unable to provide personalized recipe suggestions based on the user's emotional state. This led to decreased user satisfaction and food waste. Furthermore, the suggested recipes often didn't match the user's mood or preferences at the time, resulting in a lack of enjoyment in daily cooking.

[0733] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0734] In this invention, the server includes means for inputting information on multiple food items present in the refrigerator, means for inputting the expiration dates of the food items, means for determining priority based on the expiration dates of the food items and recognizing the user's emotional state, and means for optimizing recipes based on the emotional state. This enables the suggestion of optimal recipes according to the user's emotional state, improving the daily cooking experience and reducing food waste.

[0735] "Information on multiple food items present in the refrigerator" refers to data about various food items stored in the refrigerator, including their names, quantities, acquisition dates, and other attributes.

[0736] The "best before date" for a food item refers to the date by which the food stored in the refrigerator should be consumed appropriately, and this date varies depending on the food item.

[0737] "Means of input" refers to devices or methods for users to register information about the food in their refrigerator and its expiration date into the system, such as input devices like smartphones, tablets, or PCs.

[0738] "Methods for searching a database" refer to programs or algorithms that extract information from a database based on specific conditions and provide it to the user.

[0739] "Means for determining priority and sorting each recipe according to priority" refers to a method for determining the priority of multiple recipes based on expiration dates or other conditions, and then rearranging the recipes based on the results.

[0740] "Means for recognizing a user's emotional state" refers to devices or software that analyze a user's facial expressions, voice, etc., to evaluate their psychological state at that time.

[0741] "Methods for optimizing recipes based on emotional state" refer to methods or systems that select recipes and suggest the most suitable dishes by reflecting the recognized emotional state of the user.

[0742] "A means of suggesting the highest-priority recipe to the user and displaying how to make that recipe" refers to a function that presents the user with the most suitable recipe selected by the system and provides a detailed explanation of how to cook it.

[0743] This invention combines an emotional engine with a system that manages the remaining ingredients in a refrigerator and their expiration dates, and suggests the optimal menu and how to prepare it. The following describes specific embodiments for carrying out the invention.

[0744] System Configuration

[0745] 1. User terminal

[0746] The user terminal is a device used to input information about the food items in the refrigerator and their expiration dates. It also includes voice and image analysis functions to input the user's emotional state. Smartphones, tablets, and PCs are examples of this type of device. The terminal also has a means of communication to send the input data to the server.

[0747] 2. Server

[0748] The server is the central component that receives ingredient information and emotional states sent from the user's terminal and searches for and suggests appropriate recipes based on that information. The server accesses the database to retrieve appropriate recipes and provides data according to the user's request.

[0749] The software used includes Flask (a web server framework) and other web server frameworks.

[0750] 3. Database

[0751] The database is a system for storing ingredients, recipes, and related information. It manages details of each recipe, required ingredients and their corresponding expiration dates, as well as data on the user's past selection history and emotional state.

[0752] 4. Emotional Engine

[0753] An emotion engine is a system for recognizing a user's emotional state and includes methods for speech analysis and image analysis. Specifically, speech recognition APIs (e.g., Google Speech-to-Text API) are used for speech analysis, and libraries such as OpenCV are used for image analysis.

[0754] Specific processing flow

[0755] Users input information about the food items in their refrigerator and their expiration dates using a smartphone or other device. For example, they might enter information such as "Pork, expiration date 2023-10-20" or "Cabbage, expiration date 2023-10-18" into the application's input form. Next, they input their emotional state through an emotion engine, and their emotional state is analyzed by capturing facial expressions using voice input or a camera.

[0756] The terminal converts the entered ingredient information and emotional state into JSON data and sends it to the server. The server receives and analyzes this data, accesses the database, and searches for the corresponding recipe based on the received ingredient information and emotional state. Furthermore, it determines priorities based on the expiration date of each ingredient and selects the optimal recipe by considering the user's past selection history and preferences, as well as the emotional state recognized by the emotion engine.

[0757] Detailed information about the selected recipe is generated in JSON format and sent from the server to the terminal. The terminal parses it and displays the suggested menu and its preparation method to the user. By following these instructions, the user can efficiently utilize the ingredients in their refrigerator.

[0758] Examples of specific cases and prompt statements

[0759] Specific example

[0760] If a user inputs that they have "pork" and "cabbage" in their refrigerator and the emotion engine recognizes a "happy" emotional state, the server searches its database for recipes using these ingredients, prioritizing them based on their expiration dates. As a result, it provides the user with a suggested menu item, "Stir-fried pork and cabbage," which is appropriate for their happy emotional state, and explains how to make it in detail.

[0761] Example of a prompt

[0762] If you want to send the following data to the server, you can send data like this:

[0763] {

[0764] "materials": ["pork", "cabbage"],

[0765] "expiry_dates": ["2023-10-20", "2023-10-18"],

[0766] "image": <Image data of the user's facial expression captured by the camera>

[0767] }

[0768] In this way, users can not only effectively manage and utilize the ingredients in their refrigerator using the system, but also receive personalized recipe suggestions based on their emotional state. This makes the daily cooking experience more enjoyable and satisfying, and also contributes to reducing food waste.

[0769] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0770] Step 1:

[0771] The user enters information about the food items in their refrigerator and their expiration dates using a smartphone or other device.

[0772] Specific operation: The user launches the application and enters the name, quantity, and expiration date of the ingredients into the input form.

[0773] Input: Ingredient information (e.g., "Pork", "Cabbage") and expiration date (e.g., "2023-10-20", "2023-10-18")

[0774] Output: Input food information and expiration date data

[0775] Step 2:

[0776] Users input their emotional state through an emotion engine, and their facial expressions are captured using voice input or a camera.

[0777] Specific operation: The user uses the in-app camera function to take a picture of their face and have their emotional state recognized.

[0778] Input: User's facial expression images and audio data

[0779] Output: User's emotional state (e.g., "happy", "sad")

[0780] Step 3:

[0781] The terminal converts the entered food information and emotional state into JSON data and sends it to the server.

[0782] Specific operation: The terminal packages the ingredient information and emotional state into JSON format (for example, {"materials": ["pork", "cabbage"], "expiry_dates": ["2023-10-20", "2023-10-18"], "emotion": "happy"}) and sends it to the server using an HTTP POST request.

[0783] Input: Data in JSON format

[0784] Output: HTTP request sent to the server

[0785] Step 4:

[0786] The server receives and analyzes the data sent from the terminal.

[0787] Specific operation: The server receives an HTTP request and parses the JSON data to retrieve ingredient information, expiration date, and emotional state.

[0788] Input: JSON data sent from the device

[0789] Output: Analyzed food information, expiration date, emotional state

[0790] Step 5:

[0791] The server accesses the database and searches for the appropriate recipe based on the received ingredient information and emotional state.

[0792] Specific operation: The server executes a database query to retrieve a recipe suitable for the ingredients. The database stores recipes corresponding to the ingredients and emotional states.

[0793] Input: Analyzed food information, expiration date, emotional state

[0794] Output: List of relevant recipes

[0795] Step 6:

[0796] The server determines the priority of recipes based on the expiration date of each ingredient and sorts the recipes accordingly.

[0797] Specific operation: The server compares the expiration dates of ingredients and prioritizes recipes that include ingredients that need to be consumed the soonest.

[0798] Input: List of relevant recipes, expiration dates of ingredients

[0799] Output: High-priority recipe list

[0800] Step 7:

[0801] The server selects the optimal recipe by considering the user's past selection history, preferences, and emotional state.

[0802] Specific operation: The server retrieves the user's past selection history from the database and executes an algorithm to select the optimal recipe in conjunction with their emotional state.

[0803] Input: High-priority recipe list, user's past selection history, emotional state

[0804] Output: Optimal recipe

[0805] Step 8:

[0806] The server generates detailed information about the selected recipe in JSON format and sends it to the terminal.

[0807] Specific operation: The server formats the selected recipe and its details (e.g., ingredients, instructions) in JSON format and sends it to the terminal as an HTTP response.

[0808] Input: Optimal recipe

[0809] Output: Recipe data in JSON format

[0810] Step 9:

[0811] The terminal analyzes the response data received from the server and displays the suggested menu and its preparation method to the user.

[0812] Specific operation: The terminal parses the received JSON data and displays the recipe and cooking instructions on the screen.

[0813] Input: Recipe data in JSON format

[0814] Output: The recipe and its details displayed to the user

[0815] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0816] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0817] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0818] [Third Embodiment]

[0819] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0820] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0821] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0822] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0823] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0824] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0825] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0826] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0827] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0828] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0829] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0830] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0831] This invention relates to a system for managing leftover ingredients in a refrigerator and their expiration dates, and for suggesting optimal menus and their preparation methods. The following describes specific embodiments for carrying out the invention.

[0832] overview

[0833] This system allows users to input each ingredient in their refrigerator and its expiration date, and the server then searches for and suggests the most suitable recipe based on that data. Users can then cook based on these suggestions, efficiently utilizing the ingredients in their refrigerator.

[0834] System Configuration

[0835] 1. User terminal:

[0836] The user terminal is a device used to input information about the food items in the refrigerator and their expiration dates. This could be a smartphone, tablet, or PC.

[0837] 2. Server:

[0838] The server is the central component that receives ingredient information sent from the user's terminal and searches for and suggests appropriate recipes based on that information. The server accesses the database to retrieve appropriate recipes and provides data according to the user's request.

[0839] 3. Database:

[0840] The database is a system for storing ingredients, recipes, and related information. It manages details of each recipe, the required ingredients, and their corresponding expiration dates.

[0841] Specific processing flow

[0842] 1. Data entry phase:

[0843] Users input information about the food items in their refrigerator and their expiration dates using a smartphone or other device. For example, they might enter "Pork, expiration date 2023-10-20" and "Cabbage, expiration date 2023-10-18" into the app's input form.

[0844] The terminal converts the input data into JSON format and sends it to the server as an HTTP POST request.

[0845] 2. Data Processing Phase:

[0846] The server receives and parses data sent by the user. For example, it receives data in the format {"materials": [{"name": "Pork", "expiry_date": "2023-10-20"}, {"name": "Cabbage", "expiry_date": "2023-10-18"}]}.

[0847] The server searches the database for multiple recipes corresponding to the ingredients. For example, it retrieves relevant recipes such as "stir-fried vegetables" or "curry" from the database.

[0848] The server prioritizes recipes based on the expiration dates of the ingredients. Recipes that use ingredients with shorter expiration dates are placed higher in the list.

[0849] The server takes into account the user's past selection history and preferences to ultimately select the most suitable recipe.

[0850] 3. Data display phase:

[0851] The server generates detailed information about the selected optimal recipe in JSON format and sends it to the terminal. For example, it sends data in the format {"menu": {"name": "Stir-fried pork and cabbage", "instructions": "Chop the ingredients and stir-fry them in a frying pan"}}.

[0852] The terminal analyzes the received data and displays a suggested menu and its preparation method to the user. It displays "Today's suggested menu: Stir-fried pork and cabbage, Preparation method: Cut the ingredients and stir-fry them in a frying pan."

[0853] The user reviews the displayed information and prepares the meal based on the suggested menu.

[0854] Specific example

[0855] For example, if a user inputs that they have "pork," "cabbage," and "carrots" in their refrigerator, the server searches its database for recipes using these ingredients and prioritizes them based on their expiration dates. As a result, it provides the user with a suggested menu item, "stir-fried pork and cabbage," along with detailed instructions on how to make it. By following these instructions, the user can efficiently utilize the ingredients in their refrigerator.

[0856] This system enables effective management and efficient use of food in the refrigerator, significantly reducing food waste. Furthermore, it reduces the effort required for daily cooking, allowing users to achieve a healthier and more planned diet.

[0857] The following describes the processing flow.

[0858] Step 1:

[0859] The user collects information on multiple food items in their refrigerator, along with their expiration dates. Using a smartphone or other device, the user enters information such as "Pork, expiration date 2023-10-20" and "Cabbage, expiration date 2023-10-18" into the application's form.

[0860] Step 2:

[0861] The terminal converts the entered ingredient information into JSON format data. For example, it will be in the format {"materials": [{"name": "Pork", "expiry_date": "2023-10-20"}, {"name": "Cabbage", "expiry_date": "2023-10-18"}]}.

[0862] Step 3:

[0863] The terminal sends data to the server. The transmission method uses an HTTP POST request. For example, the request body contains the JSON data mentioned earlier.

[0864] Step 4:

[0865] The server receives data sent from the terminal. It parses the HTTP request, obtains JSON data, and then parses it. For example, it might obtain information about each ingredient in array format as the parsed data.

[0866] Step 5:

[0867] The server accesses the database and searches for relevant recipes based on the received ingredient information. The database stores detailed information and lists of required ingredients for each recipe. For example, it retrieves recipes such as "stir-fried vegetables" and "curry" that include "pork" and "cabbage."

[0868] Step 6:

[0869] The server prioritizes the retrieved recipes based on the expiration dates of each ingredient. For example, it compares the expiration dates of ingredients and sorts recipes prioritizing those with ingredients that need to be consumed the fastest.

[0870] Step 7:

[0871] The server selects the most suitable recipe by considering the user's past selection history and preferences. This enables personalized recipe suggestions for each user. For example, if "stir-fried vegetables" is determined to be the most suitable, that recipe will be selected.

[0872] Step 8:

[0873] The server generates detailed information about the selected recipe in JSON format. For example, it creates data in the format {"menu": {"name": "Stir-fried pork and cabbage", "instructions": "Chop the ingredients and stir-fry them in a frying pan"}}.

[0874] Step 9:

[0875] The server sends the generated JSON data to the terminal as an HTTP response. The response contains the selected recipe and its details.

[0876] Step 10:

[0877] The terminal analyzes the response data received from the server. It parses the JSON data to obtain the suggested menu and how to make it.

[0878] Step 11:

[0879] The device displays suggested menus and instructions for preparation to the user based on the analyzed data. For example, it might display something like, "Today's suggested menu: Stir-fried pork and cabbage, Instructions: Cut the ingredients and stir-fry them in a pan."

[0880] Step 12:

[0881] The user reviews the displayed information and prepares the dish based on the suggested menu. For example, they might cut pork and cabbage, then stir-fry them in a pan to complete the dish.

[0882] (Example 1)

[0883] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0884] Because the management and efficient use of ingredients in refrigerators are unclear, there is a need for a system that eliminates food waste and suggests the optimal cooking method based on the expiration date of the ingredients. Furthermore, conventional systems have not been able to provide suggestions that take into account the user's past selection history and preferences.

[0885] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0886] In this invention, the server includes means for inputting ingredient information, means for inputting expiration date information of the ingredient, means for retrieving multiple cooking methods using the ingredient from a data storage means based on the input information, means for determining priority based on the expiration date information of the ingredient and arranging each cooking method according to priority, and means for proposing the highest priority cooking method to the user and displaying how to prepare the cooking method. This makes it possible to efficiently utilize ingredients in the refrigerator, reduce food waste, and propose appropriate cooking methods that take into account the user's preferences.

[0887] "Means for inputting food information" refers to a device or interface for users to input the types and names of food items inside a refrigerator.

[0888] "Means for inputting expiration date information" refers to a device or interface for users to input the best-before date or expiration date for each food item.

[0889] "Data storage means" refers to a database or storage device for storing and managing information on ingredients, expiration dates, and multiple cooking methods based on these.

[0890] "Means for determining and arranging priorities" refers to a device or program that has the function of setting the priority of each cooking method based on the input expiration date information and listing them in that order.

[0891] "Means for proposing and displaying cooking methods" refers to a device or interface that presents the user with the determined optimal cooking method and displays the detailed preparation procedure.

[0892] "Communication means" refers to a network or communication interface used to transmit data input from a terminal to the main control unit.

[0893] The "main control unit" is a central control unit or server that receives and analyzes data and selects the optimal cooking method.

[0894] "Means of selecting based on the user's past selection history and preferences" refers to a device or program that has the function of recording the cooking methods and preferences previously chosen by the user and selecting the optimal cooking method taking that into consideration.

[0895] This invention is a system that manages information about food items in a refrigerator and their expiration dates, and suggests the optimal cooking method. The following describes specific embodiments for carrying out this invention.

[0896] This system allows users to input each ingredient in their refrigerator and its expiration date, and the server then searches for and suggests the most suitable cooking method based on that data. This enables users to efficiently utilize the ingredients in their refrigerator and reduce food waste.

[0897] System Configuration

[0898] 1. User terminal:

[0899] This is a device used by users to input food ingredient information and expiration date information. This includes smartphones, tablets, or PCs. Users launch a dedicated app and enter data through input forms such as text boxes.

[0900] 2. Server:

[0901] The server is the central component that receives data sent from the user terminal and searches for and suggests appropriate cooking methods based on that data. The server accesses the database, retrieves the relevant cooking methods, and provides data according to the user's request.

[0902] 3. Database:

[0903] The database is a system for storing ingredients, cooking methods, and related information. It manages details of each cooking method, required ingredients, and their corresponding expiration dates.

[0904] Specific example

[0905] For example, if a user inputs that they have "pork," "cabbage," and "carrots" in their refrigerator, the server searches its database for cooking methods using these ingredients and prioritizes them based on their expiration dates. As a result, it provides the user with a suggested cooking method, such as "stir-fried pork and cabbage," along with a detailed explanation of how to make it. By cooking based on this suggestion, the user can efficiently utilize the ingredients in their refrigerator.

[0906] Example of a prompt

[0907] Please enter the information about the ingredients in your refrigerator and their expiration dates. For example, enter it in the format of 'Pork, Expiration Date 2023-10-20', 'Cabbage, Expiration Date 2023-10-18', etc. We will then suggest the best cooking method.

[0908] This prompt allows users to easily operate the system and input ingredient and expiration date information. This enables the system to provide appropriate cooking instructions and reduce food waste.

[0909] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0910] Step 1: User enters ingredient information.

[0911] Users open their smartphones or tablets and launch the dedicated app. They then enter food information and expiration dates into the app's input form, such as "Pork, Expiration Date 2023-10-20" or "Cabbage, Expiration Date 2023-10-18". Specifically, users type the information into the text box and press the submit button to complete the input.

[0912] Input: Ingredient information (e.g., "Pork, Cabbage") and its expiration date (e.g., "2023-10-20, 2023-10-18")

[0913] Output: User-input text data

[0914] Step 2: Data conversion and transmission by the terminal

[0915] The device converts the ingredient information entered by the user into JSON format. For example, it formats it into the following format: {"materials": [{"name": "Pork", "expiry_date": "2023-10-20"}, {"name": "Cabbage", "expiry_date": "2023-10-18"}]}. Next, it sends this data to the server as an HTTP POST request. Specifically, the device sends the data to the API endpoint over the internet.

[0916] Input: User-inputted text data

[0917] Output: Data in JSON format and an HTTP POST request

[0918] Step 3: Server receives and analyzes data.

[0919] The server receives an HTTP request from the terminal and extracts and parses the data sent from the payload. Specifically, it parses JSON data in the format {"materials": [{"name": "Pork", "expiry_date": "2023-10-20"}, {"name": "Cabbage", "expiry_date": "2023-10-18"}]} to obtain the name of each ingredient and its expiration date.

[0920] Input: Data in JSON format

[0921] Output: Information on the names of ingredients and their corresponding expiration dates.

[0922] Step 4: Search for cooking methods using the server

[0923] The server uses the analyzed ingredient names to query the database and search for corresponding cooking methods. For example, to retrieve "recipes using pork and cabbage" from the database, it sends an SQL query. Specifically, it executes the query and returns a list of multiple cooking methods.

[0924] Input: Name of ingredients (e.g., "Pork, cabbage")

[0925] Output: List of suggested cooking methods

[0926] Step 5: Server prioritizes cooking method

[0927] The server evaluates the ingredient list and expiration dates for each cooking method and sets priorities. Specifically, it prioritizes cooking methods that use ingredients with shorter expiration dates. Based on the calculated priorities, it sorts the list of cooking methods.

[0928] Input: A list of suggested cooking methods and information on ingredients and their expiration dates.

[0929] Output: List of cooking methods sorted by priority

[0930] Step 6: Server selects the optimal cooking method

[0931] The server selects the optimal cooking method from among the high-priority options, taking into account the user's past selection history and preferences. For example, if the user previously preferred "stir-fry," the server will select "stir-fried pork and cabbage."

[0932] Input: A list of cooking methods sorted by priority, user selection history, and preference data.

[0933] Output: Optimal cooking method

[0934] Step 7: Server generates and transmits recipe information.

[0935] The server generates detailed information in JSON format about the selected optimal cooking method. For example, it might generate it in the format {"menu": {"name": "Stir-fried pork and cabbage", "instructions": "Cut the ingredients and stir-fry them in a frying pan"}} and send this to the terminal as an HTTP response.

[0936] Input: Optimal cooking method

[0937] Output: Detailed cooking instructions (in JSON format) and HTTP response

[0938] Step 8: Displaying the recipe on the device

[0939] The device parses the JSON data received from the server and displays the suggested cooking method and instructions to the user. Specifically, it displays "Today's suggested menu: Stir-fried pork and cabbage, Instructions: Cut the ingredients and stir-fry in a frying pan."

[0940] Input: Detailed cooking instructions (JSON format)

[0941] Output: Display of cooking instructions and preparation methods to the user.

[0942] Step 9: User performs cooking

[0943] The user checks the displayed cooking method and instructions, and then prepares the dish based on the ingredients in their refrigerator. Specifically, they prepare the ingredients according to the provided steps and complete the dish.

[0944] Input: Detailed cooking instructions and ingredients in the refrigerator

[0945] Output: Finished dish

[0946] (Application Example 1)

[0947] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0948] In modern times, there is a need to efficiently manage the food in refrigerators and reduce the amount of food that goes past its expiration date. However, it is difficult for users to keep track of the food in their refrigerators and use it appropriately in their daily lives, and food waste often occurs. Also, if necessary ingredients are missing, people have to go shopping each time, which is time-consuming and troublesome. To solve this problem, a system is needed that efficiently manages information about the food in the refrigerator and has an automatic ordering function for missing ingredients.

[0949] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0950] In this invention, the server includes means for inputting information on multiple food items present in the refrigerator, means for inputting the expiration dates of the food items, means for searching a database for multiple recipes using the food items based on the input information, means for determining priority based on the expiration dates of the food items and sorting each recipe according to priority, means for suggesting the highest-priority recipe to the user and displaying the instructions for making that recipe, and means for automatically ordering any missing ingredients based on the information on the food items. This enables centralized management of food item information in the refrigerator, allowing for optimal recipe suggestions to the user and automatic ordering of necessary ingredients.

[0951] "Means for inputting information on multiple food items present in a refrigerator" refers to devices or applications that allow users to input information such as the names, quantities, and locations of various food items stored in a refrigerator.

[0952] "Means for inputting the expiration date of the food item" refers to a device or application for the user to input the expiration date of each food item in the refrigerator.

[0953] "Means for searching a database for multiple recipes using the ingredients in question based on the input information" refers to a server or software that searches a database for multiple recipes using the ingredients entered by the user.

[0954] "Means for determining priority based on the expiration date of the ingredient and sorting each recipe according to priority" refers to a server or algorithm that determines whether an ingredient should be used preferentially based on the expiration date information of the input ingredient, and then sorts the corresponding recipes in order of priority.

[0955] "A means of suggesting a top-priority recipe to a user and displaying how to make that recipe" refers to a device or application that presents the user with the highest priority recipe and displays the user's detailed instructions for making that recipe.

[0956] "Means for automatically ordering missing ingredients based on information about the ingredients in question" refers to a server or software that automatically determines the missing ingredients needed for a dish based on ingredient information entered by the user and orders them online.

[0957] "A means of communication for sending data entered from a terminal to a server" refers to a communication interface, protocol, or system for sending information about ingredients and expiration dates from a terminal used by a user to a server.

[0958] "A means of selecting the optimal recipe from multiple recipes based on the user's past selection history and preferences" refers to an algorithm or software that analyzes the user's past selection history and preference information and suggests the most suitable recipe based on that information.

[0959] This invention relates to a system that manages the remaining ingredients in a refrigerator and their expiration dates, suggests the optimal menu and how to prepare it, and can automatically order any missing ingredients from an online shopping site.

[0960] System Configuration

[0961] 1. Method for entering information about the ingredients in the refrigerator:

[0962] Users use devices such as smartphones or tablets to input information about the food items in their refrigerator and their expiration dates. For example, they might enter "Pork, expiration date 2023-10-20" and "Cabbage, expiration date 2023-10-18" into the app's input form.

[0963] 2. Means of communication:

[0964] The aforementioned terminal sends the input data to the server via HTTP communication. It uses JSON format data for communication and sends an HTTP POST request.

[0965] 3. Means of searching for recipes based on input information:

[0966] The server analyzes the received data and searches the database for a suitable recipe based on the information about the ingredients in the refrigerator. For example, the server analyzes the received JSON data and retrieves the corresponding recipe from the database.

[0967] 4. Means for determining and sorting recipe priority:

[0968] The server prioritizes ingredients that should be consumed as soon as possible, based on their expiration dates. It then sorts the searched recipes according to this priority.

[0969] 5. Means for suggesting the highest priority recipe to the user and displaying the instructions:

[0970] The server selects the highest-priority recipe from the sorted list and sends its detailed instructions to the terminal. For example, the selected recipe for "Stir-fried Pork and Cabbage" and its instructions will be displayed.

[0971] 6. A means of automatically ordering missing materials:

[0972] The server matches the entered ingredient information with the suggested recipe and automatically orders any necessary ingredients that are not in the refrigerator from an online shopping site. For example, if the customer only has pork and cabbage and does not have carrots, which are needed for "stir-fried pork and cabbage," the server will automatically add carrots to the shopping cart from the online store.

[0973] Program Processing Description

[0974] The server uses a Python program to suggest recipes based on user input and automatically order any missing ingredients. The server uses the Python Requests library for communication. SQL is used to search for recipes in the database. Algorithms are implemented for prioritizing and sorting recipes. Furthermore, a machine learning model is used to select the optimal recipe based on the user's past selection history and preferences.

[0975] For example, if a user enters "pork, cabbage" along with expiration date information through a smartphone app, that information is sent to the server. The server then searches for a suitable recipe and suggests "stir-fried pork and cabbage," prioritizing ingredients with the earliest expiration dates. If carrots are missing in the recipe, they are automatically ordered from an online shopping site.

[0976] Example of a prompt

[0977] Please enter the ingredients in your refrigerator (e.g., "pork," "cabbage") and their expiration dates. Based on this information, we will automatically order the necessary ingredients from an online shopping site and suggest the best recipes.

[0978] As a result, this invention allows for efficient management of ingredients in the refrigerator, reduces food waste, and makes meal preparation easier for the user.

[0979] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0980] Step 1:

[0981] Users input information about the food items in their refrigerator and their expiration dates using a smartphone, tablet, or other device.

[0982] Specific examples of the information to be entered include "Pork, Expiration Date 2023-10-20" and "Cabbage, Expiration Date 2023-10-18". This will be the input data.

[0983] Step 2:

[0984] The terminal converts the entered ingredient information into JSON format and sends it to the server as an HTTP POST request.

[0985] This communication method utilizes the JSON format and the HTTP protocol. The data is sent in the format {"materials": [{"name": "Pork", "expiry_date": "2023-10-20"}, {"name": "Cabbage", "expiry_date": "2023-10-18"}]}. This serves as the input data, and the server performs the analysis.

[0986] Step 3:

[0987] The server receives an HTTP POST request and parses the received data.

[0988] The analyzed data extracts the expiration dates of each food item in the refrigerator. Based on the input data, a query is generated for the database to search for the corresponding recipe. For example, it searches for recipes using "pork" and "cabbage".

[0989] Step 4:

[0990] The server receives multiple recipes as search results from the database, determines a priority based on the expiration dates of the ingredients in each recipe, and sorts the recipes.

[0991] This sorting is performed using an algorithm that prioritizes recipes that use ingredients nearing their expiration date. For example, "Stir-fried pork and cabbage" would be one such recipe.

[0992] Step 5:

[0993] The server selects the most suitable recipe from several options based on the user's past selection history and preferences.

[0994] This process utilizes machine learning models. Based on past selection history data and preference data, it selects the most suitable recipe and prioritizes it to determine the best option.

[0995] Step 6:

[0996] The server generates detailed information about the selected top-priority recipe in JSON format and sends it to the terminal.

[0997] For example, send the following in the format: {"menu": {"name": "Stir-fried pork and cabbage", "instructions": "Cut the ingredients and stir-fry them in a frying pan"}}. This will be the output data.

[0998] Step 7:

[0999] The device analyzes the received recipe information and displays suggested menus and their preparation methods to the user.

[1000] Specifically, it displays: "Today's suggested menu: Stir-fried pork and cabbage, Instructions: Cut the ingredients and stir-fry them in a frying pan."

[1001] Step 8:

[1002] The server automatically orders any missing ingredients from online shopping sites based on the suggested recipe.

[1003] The server generates a list of all the ingredients needed for the suggested recipe and determines which ingredients are missing from the refrigerator. It then automatically orders the missing ingredients through the online shopping site's API. For example, it adds "carrots" to the online shopping site's shopping cart.

[1004] Step 9:

[1005] The user prepares the meal according to the suggested menu and instructions. They also confirm that any missing ingredients are automatically ordered from an online store, and the purchasing process proceeds. Thus, the user's meal preparation is complete.

[1006] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1007] This invention combines an emotional engine with a system that manages remaining ingredients in a refrigerator and their expiration dates, and suggests the optimal menu and preparation method. The following describes specific embodiments for carrying out the invention.

[1008] overview

[1009] This system allows users to input each ingredient in their refrigerator and its expiration date, and the server then searches for and suggests the most suitable recipe based on that data. Furthermore, it can use an emotion engine to recognize the user's emotional state and provide recipes tailored to that state.

[1010] System Configuration

[1011] 1. User terminal:

[1012] The user terminal is a device used to input information about the food items in the refrigerator and their expiration dates. It also includes voice and image analysis functions to input the user's emotional state. Examples include smartphones, tablets, and PCs.

[1013] 2. Server:

[1014] The server is the central component that receives ingredient information and emotional states sent from the user's terminal and searches for and suggests appropriate recipes based on that information. The server accesses the database to retrieve appropriate recipes and provides data according to the user's request.

[1015] 3. Database:

[1016] The database is a system for storing ingredients, recipes, and related information. It manages details of each recipe, required ingredients and their corresponding expiration dates, as well as data on the user's past selection history and emotional state.

[1017] 4. Emotional Engine:

[1018] The emotion engine is a system for recognizing a user's emotional state. It includes voice analysis and image analysis methods to analyze the user's emotions from their voice and facial expressions and transmit that data to a server.

[1019] Specific processing flow

[1020] Data entry phase

[1021] 1. The user enters information about the food items in their refrigerator and their expiration dates using a smartphone or other device. For example, they might enter information such as "Pork, expiration date 2023-10-20" or "Cabbage, expiration date 2023-10-18" into the application's input form.

[1022] 2. The user inputs their emotional state through the emotion engine. The emotional state is analyzed by capturing facial expressions using voice input or a camera.

[1023] Data transmission phase

[1024] 3. The terminal converts the entered food information and emotional state into JSON data. For example, {"materials": [{"name": "Pork", "expiry_date": "2023-10-20"}, {"name": "Cabbage", "expiry_date": "2023-10-18"}], "emotion": {"mood": "happy"}}.

[1025] 4. The terminal sends data to the server. The transmission method is an HTTP POST request.

[1026] Data Processing Phase

[1027] 5. The server receives data sent from the terminal. It parses the HTTP request, obtains the JSON data, and then parses it.

[1028] 6. The server accesses the database and searches for a suitable recipe based on the received ingredient information and emotional state. For example, it retrieves recipes such as "stir-fried vegetables" or "curry" that contain "pork" and "cabbage" from the database.

[1029] 7. The server determines the priority of the retrieved recipes based on the expiration date of each ingredient. For example, it compares the expiration dates of the ingredients and sorts the recipes with ingredients that need to be consumed the fastest to the top.

[1030] 8. The server selects the optimal recipe by considering the user's past selection history and preferences, as well as their emotional state recognized by the emotion engine. For example, it will suggest more enjoyable dishes for a "happy" emotional state.

[1031] Proposed menu generation phase

[1032] 9. The server generates detailed information about the selected recipe in JSON format. For example, it creates data in the format {"menu": {"name": "Stir-fried pork and cabbage", "instructions": "Chop the ingredients and stir-fry them in a pan"}}.

[1033] 10. The server sends the generated JSON data to the terminal as an HTTP response. The response contains the selected recipe and its details.

[1034] Data display phase

[1035] 11. The terminal analyzes the response data received from the server. It parses the JSON data to obtain the suggested menu and how to make it.

[1036] 12. The terminal displays suggested menus and instructions for preparation to the user based on the analyzed data. For example, it might display "Today's suggested menu: Stir-fried pork and cabbage, Instructions: Cut the ingredients and stir-fry them in a pan."

[1037] 13. The user reviews the displayed information and prepares the dish based on the suggested menu. For example, they might cut pork and cabbage, then stir-fry them in a pan to complete the dish.

[1038] Specific example

[1039] For example, if a user inputs that they have "pork" and "cabbage" in their refrigerator, and the emotion engine recognizes a "happy" emotional state, the server searches its database for recipes using these ingredients, prioritizing them based on their expiration dates. As a result, it provides the user with a suggested menu item, "stir-fried pork and cabbage," which is appropriate for their happy emotional state, and explains how to make it in detail. By following these instructions, the user can efficiently utilize the ingredients in their refrigerator.

[1040] This system not only allows for the effective management and use of ingredients in the refrigerator, but also enables personalized recipe suggestions tailored to the user's emotional state. This makes the daily cooking experience more enjoyable and satisfying, contributing to a reduction in food waste.

[1041] The following describes the processing flow.

[1042] Step 1:

[1043] The user collects information on multiple food items in their refrigerator, along with their expiration dates. Using a smartphone or other device, the user enters information such as "Pork, expiration date 2023-10-20" and "Cabbage, expiration date 2023-10-18" into the application's form.

[1044] Step 2:

[1045] Users record their emotional state through an emotion engine. For example, their emotional state is analyzed by capturing facial expressions using voice input or a camera.

[1046] Step 3:

[1047] The terminal converts the entered food information and emotional state into JSON data. For example, the format will be {"materials": [{"name": "Pork", "expiry_date": "2023-10-20"}, {"name": "Cabbage", "expiry_date": "2023-10-18"}], "emotion": {"mood": "happy"}}.

[1048] Step 4:

[1049] The terminal sends the converted data to the server. The transmission method uses an HTTP POST request. For example, the request body contains the JSON data mentioned earlier.

[1050] Step 5:

[1051] The server receives data sent from the terminal. The server parses the HTTP request, obtains JSON data, and parses it. For example, it might obtain information about each ingredient in array format as the parsed data.

[1052] Step 6:

[1053] The server accesses the database and searches for relevant recipes based on the received ingredient information and emotional state. The database stores detailed information and required ingredient lists for each recipe. For example, it retrieves recipes such as "stir-fried vegetables" and "curry" that contain "pork" and "cabbage."

[1054] Step 7:

[1055] The server prioritizes the retrieved recipes based on the expiration dates of each ingredient. For example, it compares the expiration dates of ingredients and sorts recipes prioritizing those with ingredients that need to be consumed the fastest.

[1056] Step 8:

[1057] The server selects the optimal recipe by considering the user's past selection history and preferences, as well as their emotional state recognized by the emotion engine. For example, it suggests more enjoyable dishes for a "happy" emotional state.

[1058] Step 9:

[1059] The server generates detailed information about the selected recipe in JSON format. For example, it creates data in the format {"menu": {"name": "Stir-fried pork and cabbage", "instructions": "Chop the ingredients and stir-fry them in a frying pan"}}.

[1060] Step 10:

[1061] The server sends the generated JSON data to the terminal as an HTTP response. The response contains the selected recipe and its details.

[1062] Step 11:

[1063] The terminal analyzes the response data received from the server. It parses the JSON data to obtain the suggested menu and how to make it.

[1064] Step 12:

[1065] The device displays suggested menus and instructions for preparation to the user based on the analyzed data. For example, it might display something like, "Today's suggested menu: Stir-fried pork and cabbage, Instructions: Cut the ingredients and stir-fry them in a pan."

[1066] Step 13:

[1067] The user reviews the displayed information and prepares the dish based on the suggested menu. For example, they might cut pork and cabbage, then stir-fry them in a pan to complete the dish.

[1068] (Example 2)

[1069] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1070] Modern households store a wide variety of ingredients in their refrigerators, but efficiently utilizing these ingredients is difficult. In particular, it is hard to plan for consuming ingredients that are nearing their expiration date, leading to food waste. Furthermore, the lack of meal suggestions tailored to the user's emotional state on any given day diminishes the enjoyment and satisfaction of daily cooking. Therefore, there is a need for a system that can suggest optimal recipes based on ingredient information and emotional state, thereby reducing food waste and providing a cooking experience that matches the user's emotions.

[1071] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for inputting information on multiple food items present in the refrigerator, means for inputting the expiration dates of the food items, and means for inputting the user's emotional state. This makes it possible to search a database for multiple recipes using the food items based on the input information and emotional state, determine priority based on the expiration dates of the food items, sort each recipe according to priority, select the optimal recipe considering the user's past selection history, preferences, and emotional state, propose the highest priority recipe to the user, and display how to make it. This makes it possible to reduce food waste and provide personalized cooking suggestions that are tailored to the user's emotions.

[1072] "A means of inputting information on multiple food items present in a refrigerator" refers to a device or software that allows a user to input the types of food items stored in the refrigerator and their respective expiration dates into a terminal.

[1073] "Means for inputting the expiration date of the food item" refers to a device or software that allows the user to input the expiration date of each food item into a terminal.

[1074] "Means for inputting the user's emotional state" refers to devices or software that analyze the user's voice and facial expressions to recognize their emotional state and input that information into the system.

[1075] "Means for searching a database for multiple recipes using the ingredient in question based on input information and emotional state" refers to an algorithm or software for searching a database for relevant recipes using ingredient information and emotional state data received from a terminal.

[1076] "Means for determining priority based on the expiration date of the ingredients and sorting each recipe according to priority" refers to an algorithm or software that determines the priority of recipes based on the expiration date of the input ingredients and rearranges the recipes accordingly.

[1077] "Means for selecting the optimal recipe by considering the user's past selection history, preferences, and emotional state" refers to algorithms or software that use data on the user's past recipe selection history and emotional state to determine and select the most suitable recipe for the user.

[1078] "Means for suggesting a top-priority recipe to the user and displaying how to make that recipe" refers to devices or software that notify the user of a high-priority recipe and display its detailed instructions.

[1079] "A means of communication for sending data entered from a terminal to a server" refers to network communication devices and protocols for sending food information and emotional states entered from a user's terminal to a server.

[1080] An "emotion engine" is an algorithm or software that analyzes a user's voice and facial expressions to identify their emotional state.

[1081] This invention relates to a system that provides an optimal recipe by taking into account information about the ingredients in the refrigerator, their expiration dates, and the user's emotional state. The specific methods for implementing this invention are described below.

[1082] System Configuration

[1083] This system consists of user terminals, a server, a database, and an emotion engine. Details of each component are as follows:

[1084] User terminal

[1085] The user terminal is a device used to input information about the food items in the refrigerator and their expiration dates. It also includes voice and image analysis functions for inputting the user's emotional state. Examples include smartphones, tablets, and PCs.

[1086] server

[1087] The server is the central component that receives ingredient information and emotional states sent from the user's terminal and searches for and suggests appropriate recipes based on that information. The server accesses a database to retrieve appropriate recipes and provides data according to the user's request. Specifically, Apache servers and Nginx servers are often used.

[1088] database

[1089] A database is a system for storing ingredients, recipes, and related information. It manages details of each recipe, required ingredients and their corresponding expiration dates, as well as data on the user's past selection history and emotional state. Specifically, relational databases such as MySQL and PostgreSQL are used.

[1090] Emotional Engine

[1091] An emotion engine is a system for recognizing a user's emotional state. It includes voice and image analysis methods to analyze emotions from the user's voice and facial expressions and send that data to a server. Specifically, it uses tools such as OpenCV and the Google Cloud Speech-to-Text API.

[1092] Specific example

[1093] The following specific examples will help to understand the present invention.

[1094] For example, if a user inputs that they have "pork" and "cabbage" in their refrigerator, and the emotion engine recognizes a "happy" emotional state, the server searches its database for recipes using these ingredients, prioritizing them based on their expiration dates. As a result, it generates a suggested menu item, such as "Pork and Cabbage Pasta," which is adapted to the user's happy emotional state. The server then sends this recipe and detailed instructions to the user's terminal, allowing the user to begin cooking.

[1095] Example of a prompt

[1096] The following are specific examples of prompts to input into a generative AI model:

[1097] The refrigerator contains pork and cabbage. The current emotional state is "happy." Please suggest a recipe using these ingredients that is appropriate for this emotional state.

[1098] This invention enables effective management of ingredients in the refrigerator and provides optimal recipe suggestions, thereby reducing food waste and realizing a personalized cooking experience that responds to the user's emotions.

[1099] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1100] Step 1:

[1101] Users input information about the food items in their refrigerator and their expiration dates using a smartphone or other device. For example, they might enter "Pork, expiration date 2023-10-20" and "Cabbage, expiration date 2023-10-18" into the app's input form. The input data is output as JSON data from the device's input form.

[1102] Step 2:

[1103] Users input their emotional state through the emotion engine. For example, they might use the voice input function to say "I feel good today," and the emotion engine analyzes the voice and recognizes the user's emotional state as "happy." The input data is output from the emotion engine as JSON data.

[1104] Step 3:

[1105] The terminal integrates the entered food information and emotional state to generate final JSON data. For example, it formats the data as follows: {"materials": [{"name": "Pork", "expiry_date": "2023-10-20"}, {"name": "Cabbage", "expiry_date": "2023-10-18"}], "emotion": {"mood": "happy"}}. The integrated data is then sent from the terminal to the server.

[1106] Step 4:

[1107] The device sends the generated JSON data to the server using an HTTP POST request. The detailed communication protocol used is HTTP / HTTPS. The transmitted data is received by the server.

[1108] Step 5:

[1109] The server parses the received HTTP request and retrieves JSON data. Specifically, it extracts JSON data from the body of the HTTP request. The extracted data is then passed to an internal processing module.

[1110] Step 6:

[1111] The server accesses the database and searches for relevant recipes based on the received ingredient information. For example, it searches the database for recipes containing "pork" and "cabbage" using an SQL query. The search results return a list of relevant recipes.

[1112] Step 7:

[1113] The server compares the expiration dates of each ingredient in the retrieved recipes to determine priority. For example, to prioritize ingredients with approaching expiration dates, it uses the SQL ORDER BY clause to sort the recipes. A sorted list of recipes is then generated.

[1114] Step 8:

[1115] The server selects the optimal recipe by considering the user's past selection history and preferences. It also takes into account the emotional state recognized by the emotion engine. For example, it might select the best recipe from those the user has previously chosen when they were in a "happy" emotional state. One optimal recipe is then selected.

[1116] Step 9:

[1117] The server generates detailed information about the selected recipe in JSON format. For example, it creates data in the format {"menu": {"name": "Pork and cabbage pasta", "instructions": "Chop the ingredients, boil the pasta, and stir-fry in a pan"}}. The generated data is then sent to the terminal.

[1118] Step 10:

[1119] The server sends the generated JSON data to the terminal as an HTTP response. The response data is sent via the HTTP / HTTPS protocol.

[1120] Step 11:

[1121] The terminal analyzes the response data received from the server and parses the data in JSON format. For example, it analyzes data such as {"menu": {"name": "Pork and cabbage pasta", "instructions": "Cut the ingredients, boil the pasta, and stir-fry in a pan"}}. The analysis results are stored as an internal data structure.

[1122] Step 12:

[1123] The device displays suggested menus and instructions for preparation based on the analyzed data. For example, it might display: "Today's suggested menu: Pork and cabbage pasta, Instructions: Cut the ingredients, boil the pasta, and stir-fry in a pan."

[1124] Step 13:

[1125] The user cooks according to the displayed recipe. For example, they might cut pork and cabbage, boil pasta, and then stir-fry it in a pan to complete the dish.

[1126] (Application Example 2)

[1127] Next, we will explain Application Example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1128] Traditional refrigerator management systems, while capable of managing information about ingredients and expiration dates, were unable to provide personalized recipe suggestions based on the user's emotional state. This led to decreased user satisfaction and food waste. Furthermore, the suggested recipes often didn't match the user's mood or preferences at the time, resulting in a lack of enjoyment in daily cooking.

[1129] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[1130] In this invention, the server includes means for inputting information on multiple food items present in the refrigerator, means for inputting the expiration dates of the food items, means for determining priority based on the expiration dates of the food items and recognizing the user's emotional state, and means for optimizing recipes based on the emotional state. This enables the suggestion of optimal recipes according to the user's emotional state, improving the daily cooking experience and reducing food waste.

[1131] "Information on multiple food items present in the refrigerator" refers to data about various food items stored in the refrigerator, including their names, quantities, acquisition dates, and other attributes.

[1132] The "best before date" for a food item refers to the date by which the food stored in the refrigerator should be consumed appropriately, and this date varies depending on the food item.

[1133] "Means of input" refers to devices or methods for users to register information about the food in their refrigerator and its expiration date into the system, such as input devices like smartphones, tablets, or PCs.

[1134] "Methods for searching a database" refer to programs or algorithms that extract information from a database based on specific conditions and provide it to the user.

[1135] "Means for determining priority and sorting each recipe according to priority" refers to a method for determining the priority of multiple recipes based on expiration dates or other conditions, and then rearranging the recipes based on the results.

[1136] "Means for recognizing a user's emotional state" refers to devices or software that analyze a user's facial expressions, voice, etc., to evaluate their psychological state at that time.

[1137] "Methods for optimizing recipes based on emotional state" refer to methods or systems that select recipes and suggest the most suitable dishes by reflecting the recognized emotional state of the user.

[1138] "A means of suggesting the highest-priority recipe to the user and displaying how to make that recipe" refers to a function that presents the user with the most suitable recipe selected by the system and provides a detailed explanation of how to cook it.

[1139] This invention combines an emotional engine with a system that manages the remaining ingredients in a refrigerator and their expiration dates, and suggests the optimal menu and how to prepare it. The following describes specific embodiments for carrying out the invention.

[1140] System Configuration

[1141] 1. User terminal

[1142] The user terminal is a device used to input information about the food items in the refrigerator and their expiration dates. It also includes voice and image analysis functions to input the user's emotional state. Smartphones, tablets, and PCs are examples of this type of device. The terminal also has a means of communication to send the input data to the server.

[1143] 2. Server

[1144] The server is the central component that receives ingredient information and emotional states sent from the user's terminal and searches for and suggests appropriate recipes based on that information. The server accesses the database to retrieve appropriate recipes and provides data according to the user's request.

[1145] The software used includes Flask (a web server framework) and other web server frameworks.

[1146] 3. Database

[1147] The database is a system for storing ingredients, recipes, and related information. It manages details of each recipe, required ingredients and their corresponding expiration dates, as well as data on the user's past selection history and emotional state.

[1148] 4. Emotional Engine

[1149] An emotion engine is a system for recognizing a user's emotional state and includes methods for speech analysis and image analysis. Specifically, speech recognition APIs (e.g., Google Speech-to-Text API) are used for speech analysis, and libraries such as OpenCV are used for image analysis.

[1150] Specific processing flow

[1151] Users input information about the food items in their refrigerator and their expiration dates using a smartphone or other device. For example, they might enter information such as "Pork, expiration date 2023-10-20" or "Cabbage, expiration date 2023-10-18" into the application's input form. Next, they input their emotional state through an emotion engine, and their emotional state is analyzed by capturing facial expressions using voice input or a camera.

[1152] The terminal converts the entered ingredient information and emotional state into JSON data and sends it to the server. The server receives and analyzes this data, accesses the database, and searches for the corresponding recipe based on the received ingredient information and emotional state. Furthermore, it determines priorities based on the expiration date of each ingredient and selects the optimal recipe by considering the user's past selection history and preferences, as well as the emotional state recognized by the emotion engine.

[1153] Detailed information about the selected recipe is generated in JSON format and sent from the server to the terminal. The terminal parses it and displays the suggested menu and its preparation method to the user. By following these instructions, the user can efficiently utilize the ingredients in their refrigerator.

[1154] Examples of specific cases and prompt statements

[1155] Specific example

[1156] If a user inputs that they have "pork" and "cabbage" in their refrigerator and the emotion engine recognizes a "happy" emotional state, the server searches its database for recipes using these ingredients, prioritizing them based on their expiration dates. As a result, it provides the user with a suggested menu item, "Stir-fried pork and cabbage," which is appropriate for their happy emotional state, and explains how to make it in detail.

[1157] Example of a prompt

[1158] If you want to send the following data to the server, you can send data like this:

[1159] {

[1160] "materials": ["pork", "cabbage"],

[1161] "expiry_dates": ["2023-10-20", "2023-10-18"],

[1162] "image": <Image data of the user's facial expression captured by the camera>

[1163] }

[1164] In this way, users can not only effectively manage and utilize the ingredients in their refrigerator using the system, but also receive personalized recipe suggestions based on their emotional state. This makes the daily cooking experience more enjoyable and satisfying, and also contributes to reducing food waste.

[1165] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1166] Step 1:

[1167] The user enters information about the food items in their refrigerator and their expiration dates using a smartphone or other device.

[1168] Specific operation: The user launches the application and enters the name, quantity, and expiration date of the ingredients into the input form.

[1169] Input: Ingredient information (e.g., "Pork", "Cabbage") and expiration date (e.g., "2023-10-20", "2023-10-18")

[1170] Output: Input food information and expiration date data

[1171] Step 2:

[1172] Users input their emotional state through an emotion engine, and their facial expressions are captured using voice input or a camera.

[1173] Specific operation: The user uses the in-app camera function to take a picture of their face and have their emotional state recognized.

[1174] Input: User's facial expression images and audio data

[1175] Output: User's emotional state (e.g., "happy", "sad")

[1176] Step 3:

[1177] The terminal converts the entered food information and emotional state into JSON data and sends it to the server.

[1178] Specific operation: The terminal packages the ingredient information and emotional state into JSON format (for example, {"materials": ["pork", "cabbage"], "expiry_dates": ["2023-10-20", "2023-10-18"], "emotion": "happy"}) and sends it to the server using an HTTP POST request.

[1179] Input: Data in JSON format

[1180] Output: HTTP request sent to the server

[1181] Step 4:

[1182] The server receives and analyzes the data sent from the terminal.

[1183] Specific operation: The server receives an HTTP request and parses the JSON data to retrieve ingredient information, expiration date, and emotional state.

[1184] Input: JSON data sent from the device

[1185] Output: Analyzed food information, expiration date, emotional state

[1186] Step 5:

[1187] The server accesses the database and searches for the appropriate recipe based on the received ingredient information and emotional state.

[1188] Specific operation: The server executes a database query to retrieve a recipe suitable for the ingredients. The database stores recipes corresponding to the ingredients and emotional states.

[1189] Input: Analyzed food information, expiration date, emotional state

[1190] Output: List of relevant recipes

[1191] Step 6:

[1192] The server determines the priority of recipes based on the expiration date of each ingredient and sorts the recipes accordingly.

[1193] Specific operation: The server compares the expiration dates of ingredients and prioritizes recipes that include ingredients that need to be consumed the soonest.

[1194] Input: List of relevant recipes, expiration dates of ingredients

[1195] Output: High-priority recipe list

[1196] Step 7:

[1197] The server selects the optimal recipe by considering the user's past selection history, preferences, and emotional state.

[1198] Specific operation: The server retrieves the user's past selection history from the database and executes an algorithm to select the optimal recipe in conjunction with their emotional state.

[1199] Input: High-priority recipe list, user's past selection history, emotional state

[1200] Output: Optimal recipe

[1201] Step 8:

[1202] The server generates detailed information about the selected recipe in JSON format and sends it to the terminal.

[1203] Specific operation: The server formats the selected recipe and its details (e.g., ingredients, instructions) in JSON format and sends it to the terminal as an HTTP response.

[1204] Input: Optimal recipe

[1205] Output: Recipe data in JSON format

[1206] Step 9:

[1207] The terminal analyzes the response data received from the server and displays the suggested menu and its preparation method to the user.

[1208] Specific operation: The terminal parses the received JSON data and displays the recipe and cooking instructions on the screen.

[1209] Input: Recipe data in JSON format

[1210] Output: The recipe and its details displayed to the user

[1211] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1212] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1213] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[1214] [Fourth Embodiment]

[1215] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[1216] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1217] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1218] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[1219] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1220] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1221] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1222] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[1223] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1224] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1225] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1226] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1227] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1228] This invention relates to a system for managing leftover ingredients in a refrigerator and their expiration dates, and for suggesting optimal menus and their preparation methods. The following describes specific embodiments for carrying out the invention.

[1229] overview

[1230] This system allows users to input each ingredient in their refrigerator and its expiration date, and the server then searches for and suggests the most suitable recipe based on that data. Users can then cook based on these suggestions, efficiently utilizing the ingredients in their refrigerator.

[1231] System Configuration

[1232] 1. User terminal:

[1233] The user terminal is a device used to input information about the food items in the refrigerator and their expiration dates. This could be a smartphone, tablet, or PC.

[1234] 2. Server:

[1235] The server is the central component that receives ingredient information sent from the user's terminal and searches for and suggests appropriate recipes based on that information. The server accesses the database to retrieve appropriate recipes and provides data according to the user's request.

[1236] 3. Database:

[1237] The database is a system for storing ingredients, recipes, and related information. It manages details of each recipe, the required ingredients, and their corresponding expiration dates.

[1238] Specific processing flow

[1239] 1. Data entry phase:

[1240] Users input information about the food items in their refrigerator and their expiration dates using a smartphone or other device. For example, they might enter "Pork, expiration date 2023-10-20" and "Cabbage, expiration date 2023-10-18" into the app's input form.

[1241] The terminal converts the input data into JSON format and sends it to the server as an HTTP POST request.

[1242] 2. Data Processing Phase:

[1243] The server receives and parses data sent by the user. For example, it receives data in the format {"materials": [{"name": "Pork", "expiry_date": "2023-10-20"}, {"name": "Cabbage", "expiry_date": "2023-10-18"}]}.

[1244] The server searches the database for multiple recipes corresponding to the ingredients. For example, it retrieves relevant recipes such as "stir-fried vegetables" or "curry" from the database.

[1245] The server prioritizes recipes based on the expiration dates of the ingredients. Recipes that use ingredients with shorter expiration dates are placed higher in the list.

[1246] The server takes into account the user's past selection history and preferences to ultimately select the most suitable recipe.

[1247] 3. Data display phase:

[1248] The server generates detailed information about the selected optimal recipe in JSON format and sends it to the terminal. For example, it sends data in the format {"menu": {"name": "Stir-fried pork and cabbage", "instructions": "Chop the ingredients and stir-fry them in a frying pan"}}.

[1249] The terminal analyzes the received data and displays a suggested menu and its preparation method to the user. It displays "Today's suggested menu: Stir-fried pork and cabbage, Preparation method: Cut the ingredients and stir-fry them in a frying pan."

[1250] The user reviews the displayed information and prepares the meal based on the suggested menu.

[1251] Specific example

[1252] For example, if a user inputs that they have "pork," "cabbage," and "carrots" in their refrigerator, the server searches its database for recipes using these ingredients and prioritizes them based on their expiration dates. As a result, it provides the user with a suggested menu item, "stir-fried pork and cabbage," along with detailed instructions on how to make it. By following these instructions, the user can efficiently utilize the ingredients in their refrigerator.

[1253] This system enables effective management and efficient use of food in the refrigerator, significantly reducing food waste. Furthermore, it reduces the effort required for daily cooking, allowing users to achieve a healthier and more planned diet.

[1254] The following describes the processing flow.

[1255] Step 1:

[1256] The user collects information on multiple food items in their refrigerator, along with their expiration dates. Using a smartphone or other device, the user enters information such as "Pork, expiration date 2023-10-20" and "Cabbage, expiration date 2023-10-18" into the application's form.

[1257] Step 2:

[1258] The terminal converts the entered ingredient information into JSON format data. For example, it will be in the format {"materials": [{"name": "Pork", "expiry_date": "2023-10-20"}, {"name": "Cabbage", "expiry_date": "2023-10-18"}]}.

[1259] Step 3:

[1260] The terminal sends data to the server. The transmission method uses an HTTP POST request. For example, the request body contains the JSON data mentioned earlier.

[1261] Step 4:

[1262] The server receives data sent from the terminal. It parses the HTTP request, obtains JSON data, and then parses it. For example, it might obtain information about each ingredient in array format as the parsed data.

[1263] Step 5:

[1264] The server accesses the database and searches for relevant recipes based on the received ingredient information. The database stores detailed information and lists of required ingredients for each recipe. For example, it retrieves recipes such as "stir-fried vegetables" and "curry" that include "pork" and "cabbage."

[1265] Step 6:

[1266] The server prioritizes the retrieved recipes based on the expiration dates of each ingredient. For example, it compares the expiration dates of ingredients and sorts recipes prioritizing those with ingredients that need to be consumed the fastest.

[1267] Step 7:

[1268] The server selects the most suitable recipe by considering the user's past selection history and preferences. This enables personalized recipe suggestions for each user. For example, if "stir-fried vegetables" is determined to be the most suitable, that recipe will be selected.

[1269] Step 8:

[1270] The server generates detailed information about the selected recipe in JSON format. For example, it creates data in the format {"menu": {"name": "Stir-fried pork and cabbage", "instructions": "Chop the ingredients and stir-fry them in a frying pan"}}.

[1271] Step 9:

[1272] The server sends the generated JSON data to the terminal as an HTTP response. The response contains the selected recipe and its details.

[1273] Step 10:

[1274] The terminal analyzes the response data received from the server. It parses the JSON data to obtain the suggested menu and how to make it.

[1275] Step 11:

[1276] The device displays suggested menus and instructions for preparation to the user based on the analyzed data. For example, it might display something like, "Today's suggested menu: Stir-fried pork and cabbage, Instructions: Cut the ingredients and stir-fry them in a pan."

[1277] Step 12:

[1278] The user reviews the displayed information and prepares the dish based on the suggested menu. For example, they might cut pork and cabbage, then stir-fry them in a pan to complete the dish.

[1279] (Example 1)

[1280] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1281] Because the management and efficient use of ingredients in refrigerators are unclear, there is a need for a system that eliminates food waste and suggests the optimal cooking method based on the expiration date of the ingredients. Furthermore, conventional systems have not been able to provide suggestions that take into account the user's past selection history and preferences.

[1282] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1283] In this invention, the server includes means for inputting ingredient information, means for inputting expiration date information of the ingredient, means for retrieving multiple cooking methods using the ingredient from a data storage means based on the input information, means for determining priority based on the expiration date information of the ingredient and arranging each cooking method according to priority, and means for proposing the highest priority cooking method to the user and displaying how to prepare the cooking method. This makes it possible to efficiently utilize ingredients in the refrigerator, reduce food waste, and propose appropriate cooking methods that take into account the user's preferences.

[1284] "Means for inputting food information" refers to a device or interface for users to input the types and names of food items inside a refrigerator.

[1285] "Means for inputting expiration date information" refers to a device or interface for users to input the best-before date or expiration date for each food item.

[1286] "Data storage means" refers to a database or storage device for storing and managing information on ingredients, expiration dates, and multiple cooking methods based on these.

[1287] "Means for determining and arranging priorities" refers to a device or program that has the function of setting the priority of each cooking method based on the input expiration date information and listing them in that order.

[1288] "Means for proposing and displaying cooking methods" refers to a device or interface that presents the user with the determined optimal cooking method and displays the detailed preparation procedure.

[1289] "Communication means" refers to a network or communication interface used to transmit data input from a terminal to the main control unit.

[1290] The "main control unit" is a central control unit or server that receives and analyzes data and selects the optimal cooking method.

[1291] "Means of selecting based on the user's past selection history and preferences" refers to a device or program that has the function of recording the cooking methods and preferences previously chosen by the user and selecting the optimal cooking method taking that into consideration.

[1292] This invention is a system that manages information about food items in a refrigerator and their expiration dates, and suggests the optimal cooking method. The following describes specific embodiments for carrying out this invention.

[1293] This system allows users to input each ingredient in their refrigerator and its expiration date, and the server then searches for and suggests the most suitable cooking method based on that data. This enables users to efficiently utilize the ingredients in their refrigerator and reduce food waste.

[1294] System Configuration

[1295] 1. User terminal:

[1296] This is a device used by users to input food ingredient information and expiration date information. This includes smartphones, tablets, or PCs. Users launch a dedicated app and enter data through input forms such as text boxes.

[1297] 2. Server:

[1298] The server is the central component that receives data sent from the user terminal and searches for and suggests appropriate cooking methods based on that data. The server accesses the database, retrieves the relevant cooking methods, and provides data according to the user's request.

[1299] 3. Database:

[1300] The database is a system for storing ingredients, cooking methods, and related information. It manages details of each cooking method, required ingredients, and their corresponding expiration dates.

[1301] Specific example

[1302] For example, if a user inputs that they have "pork," "cabbage," and "carrots" in their refrigerator, the server searches its database for cooking methods using these ingredients and prioritizes them based on their expiration dates. As a result, it provides the user with a suggested cooking method, such as "stir-fried pork and cabbage," along with a detailed explanation of how to make it. By cooking based on this suggestion, the user can efficiently utilize the ingredients in their refrigerator.

[1303] Example of a prompt

[1304] Please enter the information about the ingredients in your refrigerator and their expiration dates. For example, enter it in the format of 'Pork, Expiration Date 2023-10-20', 'Cabbage, Expiration Date 2023-10-18', etc. We will then suggest the best cooking method.

[1305] This prompt allows users to easily operate the system and input ingredient and expiration date information. This enables the system to provide appropriate cooking instructions and reduce food waste.

[1306] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1307] Step 1: User enters ingredient information.

[1308] Users open their smartphones or tablets and launch the dedicated app. They then enter food information and expiration dates into the app's input form, such as "Pork, Expiration Date 2023-10-20" or "Cabbage, Expiration Date 2023-10-18". Specifically, users type the information into the text box and press the submit button to complete the input.

[1309] Input: Ingredient information (e.g., "Pork, Cabbage") and its expiration date (e.g., "2023-10-20, 2023-10-18")

[1310] Output: User-input text data

[1311] Step 2: Data conversion and transmission by the terminal

[1312] The device converts the ingredient information entered by the user into JSON format. For example, it formats it into the following format: {"materials": [{"name": "Pork", "expiry_date": "2023-10-20"}, {"name": "Cabbage", "expiry_date": "2023-10-18"}]}. Next, it sends this data to the server as an HTTP POST request. Specifically, the device sends the data to the API endpoint over the internet.

[1313] Input: User-inputted text data

[1314] Output: Data in JSON format and an HTTP POST request

[1315] Step 3: Server receives and analyzes data.

[1316] The server receives an HTTP request from the terminal and extracts and parses the data sent from the payload. Specifically, it parses JSON data in the format {"materials": [{"name": "Pork", "expiry_date": "2023-10-20"}, {"name": "Cabbage", "expiry_date": "2023-10-18"}]} to obtain the name of each ingredient and its expiration date.

[1317] Input: Data in JSON format

[1318] Output: Information on the names of ingredients and their corresponding expiration dates.

[1319] Step 4: Search for cooking methods using the server

[1320] The server uses the analyzed ingredient names to query the database and search for corresponding cooking methods. For example, to retrieve "recipes using pork and cabbage" from the database, it sends an SQL query. Specifically, it executes the query and returns a list of multiple cooking methods.

[1321] Input: Name of ingredients (e.g., "Pork, cabbage")

[1322] Output: List of suggested cooking methods

[1323] Step 5: Server prioritizes cooking method

[1324] The server evaluates the ingredient list and expiration dates for each cooking method and sets priorities. Specifically, it prioritizes cooking methods that use ingredients with shorter expiration dates. Based on the calculated priorities, it sorts the list of cooking methods.

[1325] Input: A list of suggested cooking methods and information on ingredients and their expiration dates.

[1326] Output: List of cooking methods sorted by priority

[1327] Step 6: Server selects the optimal cooking method

[1328] The server selects the optimal cooking method from among the high-priority options, taking into account the user's past selection history and preferences. For example, if the user previously preferred "stir-fry," the server will select "stir-fried pork and cabbage."

[1329] Input: A list of cooking methods sorted by priority, user selection history, and preference data.

[1330] Output: Optimal cooking method

[1331] Step 7: Server generates and transmits recipe information.

[1332] The server generates detailed information in JSON format about the selected optimal cooking method. For example, it might generate it in the format {"menu": {"name": "Stir-fried pork and cabbage", "instructions": "Cut the ingredients and stir-fry them in a frying pan"}} and send this to the terminal as an HTTP response.

[1333] Input: Optimal cooking method

[1334] Output: Detailed cooking instructions (in JSON format) and HTTP response

[1335] Step 8: Displaying the recipe on the device

[1336] The device parses the JSON data received from the server and displays the suggested cooking method and instructions to the user. Specifically, it displays "Today's suggested menu: Stir-fried pork and cabbage, Instructions: Cut the ingredients and stir-fry in a frying pan."

[1337] Input: Detailed cooking instructions (JSON format)

[1338] Output: Display of cooking instructions and preparation methods to the user.

[1339] Step 9: User performs cooking

[1340] The user checks the displayed cooking method and instructions, and then prepares the dish based on the ingredients in their refrigerator. Specifically, they prepare the ingredients according to the provided steps and complete the dish.

[1341] Input: Detailed cooking instructions and ingredients in the refrigerator

[1342] Output: Finished dish

[1343] (Application Example 1)

[1344] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1345] In modern times, there is a need to efficiently manage the food in refrigerators and reduce the amount of food that goes past its expiration date. However, it is difficult for users to keep track of the food in their refrigerators and use it appropriately in their daily lives, and food waste often occurs. Also, if necessary ingredients are missing, people have to go shopping each time, which is time-consuming and troublesome. To solve this problem, a system is needed that efficiently manages information about the food in the refrigerator and has an automatic ordering function for missing ingredients.

[1346] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1347] In this invention, the server includes means for inputting information on multiple food items present in the refrigerator, means for inputting the expiration dates of the food items, means for searching a database for multiple recipes using the food items based on the input information, means for determining priority based on the expiration dates of the food items and sorting each recipe according to priority, means for suggesting the highest-priority recipe to the user and displaying the instructions for making that recipe, and means for automatically ordering any missing ingredients based on the information on the food items. This enables centralized management of food item information in the refrigerator, allowing for optimal recipe suggestions to the user and automatic ordering of necessary ingredients.

[1348] "Means for inputting information on multiple food items present in a refrigerator" refers to devices or applications that allow users to input information such as the names, quantities, and locations of various food items stored in a refrigerator.

[1349] "Means for inputting the expiration date of the food item" refers to a device or application for the user to input the expiration date of each food item in the refrigerator.

[1350] "Means for searching a database for multiple recipes using the ingredients in question based on the input information" refers to a server or software that searches a database for multiple recipes using the ingredients entered by the user.

[1351] "Means for determining priority based on the expiration date of the ingredient and sorting each recipe according to priority" refers to a server or algorithm that determines whether an ingredient should be used preferentially based on the expiration date information of the input ingredient, and then sorts the corresponding recipes in order of priority.

[1352] "A means of suggesting a top-priority recipe to a user and displaying how to make that recipe" refers to a device or application that presents the user with the highest priority recipe and displays the user's detailed instructions for making that recipe.

[1353] "Means for automatically ordering missing ingredients based on information about the ingredients in question" refers to a server or software that automatically determines the missing ingredients needed for a dish based on ingredient information entered by the user and orders them online.

[1354] "A means of communication for sending data entered from a terminal to a server" refers to a communication interface, protocol, or system for sending information about ingredients and expiration dates from a terminal used by a user to a server.

[1355] "A means of selecting the optimal recipe from multiple recipes based on the user's past selection history and preferences" refers to an algorithm or software that analyzes the user's past selection history and preference information and suggests the most suitable recipe based on that information.

[1356] This invention relates to a system that manages the remaining ingredients in a refrigerator and their expiration dates, suggests the optimal menu and how to prepare it, and can automatically order any missing ingredients from an online shopping site.

[1357] System Configuration

[1358] 1. Method for entering information about the ingredients in the refrigerator:

[1359] Users use devices such as smartphones or tablets to input information about the food items in their refrigerator and their expiration dates. For example, they might enter "Pork, expiration date 2023-10-20" and "Cabbage, expiration date 2023-10-18" into the app's input form.

[1360] 2. Means of communication:

[1361] The aforementioned terminal sends the input data to the server via HTTP communication. It uses JSON format data for communication and sends an HTTP POST request.

[1362] 3. Means of searching for recipes based on input information:

[1363] The server analyzes the received data and searches the database for a suitable recipe based on the information about the ingredients in the refrigerator. For example, the server analyzes the received JSON data and retrieves the corresponding recipe from the database.

[1364] 4. Means for determining and sorting recipe priority:

[1365] The server prioritizes ingredients that should be consumed as soon as possible, based on their expiration dates. It then sorts the searched recipes according to this priority.

[1366] 5. Means for suggesting the highest priority recipe to the user and displaying the instructions:

[1367] The server selects the highest-priority recipe from the sorted list and sends its detailed instructions to the terminal. For example, the selected recipe for "Stir-fried Pork and Cabbage" and its instructions will be displayed.

[1368] 6. A means of automatically ordering missing materials:

[1369] The server matches the entered ingredient information with the suggested recipe and automatically orders any necessary ingredients that are not in the refrigerator from an online shopping site. For example, if the customer only has pork and cabbage and does not have carrots, which are needed for "stir-fried pork and cabbage," the server will automatically add carrots to the shopping cart from the online store.

[1370] Program Processing Description

[1371] The server uses a Python program to suggest recipes based on user input and automatically order any missing ingredients. The server uses the Python Requests library for communication. SQL is used to search for recipes in the database. Algorithms are implemented for prioritizing and sorting recipes. Furthermore, a machine learning model is used to select the optimal recipe based on the user's past selection history and preferences.

[1372] For example, if a user enters "pork, cabbage" along with expiration date information through a smartphone app, that information is sent to the server. The server then searches for a suitable recipe and suggests "stir-fried pork and cabbage," prioritizing ingredients with the earliest expiration dates. If carrots are missing in the recipe, they are automatically ordered from an online shopping site.

[1373] Example of a prompt

[1374] Please enter the ingredients in your refrigerator (e.g., "pork," "cabbage") and their expiration dates. Based on this information, we will automatically order the necessary ingredients from an online shopping site and suggest the best recipes.

[1375] As a result, this invention allows for efficient management of ingredients in the refrigerator, reduces food waste, and makes meal preparation easier for the user.

[1376] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1377] Step 1:

[1378] Users input information about the food items in their refrigerator and their expiration dates using a smartphone, tablet, or other device.

[1379] Specific examples of the information to be entered include "Pork, Expiration Date 2023-10-20" and "Cabbage, Expiration Date 2023-10-18". This will be the input data.

[1380] Step 2:

[1381] The terminal converts the entered ingredient information into JSON format and sends it to the server as an HTTP POST request.

[1382] This communication method utilizes the JSON format and the HTTP protocol. The data is sent in the format {"materials": [{"name": "Pork", "expiry_date": "2023-10-20"}, {"name": "Cabbage", "expiry_date": "2023-10-18"}]}. This serves as the input data, and the server performs the analysis.

[1383] Step 3:

[1384] The server receives an HTTP POST request and parses the received data.

[1385] The analyzed data extracts the expiration dates of each food item in the refrigerator. Based on the input data, a query is generated for the database to search for the corresponding recipe. For example, it searches for recipes using "pork" and "cabbage".

[1386] Step 4:

[1387] The server receives multiple recipes as search results from the database, determines a priority based on the expiration dates of the ingredients in each recipe, and sorts the recipes.

[1388] This sorting is performed using an algorithm that prioritizes recipes that use ingredients nearing their expiration date. For example, "Stir-fried pork and cabbage" would be one such recipe.

[1389] Step 5:

[1390] The server selects the most suitable recipe from several options based on the user's past selection history and preferences.

[1391] This process utilizes machine learning models. Based on past selection history data and preference data, it selects the most suitable recipe and prioritizes it to determine the best option.

[1392] Step 6:

[1393] The server generates detailed information about the selected top-priority recipe in JSON format and sends it to the terminal.

[1394] For example, send the following in the format: {"menu": {"name": "Stir-fried pork and cabbage", "instructions": "Cut the ingredients and stir-fry them in a frying pan"}}. This will be the output data.

[1395] Step 7:

[1396] The device analyzes the received recipe information and displays suggested menus and their preparation methods to the user.

[1397] Specifically, it displays: "Today's suggested menu: Stir-fried pork and cabbage, Instructions: Cut the ingredients and stir-fry them in a frying pan."

[1398] Step 8:

[1399] The server automatically orders any missing ingredients from online shopping sites based on the suggested recipe.

[1400] The server generates a list of all the ingredients needed for the suggested recipe and determines which ingredients are missing from the refrigerator. It then automatically orders the missing ingredients through the online shopping site's API. For example, it adds "carrots" to the online shopping site's shopping cart.

[1401] Step 9:

[1402] The user prepares the meal according to the suggested menu and instructions. They also confirm that any missing ingredients are automatically ordered from an online store, and the purchasing process proceeds. Thus, the user's meal preparation is complete.

[1403] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1404] This invention combines an emotional engine with a system that manages remaining ingredients in a refrigerator and their expiration dates, and suggests the optimal menu and preparation method. The following describes specific embodiments for carrying out the invention.

[1405] overview

[1406] This system allows users to input each ingredient in their refrigerator and its expiration date, and the server then searches for and suggests the most suitable recipe based on that data. Furthermore, it can use an emotion engine to recognize the user's emotional state and provide recipes tailored to that state.

[1407] System Configuration

[1408] 1. User terminal:

[1409] The user terminal is a device used to input information about the food items in the refrigerator and their expiration dates. It also includes voice and image analysis functions to input the user's emotional state. Examples include smartphones, tablets, and PCs.

[1410] 2. Server:

[1411] The server is the central component that receives ingredient information and emotional states sent from the user's terminal and searches for and suggests appropriate recipes based on that information. The server accesses the database to retrieve appropriate recipes and provides data according to the user's request.

[1412] 3. Database:

[1413] The database is a system for storing ingredients, recipes, and related information. It manages details of each recipe, required ingredients and their corresponding expiration dates, as well as data on the user's past selection history and emotional state.

[1414] 4. Emotional Engine:

[1415] The emotion engine is a system for recognizing a user's emotional state. It includes voice analysis and image analysis methods to analyze the user's emotions from their voice and facial expressions and transmit that data to a server.

[1416] Specific processing flow

[1417] Data entry phase

[1418] 1. The user enters information about the food items in their refrigerator and their expiration dates using a smartphone or other device. For example, they might enter information such as "Pork, expiration date 2023-10-20" or "Cabbage, expiration date 2023-10-18" into the application's input form.

[1419] 2. The user inputs their emotional state through the emotion engine. The emotional state is analyzed by capturing facial expressions using voice input or a camera.

[1420] Data transmission phase

[1421] 3. The terminal converts the entered food information and emotional state into JSON data. For example, {"materials": [{"name": "Pork", "expiry_date": "2023-10-20"}, {"name": "Cabbage", "expiry_date": "2023-10-18"}], "emotion": {"mood": "happy"}}.

[1422] 4. The terminal sends data to the server. The transmission method is an HTTP POST request.

[1423] Data Processing Phase

[1424] 5. The server receives data sent from the terminal. It parses the HTTP request, obtains the JSON data, and then parses it.

[1425] 6. The server accesses the database and searches for a suitable recipe based on the received ingredient information and emotional state. For example, it retrieves recipes such as "stir-fried vegetables" or "curry" that contain "pork" and "cabbage" from the database.

[1426] 7. The server determines the priority of the retrieved recipes based on the expiration date of each ingredient. For example, it compares the expiration dates of the ingredients and sorts the recipes with ingredients that need to be consumed the fastest to the top.

[1427] 8. The server selects the optimal recipe by considering the user's past selection history and preferences, as well as their emotional state recognized by the emotion engine. For example, it will suggest more enjoyable dishes for a "happy" emotional state.

[1428] Proposed menu generation phase

[1429] 9. The server generates detailed information about the selected recipe in JSON format. For example, it creates data in the format {"menu": {"name": "Stir-fried pork and cabbage", "instructions": "Chop the ingredients and stir-fry them in a pan"}}.

[1430] 10. The server sends the generated JSON data to the terminal as an HTTP response. The response contains the selected recipe and its details.

[1431] Data display phase

[1432] 11. The terminal analyzes the response data received from the server. It parses the JSON data to obtain the suggested menu and how to make it.

[1433] 12. The terminal displays suggested menus and instructions for preparation to the user based on the analyzed data. For example, it might display "Today's suggested menu: Stir-fried pork and cabbage, Instructions: Cut the ingredients and stir-fry them in a pan."

[1434] 13. The user reviews the displayed information and prepares the dish based on the suggested menu. For example, they might cut pork and cabbage, then stir-fry them in a pan to complete the dish.

[1435] Specific example

[1436] For example, if a user inputs that they have "pork" and "cabbage" in their refrigerator, and the emotion engine recognizes a "happy" emotional state, the server searches its database for recipes using these ingredients, prioritizing them based on their expiration dates. As a result, it provides the user with a suggested menu item, "stir-fried pork and cabbage," which is appropriate for their happy emotional state, and explains how to make it in detail. By following these instructions, the user can efficiently utilize the ingredients in their refrigerator.

[1437] This system not only allows for the effective management and use of ingredients in the refrigerator, but also enables personalized recipe suggestions tailored to the user's emotional state. This makes the daily cooking experience more enjoyable and satisfying, contributing to a reduction in food waste.

[1438] The following describes the processing flow.

[1439] Step 1:

[1440] The user collects information on multiple food items in their refrigerator, along with their expiration dates. Using a smartphone or other device, the user enters information such as "Pork, expiration date 2023-10-20" and "Cabbage, expiration date 2023-10-18" into the application's form.

[1441] Step 2:

[1442] Users record their emotional state through an emotion engine. For example, their emotional state is analyzed by capturing facial expressions using voice input or a camera.

[1443] Step 3:

[1444] The terminal converts the entered food information and emotional state into JSON data. For example, the format will be {"materials": [{"name": "Pork", "expiry_date": "2023-10-20"}, {"name": "Cabbage", "expiry_date": "2023-10-18"}], "emotion": {"mood": "happy"}}.

[1445] Step 4:

[1446] The terminal sends the converted data to the server. The transmission method uses an HTTP POST request. For example, the request body contains the JSON data mentioned earlier.

[1447] Step 5:

[1448] The server receives data sent from the terminal. The server parses the HTTP request, obtains JSON data, and parses it. For example, it might obtain information about each ingredient in array format as the parsed data.

[1449] Step 6:

[1450] The server accesses the database and searches for relevant recipes based on the received ingredient information and emotional state. The database stores detailed information and required ingredient lists for each recipe. For example, it retrieves recipes such as "stir-fried vegetables" and "curry" that contain "pork" and "cabbage."

[1451] Step 7:

[1452] The server prioritizes the retrieved recipes based on the expiration dates of each ingredient. For example, it compares the expiration dates of ingredients and sorts recipes prioritizing those with ingredients that need to be consumed the fastest.

[1453] Step 8:

[1454] The server selects the optimal recipe by considering the user's past selection history and preferences, as well as their emotional state recognized by the emotion engine. For example, it suggests more enjoyable dishes for a "happy" emotional state.

[1455] Step 9:

[1456] The server generates detailed information about the selected recipe in JSON format. For example, it creates data in the format {"menu": {"name": "Stir-fried pork and cabbage", "instructions": "Chop the ingredients and stir-fry them in a frying pan"}}.

[1457] Step 10:

[1458] The server sends the generated JSON data to the terminal as an HTTP response. The response contains the selected recipe and its details.

[1459] Step 11:

[1460] The terminal analyzes the response data received from the server. It parses the JSON data to obtain the suggested menu and how to make it.

[1461] Step 12:

[1462] The device displays suggested menus and instructions for preparation to the user based on the analyzed data. For example, it might display something like, "Today's suggested menu: Stir-fried pork and cabbage, Instructions: Cut the ingredients and stir-fry them in a pan."

[1463] Step 13:

[1464] The user reviews the displayed information and prepares the dish based on the suggested menu. For example, they might cut pork and cabbage, then stir-fry them in a pan to complete the dish.

[1465] (Example 2)

[1466] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1467] Modern households store a wide variety of ingredients in their refrigerators, but efficiently utilizing these ingredients is difficult. In particular, it is hard to plan for consuming ingredients that are nearing their expiration date, leading to food waste. Furthermore, the lack of meal suggestions tailored to the user's emotional state on any given day diminishes the enjoyment and satisfaction of daily cooking. Therefore, there is a need for a system that can suggest optimal recipes based on ingredient information and emotional state, thereby reducing food waste and providing a cooking experience that matches the user's emotions.

[1468] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for inputting information on multiple food items present in the refrigerator, means for inputting the expiration dates of the food items, and means for inputting the user's emotional state. This makes it possible to search a database for multiple recipes using the food items based on the input information and emotional state, determine priority based on the expiration dates of the food items, sort each recipe according to priority, select the optimal recipe considering the user's past selection history, preferences, and emotional state, propose the highest priority recipe to the user, and display how to make it. This makes it possible to reduce food waste and provide personalized cooking suggestions that are tailored to the user's emotions.

[1469] "A means of inputting information on multiple food items present in a refrigerator" refers to a device or software that allows a user to input the types of food items stored in the refrigerator and their respective expiration dates into a terminal.

[1470] "Means for inputting the expiration date of the food item" refers to a device or software that allows the user to input the expiration date of each food item into a terminal.

[1471] "Means for inputting the user's emotional state" refers to devices or software that analyze the user's voice and facial expressions to recognize their emotional state and input that information into the system.

[1472] "Means for searching a database for multiple recipes using the ingredient in question based on input information and emotional state" refers to an algorithm or software for searching a database for relevant recipes using ingredient information and emotional state data received from a terminal.

[1473] "Means for determining priority based on the expiration date of the ingredients and sorting each recipe according to priority" refers to an algorithm or software that determines the priority of recipes based on the expiration date of the input ingredients and rearranges the recipes accordingly.

[1474] "Means for selecting the optimal recipe by considering the user's past selection history, preferences, and emotional state" refers to algorithms or software that use data on the user's past recipe selection history and emotional state to determine and select the most suitable recipe for the user.

[1475] "Means for suggesting a top-priority recipe to the user and displaying how to make that recipe" refers to devices or software that notify the user of a high-priority recipe and display its detailed instructions.

[1476] "A means of communication for sending data entered from a terminal to a server" refers to network communication devices and protocols for sending food information and emotional states entered from a user's terminal to a server.

[1477] An "emotion engine" is an algorithm or software that analyzes a user's voice and facial expressions to identify their emotional state.

[1478] This invention relates to a system that provides an optimal recipe by taking into account information about the ingredients in the refrigerator, their expiration dates, and the user's emotional state. The specific methods for implementing this invention are described below.

[1479] System Configuration

[1480] This system consists of user terminals, a server, a database, and an emotion engine. Details of each component are as follows:

[1481] User terminal

[1482] The user terminal is a device used to input information about the food items in the refrigerator and their expiration dates. It also includes voice and image analysis functions for inputting the user's emotional state. Examples include smartphones, tablets, and PCs.

[1483] server

[1484] The server is the central component that receives ingredient information and emotional states sent from the user's terminal and searches for and suggests appropriate recipes based on that information. The server accesses a database to retrieve appropriate recipes and provides data according to the user's request. Specifically, Apache servers and Nginx servers are often used.

[1485] database

[1486] A database is a system for storing ingredients, recipes, and related information. It manages details of each recipe, required ingredients and their corresponding expiration dates, as well as data on the user's past selection history and emotional state. Specifically, relational databases such as MySQL and PostgreSQL are used.

[1487] Emotional Engine

[1488] An emotion engine is a system for recognizing a user's emotional state. It includes voice and image analysis methods to analyze emotions from the user's voice and facial expressions and send that data to a server. Specifically, it uses tools such as OpenCV and the Google Cloud Speech-to-Text API.

[1489] Specific example

[1490] The following specific examples will help to understand the present invention.

[1491] For example, if a user inputs that they have "pork" and "cabbage" in their refrigerator, and the emotion engine recognizes a "happy" emotional state, the server searches its database for recipes using these ingredients, prioritizing them based on their expiration dates. As a result, it generates a suggested menu item, such as "Pork and Cabbage Pasta," which is adapted to the user's happy emotional state. The server then sends this recipe and detailed instructions to the user's terminal, allowing the user to begin cooking.

[1492] Example of a prompt

[1493] The following are specific examples of prompts to input into a generative AI model:

[1494] The refrigerator contains pork and cabbage. The current emotional state is "happy." Please suggest a recipe using these ingredients that is appropriate for this emotional state.

[1495] This invention enables effective management of ingredients in the refrigerator and provides optimal recipe suggestions, thereby reducing food waste and realizing a personalized cooking experience that responds to the user's emotions.

[1496] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1497] Step 1:

[1498] Users input information about the food items in their refrigerator and their expiration dates using a smartphone or other device. For example, they might enter "Pork, expiration date 2023-10-20" and "Cabbage, expiration date 2023-10-18" into the app's input form. The input data is output as JSON data from the device's input form.

[1499] Step 2:

[1500] Users input their emotional state through the emotion engine. For example, they might use the voice input function to say "I feel good today," and the emotion engine analyzes the voice and recognizes the user's emotional state as "happy." The input data is output from the emotion engine as JSON data.

[1501] Step 3:

[1502] The terminal integrates the entered food information and emotional state to generate final JSON data. For example, it formats the data as follows: {"materials": [{"name": "Pork", "expiry_date": "2023-10-20"}, {"name": "Cabbage", "expiry_date": "2023-10-18"}], "emotion": {"mood": "happy"}}. The integrated data is then sent from the terminal to the server.

[1503] Step 4:

[1504] The device sends the generated JSON data to the server using an HTTP POST request. The detailed communication protocol used is HTTP / HTTPS. The transmitted data is received by the server.

[1505] Step 5:

[1506] The server parses the received HTTP request and retrieves JSON data. Specifically, it extracts JSON data from the body of the HTTP request. The extracted data is then passed to an internal processing module.

[1507] Step 6:

[1508] The server accesses the database and searches for relevant recipes based on the received ingredient information. For example, it searches the database for recipes containing "pork" and "cabbage" using an SQL query. The search results return a list of relevant recipes.

[1509] Step 7:

[1510] The server compares the expiration dates of each ingredient in the retrieved recipes to determine priority. For example, to prioritize ingredients with approaching expiration dates, it uses the SQL ORDER BY clause to sort the recipes. A sorted list of recipes is then generated.

[1511] Step 8:

[1512] The server selects the optimal recipe by considering the user's past selection history and preferences. It also takes into account the emotional state recognized by the emotion engine. For example, it might select the best recipe from those the user has previously chosen when they were in a "happy" emotional state. One optimal recipe is then selected.

[1513] Step 9:

[1514] The server generates detailed information about the selected recipe in JSON format. For example, it creates data in the format {"menu": {"name": "Pork and cabbage pasta", "instructions": "Chop the ingredients, boil the pasta, and stir-fry in a pan"}}. The generated data is then sent to the terminal.

[1515] Step 10:

[1516] The server sends the generated JSON data to the terminal as an HTTP response. The response data is sent via the HTTP / HTTPS protocol.

[1517] Step 11:

[1518] The terminal analyzes the response data received from the server and parses the data in JSON format. For example, it analyzes data such as {"menu": {"name": "Pork and cabbage pasta", "instructions": "Cut the ingredients, boil the pasta, and stir-fry in a pan"}}. The analysis results are stored as an internal data structure.

[1519] Step 12:

[1520] The device displays suggested menus and instructions for preparation based on the analyzed data. For example, it might display: "Today's suggested menu: Pork and cabbage pasta, Instructions: Cut the ingredients, boil the pasta, and stir-fry in a pan."

[1521] Step 13:

[1522] The user cooks according to the displayed recipe. For example, they might cut pork and cabbage, boil pasta, and then stir-fry it in a pan to complete the dish.

[1523] (Application Example 2)

[1524] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1525] Traditional refrigerator management systems, while capable of managing information about ingredients and expiration dates, were unable to provide personalized recipe suggestions based on the user's emotional state. This led to decreased user satisfaction and food waste. Furthermore, the suggested recipes often didn't match the user's mood or preferences at the time, resulting in a lack of enjoyment in daily cooking.

[1526] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[1527] In this invention, the server includes means for inputting information on multiple food items present in the refrigerator, means for inputting the expiration dates of the food items, means for determining priority based on the expiration dates of the food items and recognizing the user's emotional state, and means for optimizing recipes based on the emotional state. This enables the suggestion of optimal recipes according to the user's emotional state, improving the daily cooking experience and reducing food waste.

[1528] "Information on multiple food items present in the refrigerator" refers to data about various food items stored in the refrigerator, including their names, quantities, acquisition dates, and other attributes.

[1529] The "best before date" for a food item refers to the date by which the food stored in the refrigerator should be consumed appropriately, and this date varies depending on the food item.

[1530] "Means of input" refers to devices or methods for users to register information about the food in their refrigerator and its expiration date into the system, such as input devices like smartphones, tablets, or PCs.

[1531] "Methods for searching a database" refer to programs or algorithms that extract information from a database based on specific conditions and provide it to the user.

[1532] "Means for determining priority and sorting each recipe according to priority" refers to a method for determining the priority of multiple recipes based on expiration dates or other conditions, and then rearranging the recipes based on the results.

[1533] "Means for recognizing a user's emotional state" refers to devices or software that analyze a user's facial expressions, voice, etc., to evaluate their psychological state at that time.

[1534] "Methods for optimizing recipes based on emotional state" refer to methods or systems that select recipes and suggest the most suitable dishes by reflecting the recognized emotional state of the user.

[1535] "A means of suggesting the highest-priority recipe to the user and displaying how to make that recipe" refers to a function that presents the user with the most suitable recipe selected by the system and provides a detailed explanation of how to cook it.

[1536] This invention combines an emotional engine with a system that manages the remaining ingredients in a refrigerator and their expiration dates, and suggests the optimal menu and how to prepare it. The following describes specific embodiments for carrying out the invention.

[1537] System Configuration

[1538] 1. User terminal

[1539] The user terminal is a device used to input information about the food items in the refrigerator and their expiration dates. It also includes voice and image analysis functions to input the user's emotional state. Smartphones, tablets, and PCs are examples of this type of device. The terminal also has a means of communication to send the input data to the server.

[1540] 2. Server

[1541] The server is the central component that receives ingredient information and emotional states sent from the user's terminal and searches for and suggests appropriate recipes based on that information. The server accesses the database to retrieve appropriate recipes and provides data according to the user's request.

[1542] The software used includes Flask (a web server framework) and other web server frameworks.

[1543] 3. Database

[1544] The database is a system for storing ingredients, recipes, and related information. It manages details of each recipe, required ingredients and their corresponding expiration dates, as well as data on the user's past selection history and emotional state.

[1545] 4. Emotional Engine

[1546] An emotion engine is a system for recognizing a user's emotional state and includes methods for speech analysis and image analysis. Specifically, speech recognition APIs (e.g., Google Speech-to-Text API) are used for speech analysis, and libraries such as OpenCV are used for image analysis.

[1547] Specific processing flow

[1548] Users input information about the food items in their refrigerator and their expiration dates using a smartphone or other device. For example, they might enter information such as "Pork, expiration date 2023-10-20" or "Cabbage, expiration date 2023-10-18" into the application's input form. Next, they input their emotional state through an emotion engine, and their emotional state is analyzed by capturing facial expressions using voice input or a camera.

[1549] The terminal converts the entered ingredient information and emotional state into JSON data and sends it to the server. The server receives and analyzes this data, accesses the database, and searches for the corresponding recipe based on the received ingredient information and emotional state. Furthermore, it determines priorities based on the expiration date of each ingredient and selects the optimal recipe by considering the user's past selection history and preferences, as well as the emotional state recognized by the emotion engine.

[1550] Detailed information about the selected recipe is generated in JSON format and sent from the server to the terminal. The terminal parses it and displays the suggested menu and its preparation method to the user. By following these instructions, the user can efficiently utilize the ingredients in their refrigerator.

[1551] Examples of specific cases and prompt statements

[1552] Specific example

[1553] If a user inputs that they have "pork" and "cabbage" in their refrigerator and the emotion engine recognizes a "happy" emotional state, the server searches its database for recipes using these ingredients, prioritizing them based on their expiration dates. As a result, it provides the user with a suggested menu item, "Stir-fried pork and cabbage," which is appropriate for their happy emotional state, and explains how to make it in detail.

[1554] Example of a prompt

[1555] If you want to send the following data to the server, you can send data like this:

[1556] {

[1557] "materials": ["pork", "cabbage"],

[1558] "expiry_dates": ["2023-10-20", "2023-10-18"],

[1559] "image": <Image data of the user's facial expression captured by the camera>

[1560] }

[1561] In this way, users can not only effectively manage and utilize the ingredients in their refrigerator using the system, but also receive personalized recipe suggestions based on their emotional state. This makes the daily cooking experience more enjoyable and satisfying, and also contributes to reducing food waste.

[1562] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1563] Step 1:

[1564] The user enters information about the food items in their refrigerator and their expiration dates using a smartphone or other device.

[1565] Specific operation: The user launches the application and enters the name, quantity, and expiration date of the ingredients into the input form.

[1566] Input: Ingredient information (e.g., "Pork", "Cabbage") and expiration date (e.g., "2023-10-20", "2023-10-18")

[1567] Output: Input food information and expiration date data

[1568] Step 2:

[1569] Users input their emotional state through an emotion engine, and their facial expressions are captured using voice input or a camera.

[1570] Specific operation: The user uses the in-app camera function to take a picture of their face and have their emotional state recognized.

[1571] Input: User's facial expression images and audio data

[1572] Output: User's emotional state (e.g., "happy", "sad")

[1573] Step 3:

[1574] The terminal converts the entered food information and emotional state into JSON data and sends it to the server.

[1575] Specific operation: The terminal packages the ingredient information and emotional state into JSON format (for example, {"materials": ["pork", "cabbage"], "expiry_dates": ["2023-10-20", "2023-10-18"], "emotion": "happy"}) and sends it to the server using an HTTP POST request.

[1576] Input: Data in JSON format

[1577] Output: HTTP request sent to the server

[1578] Step 4:

[1579] The server receives and analyzes the data sent from the terminal.

[1580] Specific operation: The server receives an HTTP request and parses the JSON data to retrieve ingredient information, expiration date, and emotional state.

[1581] Input: JSON data sent from the device

[1582] Output: Analyzed food information, expiration date, emotional state

[1583] Step 5:

[1584] The server accesses the database and searches for the appropriate recipe based on the received ingredient information and emotional state.

[1585] Specific operation: The server executes a database query to retrieve a recipe suitable for the ingredients. The database stores recipes corresponding to the ingredients and emotional states.

[1586] Input: Analyzed food information, expiration date, emotional state

[1587] Output: List of relevant recipes

[1588] Step 6:

[1589] The server determines the priority of recipes based on the expiration date of each ingredient and sorts the recipes accordingly.

[1590] Specific operation: The server compares the expiration dates of ingredients and prioritizes recipes that include ingredients that need to be consumed the soonest.

[1591] Input: List of relevant recipes, expiration dates of ingredients

[1592] Output: High-priority recipe list

[1593] Step 7:

[1594] The server selects the optimal recipe by considering the user's past selection history, preferences, and emotional state.

[1595] Specific operation: The server retrieves the user's past selection history from the database and executes an algorithm to select the optimal recipe in conjunction with their emotional state.

[1596] Input: High-priority recipe list, user's past selection history, emotional state

[1597] Output: Optimal recipe

[1598] Step 8:

[1599] The server generates detailed information about the selected recipe in JSON format and sends it to the terminal.

[1600] Specific operation: The server formats the selected recipe and its details (e.g., ingredients, instructions) in JSON format and sends it to the terminal as an HTTP response.

[1601] Input: Optimal recipe

[1602] Output: Recipe data in JSON format

[1603] Step 9:

[1604] The terminal analyzes the response data received from the server and displays the suggested menu and its preparation method to the user.

[1605] Specific operation: The terminal parses the received JSON data and displays the recipe and cooking instructions on the screen.

[1606] Input: Recipe data in JSON format

[1607] Output: The recipe and its details displayed to the user

[1608] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1609] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1610] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[1611] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1612] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[1613] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[1614] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[1615] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[1616] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[1617] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[1618] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[1619] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[1620] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[1621] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1622] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[1623] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[1624] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[1625] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[1626] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[1627] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[1628] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[1629] The following is further disclosed regarding the embodiments described above.

[1630] (Claim 1)

[1631] A means of inputting information about multiple food items present in the refrigerator,

[1632] A means for inputting the expiration date of the food ingredient,

[1633] A means of searching a database for multiple recipes using the ingredient in question based on the information entered,

[1634] A means for determining priority based on the expiration date of the ingredients and sorting each recipe according to priority,

[1635] A means of suggesting the highest priority recipe to the user and displaying the instructions for making that recipe,

[1636] A system that includes this.

[1637] (Claim 2)

[1638] The system according to claim 1, wherein the means for inputting information about the food items present in the refrigerator is a communication means for transmitting data entered from a terminal to a server.

[1639] (Claim 3)

[1640] The system according to claim 1, wherein the server includes means for selecting the optimal recipe from among multiple recipes based on the user's past selection history and preferences.

[1641] "Example 1"

[1642] (Claim 1)

[1643] A means of inputting ingredient information,

[1644] A means for inputting the expiration date information of the food ingredient,

[1645] A means for searching a data storage means for multiple cooking methods using the ingredient in question, based on the input information.

[1646] A means for determining priority based on the expiration date information of the ingredients and arranging each cooking method according to priority,

[1647] A means of suggesting the highest priority cooking method to the user and displaying the method for creating said cooking method,

[1648] A system that includes this.

[1649] (Claim 2)

[1650] The system according to claim 1, wherein the means for inputting information on the ingredients comprises a communication means for transmitting data input from a terminal to a main control unit.

[1651] (Claim 3)

[1652] The system according to claim 1, wherein the main control device includes means for selecting the optimal cooking method from among a plurality of cooking methods based on the user's past selection history and preferences.

[1653] "Application Example 1"

[1654] (Claim 1)

[1655] A means of inputting information about multiple food items present in the refrigerator,

[1656] A means for inputting the expiration date of the food ingredient,

[1657] A means of searching a database for multiple recipes using the ingredient in question based on the information entered,

[1658] A means for determining priority based on the expiration date of the ingredients and sorting each recipe according to priority,

[1659] A means of suggesting the highest priority recipe to the user and displaying the instructions for making that recipe,

[1660] A means for automatically ordering missing ingredients based on information about the ingredients,

[1661] A system that includes this.

[1662] (Claim 2)

[1663] The system according to claim 1, wherein the means for inputting information about the food items present in the refrigerator is a communication means for transmitting data entered from a terminal to a server.

[1664] (Claim 3)

[1665] The system according to claim 1, wherein the server includes means for selecting the optimal recipe from among multiple recipes based on the user's past selection history and preferences.

[1666] "Example 2 of combining an emotion engine"

[1667] ---

[1668] (Claim 1)

[1669] A means of inputting information about multiple food items present in the refrigerator,

[1670] A means for inputting the expiration date of the food ingredient,

[1671] A means of inputting the user's emotional state,

[1672] A means for searching a database for multiple recipes using the ingredient in question, based on the input information and emotional state.

[1673] A means for determining priority based on the expiration date of the ingredients and sorting each recipe according to priority,

[1674] A means for selecting the optimal recipe, taking into account the user's past selection history, preferences, and emotional state,

[1675] A means of suggesting the highest priority recipe to the user and displaying the instructions for making that recipe,

[1676] A system that includes this.

[1677] (Claim 2)

[1678] The system according to claim 1, wherein the means for inputting information and emotional state of food items present in the refrigerator is a communication means for transmitting data input from a terminal to a server.

[1679] (Claim 3)

[1680] The system according to claim 1, wherein the server includes means for providing recipe suggestions based on the user's emotional state using an emotion engine for analyzing the user's emotional state.

[1681] "Application example 2 when combining with an emotional engine"

[1682] (Claim 1)

[1683] A means of inputting information about multiple food items present in the refrigerator,

[1684] A means for inputting the expiration date of the food ingredient,

[1685] A means of searching a database for multiple recipes using the ingredient in question based on the information entered,

[1686] A means for determining priority based on the expiration date of the ingredients and sorting each recipe according to priority,

[1687] A means of recognizing the user's emotional state,

[1688] A means of optimizing recipes based on emotional state,

[1689] A means of suggesting the highest priority recipe to the user and displaying the instructions for making that recipe,

[1690] A system that includes this.

[1691] (Claim 2)

[1692] The system according to claim 1, wherein the means for inputting information about the food items present in the refrigerator is a communication means for transmitting data entered from a terminal to a server.

[1693] (Claim 3)

[1694] The system according to claim 1, wherein the server includes means for selecting the optimal recipe from a plurality of recipes based on the user's past selection history and preferences, and also includes means for selecting a recipe while taking into account the user's emotional state. [Explanation of Symbols]

[1695] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of inputting information about multiple food items present in the refrigerator, A means for inputting the expiration date of the food ingredient, A means of searching a database for multiple recipes using the ingredient in question based on the information entered, A means for determining priority based on the expiration date of the ingredients and sorting each recipe according to priority, A means of suggesting the highest priority recipe to the user and displaying the instructions for making that recipe, A system that includes this.

2. The system according to claim 1, wherein the means for inputting information about the food items present in the refrigerator is a communication means for transmitting data input from a terminal to a server.

3. The system according to claim 1, wherein the server includes means for selecting the optimal recipe from among multiple recipes based on the user's past selection history and preferences.

Images