system

Abstract

We provide the system. [Solution] A means for receiving food information from multiple status detection devices installed in a household storage device, A means for identifying unfulfilled food items based on the aforementioned food information, means for generating a food and drink instruction related to the aforementioned unfulfilled food, A means for placing an order to supply food based on the aforementioned food and beverage instructions, The means of distributing the aforementioned food and drink instructions to an output device and guiding the user visually or audibly, A system that includes this.

Description

Technical Field

[0001] The technology of this disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance as a response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In modern society, as the number of dual-income households increases, the time and labor spent on meal preparation within the family have become issues. In particular, for households with growing children, it is important to provide a nutritionally balanced diet, but it is difficult to create a menu considering sufficient nutrition within limited time, and to manage and purchase the necessary food ingredients. There is a need for means to solve such a situation and efficiently and effectively prepare a nutritionally balanced meal.

Means for Solving the Problems

[0005] This invention provides a system that uses a status detection device installed in a household storage device to collect current food information and identify missing food items. This enables the generation of eating instructions based on nutritional guidelines and the automatic ordering of missing foods. Furthermore, the system supports users in smoothly proceeding with cooking by delivering the generated eating instructions visually or audibly. It also provides a configuration that optimizes future eating instructions by receiving feedback that takes into account the user's individual information. This streamlines the preparation of nutritious meals and reduces the burden on households.

[0006] A "condition detection device" is a device installed in a household storage system that collects information such as the presence, quantity, and expiration date of food.

[0007] "Food information" refers to data obtained from a condition detection device, such as the type, quantity, and expiration date of the food.

[0008] "Unsatisfied food supplies" refers to food items that are necessary for the planned menu but are insufficient in current storage conditions.

[0009] "Food and beverage instructions" refer to instructions regarding cooking and replenishment of ingredients, generated based on collected food information.

[0010] "Means of executing orders" refers to the function of automatically placing orders with external supply sources to replenish unfulfilled food supplies.

[0011] An "output device" is a device used to transmit information to a user visually or audibly, and includes smartphones and smart speakers.

[0012] "Nutritional guidelines" are guidelines that show meal plans that take into account the appropriate nutritional balance for each individual user.

[0013] "Feedback" refers to the act of users submitting opinions or evaluations regarding the services they have been provided. [Brief explanation of the drawing]

[0014] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14] This is a sequence diagram showing the processing flow of the data processing system in Application Example 2, which combines an emotion engine. [Modes for carrying out the invention]

[0015] Next, an example of an embodiment of the system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0018] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0019] In the following embodiments, the numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0020] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0022] [First Embodiment]

[0023] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0024] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0027] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0030] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0034] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0035] The system of this invention utilizes an AI agent designed to assist with meal preparation at home. This system functions primarily through interaction between a server, a terminal, and the user. Its specific operation is described below.

[0036] The server periodically collects food information from status detection devices installed in home refrigerators and pantries. This information includes the type, quantity, and expiration date of the food. For example, the server periodically checks the remaining amount and expiration date of milk in the refrigerator.

[0037] Based on the collected food information, the server creates a menu that takes into account the child's age and nutritional needs. This menu also includes an overall nutritional calculation to ensure nutritional balance. For example, a plan might be made to include spinach, which is rich in iron, during certain periods.

[0038] Next, the server compares the created menu with the current food inventory to identify any missing items. Based on this information, the server automatically orders the missing items from an external online store. For example, if the menu includes tomato soup and there is a shortage of tomatoes, the server will order tomatoes.

[0039] Furthermore, the server generates a recipe based on the created menu and sends it to the device. The device then guides the user visually or audibly with this recipe information via a smartphone or smart speaker. For example, a smart speaker might instruct the user to "First, put olive oil in the pot."

[0040] Users can provide feedback through their device during or after cooking. This allows the server to collect this feedback and use it to improve future menus and suggestions. For example, if a user inputs "Please use fewer spices next time," that information will be used in the next menu.

[0041] Thus, the system of this invention helps dual-income families efficiently prepare nutritionally balanced meals, significantly reducing the time and effort spent at home.

[0042] The following describes the processing flow.

[0043] Step 1:

[0044] The server obtains food information from household status detection devices. This includes the type, quantity, and expiration date of food items in the refrigerator and pantry. For example, the server recognizes that there is 200ml of milk remaining.

[0045] Step 2:

[0046] The server analyzes the collected food information and generates a nutritionally balanced menu that takes into account the child's age and allergy information. For example, if iron is needed, sautéed spinach will be added to the menu.

[0047] Step 3:

[0048] The server creates a list of necessary ingredients based on the generated menu and compares it with the current inventory to identify any missing foods. For example, it might find that two tomatoes are missing.

[0049] Step 4:

[0050] The server automatically places orders for identified missing ingredients via an external online store API. For example, it might order 5 tomatoes online.

[0051] Step 5:

[0052] The server generates a recipe corresponding to the menu and sends it to the terminal. This recipe is then organized for later use in the guide.

[0053] Step 6:

[0054] The device provides the user with recipe information received from the server, either visually or audibly. For example, a smart speaker might start a voice guidance saying, "Season the chicken with salt and pepper."

[0055] Step 7:

[0056] Users provide feedback via their device after cooking and eating. This feedback is used to improve future menus. For example, comments such as "It was too spicy" are recorded.

[0057] (Example 1)

[0058] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0059] In modern households, the increasing time and effort required for meal preparation is a challenge. Furthermore, creating nutritionally balanced menus and appropriately supplementing any deficiencies in necessary foods is not easy. Traditional methods make these tasks cumbersome, resulting in difficulty maintaining a healthy diet. Therefore, there is a need for systems that support meal preparation in an efficient and automated way.

[0060] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0061] In this invention, the server includes means for receiving food information from a status detection device installed in a home storage device, means for generating nutritional guidelines considering the user's profile information, means for generating a menu using the nutritional guidelines and food information, and ordering any missing food items through an external supply source, and means for generating cooking procedures based on the menu and guiding the user through an output device. This automates the process of efficient and healthy meal preparation, making it easier for users to manage their diet.

[0062] "Storage equipment" refers to specialized equipment used to preserve food and maintain its quality, and includes refrigerators and pantries.

[0063] A "condition detection device" is a technology used to detect the type, quantity, and expiration date of food within a storage device, and includes RFID sensors and smart cameras.

[0064] "Food information" refers to data obtained from a condition detection device, such as the type of food, remaining quantity, and expiration date.

[0065] "User profile information" refers to information about the user and their family, such as age, health status, and nutritional requirements.

[0066] "Nutritional guidelines" refer to standards that suggest optimal nutritional intake based on the user's profile information.

[0067] A "menu" is a list that shows the combination of dishes and ingredients planned for each meal.

[0068] "Unsold food" refers to ingredients that are necessary to fulfill a menu but are not sufficiently available in the storage device.

[0069] "External sources" refer to online stores or food suppliers that can provide unmet food needs.

[0070] "Cooking instructions" refer to a guide that outlines the instructions and steps for completing a dish based on a specific menu.

[0071] An "output device" is a device used to communicate cooking instructions to the user, and examples include smartphones and smart speakers.

[0072] The embodiment of this invention is based on a storage device, a status detection device, a server, a terminal, and user interaction within the home. The aim of this system is to automate food management in each household and to provide nutritionally balanced meals in a planned manner.

[0073] The server first collects food information from condition detection devices installed in home storage systems. Using hardware such as RFID sensors and smart cameras, it detects the type, quantity, and expiration date of food, and stores this information in a database system (e.g., MySQL®). Based on this information, the server utilizes AI algorithms (e.g., scikit-learn), taking into account the user's profile, to generate nutritional guidelines. For example, growing children might be recommended a menu high in protein.

[0074] The server then combines nutritional guidelines and food information to generate meal plans. During this process, it also uses a nutrition database (e.g., the USDA food database) to optimize the overall nutritional value. Based on the generated meal plans, the server identifies any missing foods and places orders with external suppliers (e.g., online stores).

[0075] The device receives cooking instructions generated from the server and provides users with visual or audio guidance. Smartphones and smart speakers are used to guide users through specific steps, such as "add olive oil to the pot." Users can send feedback to the server via the device during and after cooking. This feedback is then used to generate future nutritional guidelines and menus.

[0076] A concrete example of sending a prompt to a generative AI model is, "Please generate a nutritionally balanced weekly meal plan and specific recipes based on the latest food inventory and family nutrition profile." This prompt allows the system to provide suggestions tailored to the family's needs and support efficient meal preparation.

[0077] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0078] Step 1:

[0079] The server collects food information obtained from status detection devices. This information is captured by RFID sensors and smart cameras. Specific inputs include food type, remaining quantity, and expiration date, which are stored in a database system. This allows for monitoring of the food situation within the household.

[0080] Step 2:

[0081] The server uses collected food information to match it with user profile information and generate nutritional guidelines. Input includes profile information such as age, health status, and allergy information, and an AI algorithm is used to generate nutritional guidelines optimized for each household as output. In this process, a baseline menu is determined based on health data.

[0082] Step 3:

[0083] The server generates menus based on nutritional guidelines and food information. This step involves referencing a nutrition database and calculating nutritional values. Specifically, it outputs a menu list that takes into account seasonal ingredients and necessary vitamins and minerals. This ensures that menus are suggested that meet the user's nutritional needs.

[0084] Step 4:

[0085] The server compares the generated menu with current inventory to identify any missing food items. Input data includes a list of required ingredients based on the menu and inventory information. By comparing these, the server outputs a list of insufficient ingredients. This information enables efficient ingredient management.

[0086] Step 5:

[0087] The server automatically orders any missing food items from external suppliers. Specifically, it makes requests through an ordering platform API, enabling the automatic procurement of unmet food needs. Order data is generated and sent to the external service, resulting in the replenishment of the food.

[0088] Step 6:

[0089] The terminal provides cooking instructions to the user based on information received from the server. Input data includes detailed recipes for the menu, and output provides visual or audio instructions. This allows the user to proceed with cooking by following the guide.

[0090] Step 7:

[0091] Users send feedback on their cooking experience to the server via their device. This feedback consists of comments and ratings from users, which are accumulated and used by the AI ​​model to improve future menu generation. This feedback system encourages continuous improvement.

[0092] (Application Example 1)

[0093] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0094] In modern households, preparing efficient and nutritionally balanced meals is difficult, and many families face time constraints and considerable effort. This problem is particularly pronounced in dual-income and busy families. Furthermore, it is difficult to satisfy the individual preferences and nutritional needs of each family, making solutions to these challenges essential.

[0095] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0096] In this invention, the server includes means for acquiring food information from a plurality of sensors installed in a home storage device, means for identifying missing ingredients based on the food information, and means for generating cooking instructions related to the missing ingredients. This makes it possible for even busy households to easily prepare nutritionally balanced meals.

[0097] A "sensor" is a device installed in a home storage system to acquire food information.

[0098] "Food information" refers to data regarding the type, quantity, and expiration date of stored food items.

[0099] "Ingredients that are not yet available" are ingredients that are necessary for cooking based on nutritional balance but are not currently on hand.

[0100] "Cooking instructions" refer to the cooking procedures and guides provided to users based on the generated menu.

[0101] "Feedback" refers to evaluations and opinions provided by users based on their past cooking experiences.

[0102] The "means of automation" refer to a system that orders necessary ingredients through an external purchasing system.

[0103] An "output device" is a device that provides information or instructions to users visually or audibly.

[0104] "Individual user information" refers to data about the user's age, preferences, health status, etc.

[0105] A "nutritional plan" is a nutritionally balanced menu created taking into account the individual information of the user.

[0106] A "cooking support device" is a device used to assist and provide instructions during the cooking process at home.

[0107] This invention's system supports the efficient and nutritionally balanced preparation of meals within the home. Specifically, it uses multiple sensors installed in the home's storage device to acquire food information such as the type, quantity, and expiration date of the food. The sensors are responsible for transmitting the food information to a server in real time.

[0108] The server analyzes current inventory levels based on collected food information and identifies any missing ingredients. This analysis process uses an AI algorithm to generate menus. Furthermore, it creates cooking instructions based on the generated menus and develops detailed cooking guides that reflect nutritional balance to be considered during cooking and the individual preferences of the user.

[0109] If a robot is present as a cooking assistance device, the server sends cooking instructions to the robot and guides the user through the cooking process using voice or a display. A robot equipped with voice recognition capabilities receives feedback from the user, stores this information, and uses it to generate menus for the next time.

[0110] For example, if the server suggests "stir-fried vegetables" and "soup," a sensor detects a shortage of carrots and automatically orders additional carrots using the ordering system. During cooking, the robot provides instructions such as, "Please use one tablespoon of sesame oil when stir-frying." Based on the feedback, the amount of carrots can be adjusted next time.

[0111] Using a generative AI model, the following prompt statements can be used.

[0112] "Based on today's refrigerator inventory, please suggest a dinner menu that meets the following conditions: serves 4 people, can be prepared in 30 minutes or less, and uses ingredients rich in iron."

[0113] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0114] Step 1:

[0115] The server acquires food information from multiple sensors installed in home storage devices. Input includes real-time data from the sensors, including food type, quantity, and expiration date. The server processes this data to generate output that clearly shows the current inventory status.

[0116] Step 2:

[0117] The server uses an AI algorithm to generate nutritionally balanced menus based on acquired food information. Inputs include information on available food items and the user's individual nutritional needs, and the output is a suggested menu. At this stage, an analysis is performed to identify any missing ingredients in the menu.

[0118] Step 3:

[0119] The server operates an automated ordering system based on information about unfulfilled ingredients. The input is a list of missing ingredients. Based on this, it sends order requests to an external purchasing system via a database and API, and arranges for the supply of the necessary ingredients.

[0120] Step 4:

[0121] The server creates detailed cooking instructions based on the generated menu and transmits them to the robot, which is a cooking support device. The input is the menu details, and the output is specific cooking instructions for the user. The robot assists with the cooking process through voice and a display.

[0122] Step 5:

[0123] The user cooks according to voice guidance and display information from the cooking assistance device, and sends feedback received during the process to the server via a terminal. This feedback is sent to the server as input data and used to improve the next menu and cooking instructions.

[0124] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0125] This invention is a system that includes an AI agent that works in conjunction with in-home storage devices to streamline meal preparation for dual-income households. It also incorporates an emotion engine that recognizes the user's emotions and optimizes eating and drinking instructions based on those emotions.

[0126] The server periodically receives food information from status detection devices installed in home refrigerators and pantries. This food information includes the type, quantity, and expiration date of the food. For example, the server might retrieve information indicating that there are six eggs remaining in the refrigerator.

[0127] Based on the collected food information, the server creates menus that take nutritional guidelines into consideration. In this process, algorithms are used that take into account the user's individual health status and preferences. Furthermore, the user's emotional state is also analyzed, and the menu is fine-tuned accordingly. For example, if a user has recently been feeling stressed, the menu will be adjusted to include foods with relaxing effects.

[0128] The server compares current food inventory with the planned menu, identifies any missing ingredients, and automatically places orders. For example, if it determines that there isn't enough lettuce for a salad, the server automatically orders lettuce from a partnered online store.

[0129] The emotion engine analyzes the user's emotions through voice input and facial recognition via the camera. This analysis is sent to a server and used to generate instructions for the next meal. For example, if the user smiles frequently during meals, the instructions will be optimized to maintain a similar menu configuration for the next meal.

[0130] The terminal provides the user with cooking instructions based on the menu received from the server. Specific guidance, such as "Next, slice the tomatoes thinly," is provided via voice through a smart speaker.

[0131] Users can provide feedback via their device after a meal, including emotional responses regarding their food preferences. This feedback is used to improve future menus and analyze emotional responses.

[0132] In this way, this system effectively supports meal preparation at home and enables a personalized dining experience that responds to the user's emotions.

[0133] The following describes the processing flow.

[0134] Step 1:

[0135] The server receives food information from status detection devices installed in refrigerators and pantries within the home. This information includes the type, quantity, and expiration date of the food. For example, the server confirms that there are two packs of yogurt and that the expiration date is in three days.

[0136] Step 2:

[0137] The server generates nutritionally balanced menus based on collected food information. This process also considers the user's health status, allergy information, and past eating history. For example, if a user is prone to vitamin deficiencies, the server might suggest a menu that includes a salad.

[0138] Step 3:

[0139] The emotion engine identifies the user's current emotional state through the device. This uses voice input and facial recognition via the camera to determine the user's stress level and interests. For example, if a user is often smiling, that emotion is registered as "relaxed."

[0140] Step 4:

[0141] The server adjusts the menu based on the analysis results of the emotion engine. If the user is in a relaxed state, it will either maintain the planned dishes in the existing menu or add ingredients that are known to have a relaxing effect. For example, it might suggest adding chamomile tea.

[0142] Step 5:

[0143] The server re-checks inventory levels based on the created menu and identifies any missing ingredients. Based on this, it automatically orders the missing food items via an external online store API. For example, if croutons for a salad are missing, it will order croutons from an online store.

[0144] Step 6:

[0145] The terminal provides the user with menus and cooking instructions received from the server. Through a smartphone app or smart speaker, it provides cooking guidance visually or audibly, such as "We're going to boil the pasta now, I'll set an 8-minute timer."

[0146] Step 7:

[0147] Users submit feedback about their cooked meals via their devices. This feedback includes emotional satisfaction with specific dishes. For example, they might send comments to the server such as, "This soup was too salty."

[0148] Step 8:

[0149] The server uses user feedback and emotion engine data to inform the next menu. This ensures that users continuously receive the most satisfying dining experience.

[0150] (Example 2)

[0151] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0152] In dual-income households, daily meal preparation must be done efficiently within a limited timeframe, requiring careful food management within the home, maintenance of health, and optimization of meals according to individual emotional states. To address this challenge, a system is needed that comprehensively utilizes food information, individual user information, and emotional states to proactively support meal preparation, reduce food waste, and provide nutritionally balanced meals.

[0153] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0154] In this invention, the server includes means for receiving food information from multiple state detection devices installed in a home storage device, means for identifying unfulfilled food items, and means for collecting and analyzing the user's emotions. This makes it possible to suggest meals that take into account the user's health condition and emotions.

[0155] A "condition detection device" is a sensor system installed in household storage equipment that periodically detects information such as the type, quantity, and expiration date of food.

[0156] "Food information" refers to data on the type, quantity, and expiration date of food items acquired by a condition detection device, and is used for managing food ingredients in the home.

[0157] "Unavailable food items" refer to food items that the server determines are not currently present in the household and need to be procured when creating a menu.

[0158] "Dining instructions" refer to instructions generated by AI, taking into account the user's health condition, preferences, emotions, etc., that include specific meal content and necessary cooking procedures.

[0159] A "means of collecting and analyzing emotions" refers to a system that captures a user's voice input and facial expressions and analyzes their emotional state; it is sometimes called an emotion engine.

[0160] "Nutritional guidelines" are standards and guidelines established to suggest a balanced diet based on the user's health condition, preferences, and even their current emotional state.

[0161] This invention is a system that includes an AI agent to streamline meal preparation for dual-income households. A server receives food information from a status detection device installed in the household's storage system and creates a menu based on this information. The server uses IoT devices to collect data such as the type, quantity, and expiration date of food, and then processes it. Specifically, it utilizes a generative AI model to generate a menu that takes nutritional guidelines into consideration based on the collected data. This process takes into account not only the user's health status and preferences, but also the collected emotional data.

[0162] The device collects user emotional data using voice input and a camera, and analyzes it using an emotion engine. The analysis results are sent to a server and used to provide guidance for future meals and drinks. If the user's emotional state indicates stress, the server suggests a menu that includes foods with relaxing effects. The server also checks for missing foods and automatically places orders with partner online stores.

[0163] Cooking instructions are provided to the user via a device and guided by voice through a smart speaker. For example, specific instructions such as "Next, slice the tomatoes thinly" are provided by voice. After the meal, the user provides feedback via the device, which is used to improve the menu for the next meal.

[0164] As a concrete example, there is a server that receives information that "six eggs remain" in the refrigerator. If the user feels "tired recently," the server suggests providing a relaxing meal. The following prompt is entered into the generative AI model: "Describe in detail a system that creates a menu considering nutritional guidelines based on data from a home refrigerator and pantry, and personalizes the dining experience using data obtained from emotion recognition."

[0165] This system can improve users' time efficiency, contribute to optimizing meal content, and reduce the burden of meal preparation for dual-income households by providing nutritionally balanced menus.

[0166] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0167] Step 1:

[0168] The server receives food information from a status detection device installed in the home storage unit. Specifically, the status detection device detects the type, quantity, and expiration date of the food and sends this data to the server. Based on this information, the server builds a list of the food currently in the storage unit.

[0169] Step 2:

[0170] The server uses a generative AI model to create menus based on collected food information. It receives food data and user health and preference information as input, performs data calculations, and outputs optimal menu suggestions. Specifically, the model generates menus while considering nutritional balance.

[0171] Step 3:

[0172] The device collects and analyzes the user's emotions. Specifically, it detects the user's facial expressions using a camera on the device and recognizes their emotional state through voice input. This input data is sent to an emotion engine, which generates a report on the user's emotional state and outputs it to the server.

[0173] Step 4:

[0174] The server optimizes the suggested menu based on the emotional state report. Specifically, it uses the input emotional data to add ingredients with relaxing effects or make adjustments to suit specific emotions. The adjusted menu is output as the following instructions.

[0175] Step 5:

[0176] The server identifies ingredients that are missing from the household based on the suggested menu and automatically places an order with the online store. It receives a comparison of the suggested menu's ingredient list with the household's inventory as input, identifies the missing ingredients, and outputs an order request.

[0177] Step 6:

[0178] The terminal provides the user with the adjusted menu and cooking instructions received from the server via voice. Specifically, the smart speaker assists the user in cooking by giving instructions such as, "Next, slice the tomatoes thinly."

[0179] Step 7:

[0180] Users provide feedback via their device after their meal. This feedback includes emotional responses such as satisfaction with the meal and taste preferences, and this data is sent to the server to be used in creating future menus.

[0181] (Application Example 2)

[0182] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0183] For dual-income households, preparing daily meals is a time-consuming and laborious task. In this environment, efficiently preparing personalized meals based on the user's emotions and health condition is difficult. A system is needed to solve this problem and enable users to prepare meals comfortably.

[0184] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0185] In this invention, the server includes means for receiving item information from multiple situation detection devices installed in a storage device within the home, means for generating food and drink instructions related to unfulfilled items, and means for recognizing the user's emotions and reflecting them in the next meal instructions. This enables personalized meal suggestions and automatic ordering that take into account the user's emotional state.

[0186] "Household storage devices" refer to devices used to store and preserve items within the home, such as refrigerators and pantries.

[0187] A "situation detection device" refers to a device capable of detecting information such as the type, quantity, quality, or expiration date of an item.

[0188] "Item information" refers to data about items stored in the home, acquired by a situation detection device.

[0189] "Items not yet fulfilled" refers to items that need to be purchased additionally to meet pre-set conditions or the user's preferences.

[0190] "Food and beverage instructions" refer to specific meal menus and instructions regarding the preparation of ingredients, generated based on product information.

[0191] "Information device" refers to a device used to display or notify information visually or audibly.

[0192] "User's emotions" refers to the specific psychological state analyzed through voice input and image processing.

[0193] To realize this invention, multiple situation detection devices placed within the home will periodically acquire information about the items in refrigerators and pantries. Temperature sensors and RFID scanners, among others, will be used for this purpose. A server will receive this information from these devices and store it in a database. This database will store detailed information such as the type, quantity, and expiration date of the items.

[0194] The server is equipped with sentiment analysis software to recognize the user's emotions. This software uses the camera and microphone on the user's device (e.g., smartphone or consumer robot) to analyze facial expressions and voice tone. External services such as Google's Cloud Vision API and Microsoft's Emotion API are used for sentiment analysis.

[0195] The server generates food and drink instructions using an AI engine based on item information and emotional data. This AI engine considers the user's health condition and preferences, and proposes the optimal menu using a generated AI model. The server also utilizes machine learning algorithms to improve the accuracy of the menu based on past data and feedback.

[0196] As information devices, smart devices with voice assistants are commonly used. Food and drink instructions are provided to the user through voice or display. For example, instructions such as "Next, slice the tomatoes thinly" are provided audibly through a smart speaker.

[0197] As a concrete example, imagine a system where, upon returning home from work, a robot says, "Welcome back. You seem tired today, so I suggest a relaxing herb salad," and the server automatically orders the appropriate ingredients. Examples of prompts to consider in this case include, "Please use an AI model that suggests tonight's dinner menu, taking into account the user's emotional state," or "Please guide the user through the process of checking the refrigerator inventory and ordering the necessary ingredients."

[0198] In this way, meal preparation at home becomes more efficient, and a personalized dining experience based on the user's emotional state is realized.

[0199] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0200] Step 1:

[0201] The server periodically collects item information from household status detection devices. It receives item data transmitted from these devices as input. The server analyzes this data to identify the type, quantity, and expiration date of items, and registers this information in a database. This ensures that the latest inventory status is always known.

[0202] Step 2:

[0203] The user sends emotional data to the server via their device. An emotion recognition application running on the device analyzes the user's facial expressions and voice based on data acquired from the camera and microphone. Image and audio data related to emotions are taken as input, and the analyzed emotional state (e.g., stress, relaxation) is provided to the server as output.

[0204] Step 3:

[0205] The server generates food and drink instructions using a generative AI model based on item information and user sentiment data. The input is the data accumulated in the previous two steps. The AI ​​engine uses this data to calculate the optimal menu, taking into account health status and preferences. The output is specific menu information provided to the user.

[0206] Step 4:

[0207] The server identifies missing items and places an automated order. It confirms the items required based on the food and beverage instructions and executes the ordering process through the online store's API. The input is a list of required ingredients based on the menu determined in step 3, and the output is the confirmed order information.

[0208] Step 5:

[0209] The user receives cooking instructions via a terminal. Eating instructions sent from the server are displayed on the terminal. Cooking steps are guided as specific instructions using a voice output device such as a smart speaker. The input is eating instructions from the server, and the output is visual or audio instructions received by the user to perform the cooking.

[0210] Step 6:

[0211] After a meal, the user sends feedback to the server via their device. This feedback is then used to inform future dining instructions. The input is user feedback data, and the output is the server updating its stored data for future use.

[0212] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0213] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0214] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0215] [Second Embodiment]

[0216] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0217] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0218] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0219] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0220] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0221] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0222] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0223] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0224] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0225] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0226] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0227] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0228] The system of this invention utilizes an AI agent designed to assist with meal preparation at home. This system functions primarily through interaction between a server, a terminal, and the user. Its specific operation is described below.

[0229] The server periodically collects food information from status detection devices installed in home refrigerators and pantries. This information includes the type, quantity, and expiration date of the food. For example, the server periodically checks the remaining amount and expiration date of milk in the refrigerator.

[0230] Based on the collected food information, the server creates a menu that takes into account the child's age and nutritional needs. This menu also includes an overall nutritional calculation to ensure nutritional balance. For example, a plan might be made to include spinach, which is rich in iron, during certain periods.

[0231] Next, the server compares the created menu with the current food inventory to identify any missing items. Based on this information, the server automatically orders the missing items from an external online store. For example, if the menu includes tomato soup and there is a shortage of tomatoes, the server will order tomatoes.

[0232] Furthermore, the server generates a recipe based on the created menu and sends it to the device. The device then guides the user visually or audibly with this recipe information via a smartphone or smart speaker. For example, a smart speaker might instruct the user to "First, put olive oil in the pot."

[0233] Users can provide feedback through their device during or after cooking. This allows the server to collect this feedback and use it to improve future menus and suggestions. For example, if a user inputs "Please use fewer spices next time," that information will be used in the next menu.

[0234] Thus, the system of this invention helps dual-income families efficiently prepare nutritionally balanced meals, significantly reducing the time and effort spent at home.

[0235] The following describes the processing flow.

[0236] Step 1:

[0237] The server obtains food information from household status detection devices. This includes the type, quantity, and expiration date of food items in the refrigerator and pantry. For example, the server recognizes that there is 200ml of milk remaining.

[0238] Step 2:

[0239] The server analyzes the collected food information and generates a nutritionally balanced menu that takes into account the child's age and allergy information. For example, if iron is needed, sautéed spinach will be added to the menu.

[0240] Step 3:

[0241] The server creates a list of necessary ingredients based on the generated menu and compares it with the current inventory to identify any missing foods. For example, it might find that two tomatoes are missing.

[0242] Step 4:

[0243] The server automatically places orders for identified missing ingredients via an external online store API. For example, it might order 5 tomatoes online.

[0244] Step 5:

[0245] The server generates a recipe corresponding to the menu and sends it to the terminal. This recipe is then organized for later use in the guide.

[0246] Step 6:

[0247] The device provides the user with recipe information received from the server, either visually or audibly. For example, a smart speaker might start a voice guidance saying, "Season the chicken with salt and pepper."

[0248] Step 7:

[0249] Users provide feedback via their device after cooking and eating. This feedback is used to improve future menus. For example, comments such as "It was too spicy" are recorded.

[0250] (Example 1)

[0251] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0252] In modern households, the increasing time and effort required for meal preparation is a challenge. Furthermore, creating nutritionally balanced menus and appropriately supplementing any deficiencies in necessary foods is not easy. Traditional methods make these tasks cumbersome, resulting in difficulty maintaining a healthy diet. Therefore, there is a need for systems that support meal preparation in an efficient and automated way.

[0253] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0254] In this invention, the server includes means for receiving food information from a status detection device installed in a home storage device, means for generating nutritional guidelines considering the user's profile information, means for generating a menu using the nutritional guidelines and food information, and ordering any missing food items through an external supply source, and means for generating cooking procedures based on the menu and guiding the user through an output device. This automates the process of efficient and healthy meal preparation, making it easier for users to manage their diet.

[0255] "Storage equipment" refers to specialized equipment used to preserve food and maintain its quality, and includes refrigerators and pantries.

[0256] A "condition detection device" is a technology used to detect the type, quantity, and expiration date of food within a storage device, and includes RFID sensors and smart cameras.

[0257] "Food information" refers to data obtained from a condition detection device, such as the type of food, remaining quantity, and expiration date.

[0258] "User profile information" refers to information about the user and their family, such as age, health status, and nutritional requirements.

[0259] "Nutritional guidelines" refer to standards that suggest optimal nutritional intake based on the user's profile information.

[0260] A "menu" is a list that shows the combination of dishes and ingredients planned for each meal.

[0261] "Unsold food" refers to ingredients that are necessary to fulfill a menu but are not sufficiently available in the storage device.

[0262] "External sources" refer to online stores or food suppliers that can provide unmet food needs.

[0263] "Cooking instructions" refer to a guide that outlines the instructions and steps for completing a dish based on a specific menu.

[0264] An "output device" is a device used to communicate cooking instructions to the user, and examples include smartphones and smart speakers.

[0265] The embodiment of this invention is based on a storage device, a status detection device, a server, a terminal, and user interaction within the home. The aim of this system is to automate food management in each household and to provide nutritionally balanced meals in a planned manner.

[0266] The server first collects food information from condition detection devices installed in home storage systems. Using hardware such as RFID sensors and smart cameras, it detects the type, quantity, and expiration date of food, and stores this information in a database system (e.g., MySQL). Based on this information, the server utilizes AI algorithms (e.g., scikit-learn), taking into account the user's profile, to generate nutritional guidelines. For example, growing children might be recommended a menu high in protein.

[0267] The server then combines nutritional guidelines and food information to generate meal plans. During this process, it also uses a nutrition database (e.g., the USDA food database) to optimize the overall nutritional value. Based on the generated meal plans, the server identifies any missing foods and places orders with external suppliers (e.g., online stores).

[0268] The device receives cooking instructions generated from the server and provides users with visual or audio guidance. Smartphones and smart speakers are used to guide users through specific steps, such as "add olive oil to the pot." Users can send feedback to the server via the device during and after cooking. This feedback is then used to generate future nutritional guidelines and menus.

[0269] A concrete example of sending a prompt to a generative AI model is, "Please generate a nutritionally balanced weekly meal plan and specific recipes based on the latest food inventory and family nutrition profile." This prompt allows the system to provide suggestions tailored to the family's needs and support efficient meal preparation.

[0270] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0271] Step 1:

[0272] The server collects food information obtained from status detection devices. This information is captured by RFID sensors and smart cameras. Specific inputs include food type, remaining quantity, and expiration date, which are stored in a database system. This allows for monitoring of the food situation within the household.

[0273] Step 2:

[0274] The server uses collected food information to match it with user profile information and generate nutritional guidelines. Input includes profile information such as age, health status, and allergy information, and an AI algorithm is used to generate nutritional guidelines optimized for each household as output. In this process, a baseline menu is determined based on health data.

[0275] Step 3:

[0276] The server generates menus based on nutritional guidelines and food information. This step involves referencing a nutrition database and calculating nutritional values. Specifically, it outputs a menu list that takes into account seasonal ingredients and necessary vitamins and minerals. This ensures that menus are suggested that meet the user's nutritional needs.

[0277] Step 4:

[0278] The server compares the generated menu with current inventory to identify any missing food items. Input data includes a list of required ingredients based on the menu and inventory information. By comparing these, the server outputs a list of insufficient ingredients. This information enables efficient ingredient management.

[0279] Step 5:

[0280] The server automatically orders any missing food items from external suppliers. Specifically, it makes requests through an ordering platform API, enabling the automatic procurement of unmet food needs. Order data is generated and sent to the external service, resulting in the replenishment of the food.

[0281] Step 6:

[0282] The terminal provides cooking instructions to the user based on information received from the server. Input data includes detailed recipes for the menu, and output provides visual or audio instructions. This allows the user to proceed with cooking by following the guide.

[0283] Step 7:

[0284] The user sends feedback on the cooking experience to the server through the terminal. The input is comments or evaluations from the user, which are accumulated and reflected by the AI model in the generation of the next menu. The feedback system promotes continuous improvement.

[0285] (Application Example 1)

[0286] Next, Application Example 1 will be described. In the following description, the data processing device 12 is referred to as the "server", and the smart glasses 214 are referred to as the "terminal".

[0287] In modern households, it is difficult to prepare efficient and nutritionally balanced meals. In many households, there are time constraints and it is laborious. This problem is particularly prominent in dual-income households and busy households. In addition, it is difficult to meet the preferences and nutritional needs of individual households, so there is a need to solve these problems.

[0288] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0289] In this invention, the server includes means for acquiring food information from a plurality of sensors installed in a storage device within a household, means for identifying insufficient ingredients based on the food information, and means for generating cooking instructions related to the insufficient ingredients. This makes it possible for busy households to easily prepare nutritionally balanced meals.

[0290] A "sensor" is a device installed in a storage device within a household to acquire food information.

[0291] "Food information" is data regarding the types, amounts, and expiration dates of the stored ingredients.

[0292] "Insufficient ingredients" are ingredients that are necessary for cooking based on nutritional balance but are not currently available.

[0293] "Cooking instructions" refer to the cooking procedures and guides provided to users based on the generated menu.

[0294] "Feedback" refers to evaluations and opinions provided by users based on their past cooking experiences.

[0295] The "means of automation" refer to a system that orders necessary ingredients through an external purchasing system.

[0296] An "output device" is a device that provides information or instructions to users visually or audibly.

[0297] "Individual user information" refers to data about the user's age, preferences, health status, etc.

[0298] A "nutritional plan" is a nutritionally balanced menu created taking into account the individual information of the user.

[0299] A "cooking support device" is a device used to assist and provide instructions during the cooking process at home.

[0300] This invention's system supports the efficient and nutritionally balanced preparation of meals within the home. Specifically, it uses multiple sensors installed in the home's storage device to acquire food information such as the type, quantity, and expiration date of the food. The sensors are responsible for transmitting the food information to a server in real time.

[0301] The server analyzes current inventory levels based on collected food information and identifies any missing ingredients. This analysis process uses an AI algorithm to generate menus. Furthermore, it creates cooking instructions based on the generated menus and develops detailed cooking guides that reflect nutritional balance to be considered during cooking and the individual preferences of the user.

[0302] When there is a robot as a cooking support device, the server sends cooking instructions to the robot and guides the user through the cooking procedure using voice or a display. A robot equipped with a voice recognition function receives feedback from the user, accumulates the information, and uses it for the next menu generation.

[0303] As a specific example, when the server proposes "fried vegetables" and "soup", if the sensor detects a shortage of carrots, it automatically orders additional carrots using an ordering system. During cooking, the robot guides, saying "Please use 1 tablespoon of sesame oil when frying." Based on the feedback, it becomes possible to adjust the amount of carrots next time.

[0304] Using a generation AI model, the following prompt sentences can be used.

[0305] "Based on today's refrigerator inventory information, please propose a dinner menu that satisfies the following conditions: for a family of 4, can be cooked within 30 minutes, and uses ingredients rich in iron."

[0306] The flow of the specific process in Application Example 1 will be described using FIG. 12.

[0307] Step 1:

[0308] The server acquires food information from a plurality of sensors installed in storage devices within the home. The input includes real-time data from the sensors, including the type, quantity, and expiration date of the food. The server processes this data and generates an output that clarifies the current inventory status.

[0309] [[ID=​​​​

[0311] Step 3:

[0312] The server operates an automated ordering system based on information about unfulfilled ingredients. The input is a list of missing ingredients. Based on this, it sends order requests to an external purchasing system via a database and API, and arranges for the supply of the necessary ingredients.

[0313] Step 4:

[0314] The server creates detailed cooking instructions based on the generated menu and transmits them to the robot, which is a cooking support device. The input is the menu details, and the output is specific cooking instructions for the user. The robot assists with the cooking process through voice and a display.

[0315] Step 5:

[0316] The user cooks according to voice guidance and display information from the cooking assistance device, and sends feedback received during the process to the server via a terminal. This feedback is sent to the server as input data and used to improve the next menu and cooking instructions.

[0317] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0318] This invention is a system that includes an AI agent that works in conjunction with in-home storage devices to streamline meal preparation for dual-income households. It also incorporates an emotion engine that recognizes the user's emotions and optimizes eating and drinking instructions based on those emotions.

[0319] The server periodically receives food information from status detection devices installed in home refrigerators and pantries. This food information includes the type, quantity, and expiration date of the food. For example, the server might retrieve information indicating that there are six eggs remaining in the refrigerator.

[0320] Based on the collected food information, the server creates menus that take nutritional guidelines into consideration. In this process, algorithms are used that take into account the user's individual health status and preferences. Furthermore, the user's emotional state is also analyzed, and the menu is fine-tuned accordingly. For example, if a user has recently been feeling stressed, the menu will be adjusted to include foods with relaxing effects.

[0321] The server compares current food inventory with the planned menu, identifies any missing ingredients, and automatically places orders. For example, if it determines that there isn't enough lettuce for a salad, the server automatically orders lettuce from a partnered online store.

[0322] The emotion engine analyzes the user's emotions through voice input and facial recognition via the camera. This analysis is sent to a server and used to generate instructions for the next meal. For example, if the user smiles frequently during meals, the instructions will be optimized to maintain a similar menu configuration for the next meal.

[0323] The terminal provides the user with cooking instructions based on the menu received from the server. Specific guidance, such as "Next, slice the tomatoes thinly," is provided via voice through a smart speaker.

[0324] Users can provide feedback via their device after a meal, including emotional responses regarding their food preferences. This feedback is used to improve future menus and analyze emotional responses.

[0325] In this way, this system effectively supports meal preparation at home and enables a personalized dining experience that responds to the user's emotions.

[0326] The following describes the processing flow.

[0327] Step 1:

[0328] The server receives food information from status detection devices installed in refrigerators and pantries within the home. This information includes the type, quantity, and expiration date of the food. For example, the server confirms that there are two packs of yogurt and that the expiration date is in three days.

[0329] Step 2:

[0330] The server generates nutritionally balanced menus based on collected food information. This process also considers the user's health status, allergy information, and past eating history. For example, if a user is prone to vitamin deficiencies, the server might suggest a menu that includes a salad.

[0331] Step 3:

[0332] The emotion engine identifies the user's current emotional state through the device. This uses voice input and facial recognition via the camera to determine the user's stress level and interests. For example, if a user is often smiling, that emotion is registered as "relaxed."

[0333] Step 4:

[0334] The server adjusts the menu based on the analysis results of the emotion engine. If the user is in a relaxed state, it will either maintain the planned dishes in the existing menu or add ingredients that are known to have a relaxing effect. For example, it might suggest adding chamomile tea.

[0335] Step 5:

[0336] The server re-checks inventory levels based on the created menu and identifies any missing ingredients. Based on this, it automatically orders the missing food items via an external online store API. For example, if croutons for a salad are missing, it will order croutons from an online store.

[0337] Step 6:

[0338] The terminal provides the user with menus and cooking instructions received from the server. Through a smartphone app or smart speaker, it provides cooking guidance visually or audibly, such as "We're going to boil the pasta now, I'll set an 8-minute timer."

[0339] Step 7:

[0340] Users submit feedback about their cooked meals via their devices. This feedback includes emotional satisfaction with specific dishes. For example, they might send comments to the server such as, "This soup was too salty."

[0341] Step 8:

[0342] The server uses user feedback and emotion engine data to inform the next menu. This ensures that users continuously receive the most satisfying dining experience.

[0343] (Example 2)

[0344] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0345] In dual-income households, daily meal preparation must be done efficiently within a limited timeframe, requiring careful food management within the home, maintenance of health, and optimization of meals according to individual emotional states. To address this challenge, a system is needed that comprehensively utilizes food information, individual user information, and emotional states to proactively support meal preparation, reduce food waste, and provide nutritionally balanced meals.

[0346] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0347] In this invention, the server includes means for receiving food information from multiple state detection devices installed in a home storage device, means for identifying unfulfilled food items, and means for collecting and analyzing the user's emotions. This makes it possible to suggest meals that take into account the user's health condition and emotions.

[0348] A "condition detection device" is a sensor system installed in household storage equipment that periodically detects information such as the type, quantity, and expiration date of food.

[0349] "Food information" refers to data on the type, quantity, and expiration date of food items acquired by a condition detection device, and is used for managing food ingredients in the home.

[0350] "Unavailable food items" refer to food items that the server determines are not currently present in the household and need to be procured when creating a menu.

[0351] "Dining instructions" refer to instructions generated by AI, taking into account the user's health condition, preferences, emotions, etc., that include specific meal content and necessary cooking procedures.

[0352] A "means of collecting and analyzing emotions" refers to a system that captures a user's voice input and facial expressions and analyzes their emotional state; it is sometimes called an emotion engine.

[0353] "Nutritional guidelines" are standards and guidelines established to suggest a balanced diet based on the user's health condition, preferences, and even their current emotional state.

[0354] This invention is a system that includes an AI agent to streamline meal preparation for dual-income households. A server receives food information from a status detection device installed in the household's storage system and creates a menu based on this information. The server uses IoT devices to collect data such as the type, quantity, and expiration date of food, and then processes it. Specifically, it utilizes a generative AI model to generate a menu that takes nutritional guidelines into consideration based on the collected data. This process takes into account not only the user's health status and preferences, but also the collected emotional data.

[0355] The device collects user emotional data using voice input and a camera, and analyzes it using an emotion engine. The analysis results are sent to a server and used to provide guidance for future meals and drinks. If the user's emotional state indicates stress, the server suggests a menu that includes foods with relaxing effects. The server also checks for missing foods and automatically places orders with partner online stores.

[0356] Cooking instructions are provided to the user via a device and guided by voice through a smart speaker. For example, specific instructions such as "Next, slice the tomatoes thinly" are provided by voice. After the meal, the user provides feedback via the device, which is used to improve the menu for the next meal.

[0357] As a concrete example, there is a server that receives information that "six eggs remain" in the refrigerator. If the user feels "tired recently," the server suggests providing a relaxing meal. The following prompt is entered into the generative AI model: "Describe in detail a system that creates a menu considering nutritional guidelines based on data from a home refrigerator and pantry, and personalizes the dining experience using data obtained from emotion recognition."

[0358] This system can improve users' time efficiency, contribute to optimizing meal content, and reduce the burden of meal preparation for dual-income households by providing nutritionally balanced menus.

[0359] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0360] Step 1:

[0361] The server receives food information from a status detection device installed in the home storage unit. Specifically, the status detection device detects the type, quantity, and expiration date of the food and sends this data to the server. Based on this information, the server builds a list of the food currently in the storage unit.

[0362] Step 2:

[0363] The server uses a generative AI model to create menus based on collected food information. It receives food data and user health and preference information as input, performs data calculations, and outputs optimal menu suggestions. Specifically, the model generates menus while considering nutritional balance.

[0364] Step 3:

[0365] The device collects and analyzes the user's emotions. Specifically, it detects the user's facial expressions using a camera on the device and recognizes their emotional state through voice input. This input data is sent to an emotion engine, which generates a report on the user's emotional state and outputs it to the server.

[0366] Step 4:

[0367] The server optimizes the suggested menu based on the emotional state report. Specifically, it uses the input emotional data to add ingredients with relaxing effects or make adjustments to suit specific emotions. The adjusted menu is output as the following instructions.

[0368] Step 5:

[0369] The server identifies ingredients that are missing from the household based on the suggested menu and automatically places an order with the online store. It receives a comparison of the suggested menu's ingredient list with the household's inventory as input, identifies the missing ingredients, and outputs an order request.

[0370] Step 6:

[0371] The terminal provides the user with the adjusted menu and cooking instructions received from the server via voice. Specifically, the smart speaker assists the user in cooking by giving instructions such as, "Next, slice the tomatoes thinly."

[0372] Step 7:

[0373] Users provide feedback via their device after their meal. This feedback includes emotional responses such as satisfaction with the meal and taste preferences, and this data is sent to the server to be used in creating future menus.

[0374] (Application Example 2)

[0375] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0376] For dual-income households, preparing daily meals is a time-consuming and laborious task. In this environment, efficiently preparing personalized meals based on the user's emotions and health condition is difficult. A system is needed to solve this problem and enable users to prepare meals comfortably.

[0377] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0378] In this invention, the server includes means for receiving item information from multiple situation detection devices installed in a storage device within the home, means for generating food and drink instructions related to unfulfilled items, and means for recognizing the user's emotions and reflecting them in the next meal instructions. This enables personalized meal suggestions and automatic ordering that take into account the user's emotional state.

[0379] "Household storage devices" refer to devices used to store and preserve items within the home, such as refrigerators and pantries.

[0380] A "situation detection device" refers to a device capable of detecting information such as the type, quantity, quality, or expiration date of an item.

[0381] "Item information" refers to data about items stored in the home, acquired by a situation detection device.

[0382] "Items not yet fulfilled" refers to items that need to be purchased additionally to meet pre-set conditions or the user's preferences.

[0383] "Food and beverage instructions" refer to specific meal menus and instructions regarding the preparation of ingredients, generated based on product information.

[0384] "Information device" refers to a device used to display or notify information visually or audibly.

[0385] "User's emotions" refers to the specific psychological state analyzed through voice input and image processing.

[0386] To realize this invention, multiple situation detection devices placed within the home will periodically acquire information about the items in refrigerators and pantries. Temperature sensors and RFID scanners, among others, will be used for this purpose. A server will receive this information from these devices and store it in a database. This database will store detailed information such as the type, quantity, and expiration date of the items.

[0387] The server is equipped with sentiment analysis software to recognize the user's emotions. This software uses the camera and microphone on the user's device (e.g., smartphone or consumer robot) to analyze facial expressions and voice tone. External services such as Google's Cloud Vision API and Microsoft's Emotion API are used for sentiment analysis.

[0388] The server generates food and drink instructions using an AI engine based on item information and emotional data. This AI engine considers the user's health condition and preferences, and proposes the optimal menu using a generated AI model. The server also utilizes machine learning algorithms to improve the accuracy of the menu based on past data and feedback.

[0389] As information devices, smart devices with voice assistants are commonly used. Food and drink instructions are provided to the user through voice or display. For example, instructions such as "Next, slice the tomatoes thinly" are provided audibly through a smart speaker.

[0390] As a concrete example, imagine a system where, upon returning home from work, a robot says, "Welcome back. You seem tired today, so I suggest a relaxing herb salad," and the server automatically orders the appropriate ingredients. Examples of prompts to consider in this case include, "Please use an AI model that suggests tonight's dinner menu, taking into account the user's emotional state," or "Please guide the user through the process of checking the refrigerator inventory and ordering the necessary ingredients."

[0391] In this way, meal preparation at home becomes more efficient, and a personalized dining experience based on the user's emotional state is realized.

[0392] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0393] Step 1:

[0394] The server periodically collects item information from household status detection devices. It receives item data transmitted from these devices as input. The server analyzes this data to identify the type, quantity, and expiration date of items, and registers this information in a database. This ensures that the latest inventory status is always known.

[0395] Step 2:

[0396] The user sends emotional data to the server via their device. An emotion recognition application running on the device analyzes the user's facial expressions and voice based on data acquired from the camera and microphone. Image and audio data related to emotions are taken as input, and the analyzed emotional state (e.g., stress, relaxation) is provided to the server as output.

[0397] Step 3:

[0398] The server generates food and drink instructions using a generative AI model based on item information and user sentiment data. The input is the data accumulated in the previous two steps. The AI ​​engine uses this data to calculate the optimal menu, taking into account health status and preferences. The output is specific menu information provided to the user.

[0399] Step 4:

[0400] The server identifies missing items and places an automated order. It confirms the items required based on the food and beverage instructions and executes the ordering process through the online store's API. The input is a list of required ingredients based on the menu determined in step 3, and the output is the confirmed order information.

[0401] Step 5:

[0402] The user receives cooking instructions via a terminal. Eating instructions sent from the server are displayed on the terminal. Cooking steps are guided as specific instructions using a voice output device such as a smart speaker. The input is eating instructions from the server, and the output is visual or audio instructions received by the user to perform the cooking.

[0403] Step 6:

[0404] After a meal, the user sends feedback to the server via their device. This feedback is then used to inform future dining instructions. The input is user feedback data, and the output is the server updating its stored data for future use.

[0405] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0406] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0407] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0408] [Third Embodiment]

[0409] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0410] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0411] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0412] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0413] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0414] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0415] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0416] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0417] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0418] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0419] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0420] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0421] The system of this invention utilizes an AI agent designed to assist with meal preparation at home. This system functions primarily through interaction between a server, a terminal, and the user. Its specific operation is described below.

[0422] The server periodically collects food information from status detection devices installed in home refrigerators and pantries. This information includes the type, quantity, and expiration date of the food. For example, the server periodically checks the remaining amount and expiration date of milk in the refrigerator.

[0423] Based on the collected food information, the server creates a menu that takes into account the child's age and nutritional needs. This menu also includes an overall nutritional calculation to ensure nutritional balance. For example, a plan might be made to include spinach, which is rich in iron, during certain periods.

[0424] Next, the server compares the created menu with the current food inventory to identify any missing items. Based on this information, the server automatically orders the missing items from an external online store. For example, if the menu includes tomato soup and there is a shortage of tomatoes, the server will order tomatoes.

[0425] Furthermore, the server generates a recipe based on the created menu and sends it to the device. The device then guides the user visually or audibly with this recipe information via a smartphone or smart speaker. For example, a smart speaker might instruct the user to "First, put olive oil in the pot."

[0426] Users can provide feedback through their device during or after cooking. This allows the server to collect this feedback and use it to improve future menus and suggestions. For example, if a user inputs "Please use fewer spices next time," that information will be used in the next menu.

[0427] Thus, the system of this invention helps dual-income families efficiently prepare nutritionally balanced meals, significantly reducing the time and effort spent at home.

[0428] The following describes the processing flow.

[0429] Step 1:

[0430] The server obtains food information from household status detection devices. This includes the type, quantity, and expiration date of food items in the refrigerator and pantry. For example, the server recognizes that there is 200ml of milk remaining.

[0431] Step 2:

[0432] The server analyzes the collected food information and generates a nutritionally balanced menu that takes into account the child's age and allergy information. For example, if iron is needed, sautéed spinach will be added to the menu.

[0433] Step 3:

[0434] The server creates a list of necessary ingredients based on the generated menu and compares it with the current inventory to identify any missing foods. For example, it might find that two tomatoes are missing.

[0435] Step 4:

[0436] The server automatically places orders for identified missing ingredients via an external online store API. For example, it might order 5 tomatoes online.

[0437] Step 5:

[0438] The server generates a recipe corresponding to the menu and sends it to the terminal. This recipe is then organized for later use in the guide.

[0439] Step 6:

[0440] The device provides the user with recipe information received from the server, either visually or audibly. For example, a smart speaker might start a voice guidance saying, "Season the chicken with salt and pepper."

[0441] Step 7:

[0442] Users provide feedback via their device after cooking and eating. This feedback is used to improve future menus. For example, comments such as "It was too spicy" are recorded.

[0443] (Example 1)

[0444] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0445] In modern households, the increasing time and effort required for meal preparation is a challenge. Furthermore, creating nutritionally balanced menus and appropriately supplementing any deficiencies in necessary foods is not easy. Traditional methods make these tasks cumbersome, resulting in difficulty maintaining a healthy diet. Therefore, there is a need for systems that support meal preparation in an efficient and automated way.

[0446] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0447] In this invention, the server includes means for receiving food information from a status detection device installed in a home storage device, means for generating nutritional guidelines considering the user's profile information, means for generating a menu using the nutritional guidelines and food information, and ordering any missing food items through an external supply source, and means for generating cooking procedures based on the menu and guiding the user through an output device. This automates the process of efficient and healthy meal preparation, making it easier for users to manage their diet.

[0448] "Storage equipment" refers to specialized equipment used to preserve food and maintain its quality, and includes refrigerators and pantries.

[0449] A "condition detection device" is a technology used to detect the type, quantity, and expiration date of food within a storage device, and includes RFID sensors and smart cameras.

[0450] "Food information" refers to data obtained from a condition detection device, such as the type of food, remaining quantity, and expiration date.

[0451] "User profile information" refers to information about the user and their family, such as age, health status, and nutritional requirements.

[0452] "Nutritional guidelines" refer to standards that suggest optimal nutritional intake based on the user's profile information.

[0453] A "menu" is a list that shows the combination of dishes and ingredients planned for each meal.

[0454] "Unsold food" refers to ingredients that are necessary to fulfill a menu but are not sufficiently available in the storage device.

[0455] "External sources" refer to online stores or food suppliers that can provide unmet food needs.

[0456] "Cooking instructions" refer to a guide that outlines the instructions and steps for completing a dish based on a specific menu.

[0457] An "output device" is a device used to communicate cooking instructions to the user, and examples include smartphones and smart speakers.

[0458] The embodiment of this invention is based on a storage device, a status detection device, a server, a terminal, and user interaction within the home. The aim of this system is to automate food management in each household and to provide nutritionally balanced meals in a planned manner.

[0459] The server first collects food information from condition detection devices installed in home storage systems. Using hardware such as RFID sensors and smart cameras, it detects the type, quantity, and expiration date of food, and stores this information in a database system (e.g., MySQL). Based on this information, the server utilizes AI algorithms (e.g., scikit-learn), taking into account the user's profile, to generate nutritional guidelines. For example, growing children might be recommended a menu high in protein.

[0460] The server then combines nutritional guidelines and food information to generate meal plans. During this process, it also uses a nutrition database (e.g., the USDA food database) to optimize the overall nutritional value. Based on the generated meal plans, the server identifies any missing foods and places orders with external suppliers (e.g., online stores).

[0461] The device receives cooking instructions generated from the server and provides users with visual or audio guidance. Smartphones and smart speakers are used to guide users through specific steps, such as "add olive oil to the pot." Users can send feedback to the server via the device during and after cooking. This feedback is then used to generate future nutritional guidelines and menus.

[0462] A concrete example of sending a prompt to a generative AI model is, "Please generate a nutritionally balanced weekly meal plan and specific recipes based on the latest food inventory and family nutrition profile." This prompt allows the system to provide suggestions tailored to the family's needs and support efficient meal preparation.

[0463] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0464] Step 1:

[0465] The server collects food information obtained from status detection devices. This information is captured by RFID sensors and smart cameras. Specific inputs include food type, remaining quantity, and expiration date, which are stored in a database system. This allows for monitoring of the food situation within the household.

[0466] Step 2:

[0467] The server uses collected food information to match it with user profile information and generate nutritional guidelines. Input includes profile information such as age, health status, and allergy information, and an AI algorithm is used to generate nutritional guidelines optimized for each household as output. In this process, a baseline menu is determined based on health data.

[0468] Step 3:

[0469] The server generates menus based on nutritional guidelines and food information. This step involves referencing a nutrition database and calculating nutritional values. Specifically, it outputs a menu list that takes into account seasonal ingredients and necessary vitamins and minerals. This ensures that menus are suggested that meet the user's nutritional needs.

[0470] Step 4:

[0471] The server compares the generated menu with current inventory to identify any missing food items. Input data includes a list of required ingredients based on the menu and inventory information. By comparing these, the server outputs a list of insufficient ingredients. This information enables efficient ingredient management.

[0472] Step 5:

[0473] The server automatically orders any missing food items from external suppliers. Specifically, it makes requests through an ordering platform API, enabling the automatic procurement of unmet food needs. Order data is generated and sent to the external service, resulting in the replenishment of the food.

[0474] Step 6:

[0475] The terminal provides cooking instructions to the user based on information received from the server. Input data includes detailed recipes for the menu, and output provides visual or audio instructions. This allows the user to proceed with cooking by following the guide.

[0476] Step 7:

[0477] Users send feedback on their cooking experience to the server via their device. This feedback consists of comments and ratings from users, which are accumulated and used by the AI ​​model to improve future menu generation. This feedback system encourages continuous improvement.

[0478] (Application Example 1)

[0479] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0480] In modern households, preparing efficient and nutritionally balanced meals is difficult, and many families face time constraints and considerable effort. This problem is particularly pronounced in dual-income and busy families. Furthermore, it is difficult to satisfy the individual preferences and nutritional needs of each family, making solutions to these challenges essential.

[0481] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0482] In this invention, the server includes means for acquiring food information from a plurality of sensors installed in a home storage device, means for identifying missing ingredients based on the food information, and means for generating cooking instructions related to the missing ingredients. This makes it possible for even busy households to easily prepare nutritionally balanced meals.

[0483] A "sensor" is a device installed in a home storage system to acquire food information.

[0484] "Food information" refers to data regarding the type, quantity, and expiration date of stored food items.

[0485] "Ingredients that are not yet available" are ingredients that are necessary for cooking based on nutritional balance but are not currently on hand.

[0486] "Cooking instructions" refer to the cooking procedures and guides provided to users based on the generated menu.

[0487] "Feedback" refers to evaluations and opinions provided by users based on their past cooking experiences.

[0488] The "means of automation" refer to a system that orders necessary ingredients through an external purchasing system.

[0489] An "output device" is a device that provides information or instructions to users visually or audibly.

[0490] "Individual user information" refers to data about the user's age, preferences, health status, etc.

[0491] A "nutritional plan" is a nutritionally balanced menu created taking into account the individual information of the user.

[0492] A "cooking support device" is a device used to assist and provide instructions during the cooking process at home.

[0493] This invention's system supports the efficient and nutritionally balanced preparation of meals within the home. Specifically, it uses multiple sensors installed in the home's storage device to acquire food information such as the type, quantity, and expiration date of the food. The sensors are responsible for transmitting the food information to a server in real time.

[0494] The server analyzes current inventory levels based on collected food information and identifies any missing ingredients. This analysis process uses an AI algorithm to generate menus. Furthermore, it creates cooking instructions based on the generated menus and develops detailed cooking guides that reflect nutritional balance to be considered during cooking and the individual preferences of the user.

[0495] If a robot is present as a cooking assistance device, the server sends cooking instructions to the robot and guides the user through the cooking process using voice or a display. A robot equipped with voice recognition capabilities receives feedback from the user, stores this information, and uses it to generate menus for the next time.

[0496] For example, if the server suggests "stir-fried vegetables" and "soup," a sensor detects a shortage of carrots and automatically orders additional carrots using the ordering system. During cooking, the robot provides instructions such as, "Please use one tablespoon of sesame oil when stir-frying." Based on the feedback, the amount of carrots can be adjusted next time.

[0497] Using a generative AI model, the following prompt statements can be used.

[0498] "Based on today's refrigerator inventory, please suggest a dinner menu that meets the following conditions: serves 4 people, can be prepared in 30 minutes or less, and uses ingredients rich in iron."

[0499] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0500] Step 1:

[0501] The server acquires food information from multiple sensors installed in home storage devices. Input includes real-time data from the sensors, including food type, quantity, and expiration date. The server processes this data to generate output that clearly shows the current inventory status.

[0502] Step 2:

[0503] The server uses an AI algorithm to generate nutritionally balanced menus based on acquired food information. Inputs include information on available food items and the user's individual nutritional needs, and the output is a suggested menu. At this stage, an analysis is performed to identify any missing ingredients in the menu.

[0504] Step 3:

[0505] The server operates an automated ordering system based on information about unfulfilled ingredients. The input is a list of missing ingredients. Based on this, it sends order requests to an external purchasing system via a database and API, and arranges for the supply of the necessary ingredients.

[0506] Step 4:

[0507] The server creates detailed cooking instructions based on the generated menu and transmits them to the robot, which is a cooking support device. The input is the menu details, and the output is specific cooking instructions for the user. The robot assists with the cooking process through voice and a display.

[0508] Step 5:

[0509] The user cooks according to voice guidance and display information from the cooking assistance device, and sends feedback received during the process to the server via a terminal. This feedback is sent to the server as input data and used to improve the next menu and cooking instructions.

[0510] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0511] This invention is a system that includes an AI agent that works in conjunction with in-home storage devices to streamline meal preparation for dual-income households. It also incorporates an emotion engine that recognizes the user's emotions and optimizes eating and drinking instructions based on those emotions.

[0512] The server periodically receives food information from status detection devices installed in home refrigerators and pantries. This food information includes the type, quantity, and expiration date of the food. For example, the server might retrieve information indicating that there are six eggs remaining in the refrigerator.

[0513] Based on the collected food information, the server creates menus that take nutritional guidelines into consideration. In this process, algorithms are used that take into account the user's individual health status and preferences. Furthermore, the user's emotional state is also analyzed, and the menu is fine-tuned accordingly. For example, if a user has recently been feeling stressed, the menu will be adjusted to include foods with relaxing effects.

[0514] The server compares current food inventory with the planned menu, identifies any missing ingredients, and automatically places orders. For example, if it determines that there isn't enough lettuce for a salad, the server automatically orders lettuce from a partnered online store.

[0515] The emotion engine analyzes the user's emotions through voice input and facial recognition via the camera. This analysis is sent to a server and used to generate instructions for the next meal. For example, if the user smiles frequently during meals, the instructions will be optimized to maintain a similar menu configuration for the next meal.

[0516] The terminal provides the user with cooking instructions based on the menu received from the server. Specific guidance, such as "Next, slice the tomatoes thinly," is provided via voice through a smart speaker.

[0517] Users can provide feedback via their device after a meal, including emotional responses regarding their food preferences. This feedback is used to improve future menus and analyze emotional responses.

[0518] In this way, this system effectively supports meal preparation at home and enables a personalized dining experience that responds to the user's emotions.

[0519] The following describes the processing flow.

[0520] Step 1:

[0521] The server receives food information from status detection devices installed in refrigerators and pantries within the home. This information includes the type, quantity, and expiration date of the food. For example, the server confirms that there are two packs of yogurt and that the expiration date is in three days.

[0522] Step 2:

[0523] The server generates nutritionally balanced menus based on collected food information. This process also considers the user's health status, allergy information, and past eating history. For example, if a user is prone to vitamin deficiencies, the server might suggest a menu that includes a salad.

[0524] Step 3:

[0525] The emotion engine identifies the user's current emotional state through the device. This uses voice input and facial recognition via the camera to determine the user's stress level and interests. For example, if a user is often smiling, that emotion is registered as "relaxed."

[0526] Step 4:

[0527] The server adjusts the menu based on the analysis results of the emotion engine. If the user is in a relaxed state, it will either maintain the planned dishes in the existing menu or add ingredients that are known to have a relaxing effect. For example, it might suggest adding chamomile tea.

[0528] Step 5:

[0529] The server re-checks inventory levels based on the created menu and identifies any missing ingredients. Based on this, it automatically orders the missing food items via an external online store API. For example, if croutons for a salad are missing, it will order croutons from an online store.

[0530] Step 6:

[0531] The terminal provides the user with menus and cooking instructions received from the server. Through a smartphone app or smart speaker, it provides cooking guidance visually or audibly, such as "We're going to boil the pasta now, I'll set an 8-minute timer."

[0532] Step 7:

[0533] Users submit feedback about their cooked meals via their devices. This feedback includes emotional satisfaction with specific dishes. For example, they might send comments to the server such as, "This soup was too salty."

[0534] Step 8:

[0535] The server uses user feedback and emotion engine data to inform the next menu. This ensures that users continuously receive the most satisfying dining experience.

[0536] (Example 2)

[0537] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0538] In dual-income households, daily meal preparation must be done efficiently within a limited timeframe, requiring careful food management within the home, maintenance of health, and optimization of meals according to individual emotional states. To address this challenge, a system is needed that comprehensively utilizes food information, individual user information, and emotional states to proactively support meal preparation, reduce food waste, and provide nutritionally balanced meals.

[0539] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0540] In this invention, the server includes means for receiving food information from multiple state detection devices installed in a home storage device, means for identifying unfulfilled food items, and means for collecting and analyzing the user's emotions. This makes it possible to suggest meals that take into account the user's health condition and emotions.

[0541] A "condition detection device" is a sensor system installed in household storage equipment that periodically detects information such as the type, quantity, and expiration date of food.

[0542] "Food information" refers to data on the type, quantity, and expiration date of food items acquired by a condition detection device, and is used for managing food ingredients in the home.

[0543] "Unavailable food items" refer to food items that the server determines are not currently present in the household and need to be procured when creating a menu.

[0544] "Dining instructions" refer to instructions generated by AI, taking into account the user's health condition, preferences, emotions, etc., that include specific meal content and necessary cooking procedures.

[0545] A "means of collecting and analyzing emotions" refers to a system that captures a user's voice input and facial expressions and analyzes their emotional state; it is sometimes called an emotion engine.

[0546] "Nutritional guidelines" are standards and guidelines established to suggest a balanced diet based on the user's health condition, preferences, and even their current emotional state.

[0547] This invention is a system that includes an AI agent to streamline meal preparation for dual-income households. A server receives food information from a status detection device installed in the household's storage system and creates a menu based on this information. The server uses IoT devices to collect data such as the type, quantity, and expiration date of food, and then processes it. Specifically, it utilizes a generative AI model to generate a menu that takes nutritional guidelines into consideration based on the collected data. This process takes into account not only the user's health status and preferences, but also the collected emotional data.

[0548] The device collects user emotional data using voice input and a camera, and analyzes it using an emotion engine. The analysis results are sent to a server and used to provide guidance for future meals and drinks. If the user's emotional state indicates stress, the server suggests a menu that includes foods with relaxing effects. The server also checks for missing foods and automatically places orders with partner online stores.

[0549] Cooking instructions are provided to the user via a device and guided by voice through a smart speaker. For example, specific instructions such as "Next, slice the tomatoes thinly" are provided by voice. After the meal, the user provides feedback via the device, which is used to improve the menu for the next meal.

[0550] As a concrete example, there is a server that receives information that "six eggs remain" in the refrigerator. If the user feels "tired recently," the server suggests providing a relaxing meal. The following prompt is entered into the generative AI model: "Describe in detail a system that creates a menu considering nutritional guidelines based on data from a home refrigerator and pantry, and personalizes the dining experience using data obtained from emotion recognition."

[0551] This system can improve users' time efficiency, contribute to optimizing meal content, and reduce the burden of meal preparation for dual-income households by providing nutritionally balanced menus.

[0552] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0553] Step 1:

[0554] The server receives food information from a status detection device installed in the home storage unit. Specifically, the status detection device detects the type, quantity, and expiration date of the food and sends this data to the server. Based on this information, the server builds a list of the food currently in the storage unit.

[0555] Step 2:

[0556] The server uses a generative AI model to create menus based on collected food information. It receives food data and user health and preference information as input, performs data calculations, and outputs optimal menu suggestions. Specifically, the model generates menus while considering nutritional balance.

[0557] Step 3:

[0558] The device collects and analyzes the user's emotions. Specifically, it detects the user's facial expressions using a camera on the device and recognizes their emotional state through voice input. This input data is sent to an emotion engine, which generates a report on the user's emotional state and outputs it to the server.

[0559] Step 4:

[0560] The server optimizes the suggested menu based on the emotional state report. Specifically, it uses the input emotional data to add ingredients with relaxing effects or make adjustments to suit specific emotions. The adjusted menu is output as the following instructions.

[0561] Step 5:

[0562] The server identifies ingredients that are missing from the household based on the suggested menu and automatically places an order with the online store. It receives a comparison of the suggested menu's ingredient list with the household's inventory as input, identifies the missing ingredients, and outputs an order request.

[0563] Step 6:

[0564] The terminal provides the user with the adjusted menu and cooking instructions received from the server via voice. Specifically, the smart speaker assists the user in cooking by giving instructions such as, "Next, slice the tomatoes thinly."

[0565] Step 7:

[0566] Users provide feedback via their device after their meal. This feedback includes emotional responses such as satisfaction with the meal and taste preferences, and this data is sent to the server to be used in creating future menus.

[0567] (Application Example 2)

[0568] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0569] For dual-income households, preparing daily meals is a time-consuming and laborious task. In this environment, efficiently preparing personalized meals based on the user's emotions and health condition is difficult. A system is needed to solve this problem and enable users to prepare meals comfortably.

[0570] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0571] In this invention, the server includes means for receiving item information from multiple situation detection devices installed in a storage device within the home, means for generating food and drink instructions related to unfulfilled items, and means for recognizing the user's emotions and reflecting them in the next meal instructions. This enables personalized meal suggestions and automatic ordering that take into account the user's emotional state.

[0572] "Household storage devices" refer to devices used to store and preserve items within the home, such as refrigerators and pantries.

[0573] A "situation detection device" refers to a device capable of detecting information such as the type, quantity, quality, or expiration date of an item.

[0574] "Item information" refers to data about items stored in the home, acquired by a situation detection device.

[0575] "Items not yet fulfilled" refers to items that need to be purchased additionally to meet pre-set conditions or the user's preferences.

[0576] "Food and beverage instructions" refer to specific meal menus and instructions regarding the preparation of ingredients, generated based on product information.

[0577] "Information device" refers to a device used to display or notify information visually or audibly.

[0578] "User's emotions" refers to the specific psychological state analyzed through voice input and image processing.

[0579] To realize this invention, multiple situation detection devices placed within the home will periodically acquire information about the items in refrigerators and pantries. Temperature sensors and RFID scanners, among others, will be used for this purpose. A server will receive this information from these devices and store it in a database. This database will store detailed information such as the type, quantity, and expiration date of the items.

[0580] The server is equipped with sentiment analysis software to recognize the user's emotions. This software uses the camera and microphone on the user's device (e.g., smartphone or consumer robot) to analyze facial expressions and voice tone. External services such as Google's Cloud Vision API and Microsoft's Emotion API are used for sentiment analysis.

[0581] The server generates food and drink instructions using an AI engine based on item information and emotional data. This AI engine considers the user's health condition and preferences, and proposes the optimal menu using a generated AI model. The server also utilizes machine learning algorithms to improve the accuracy of the menu based on past data and feedback.

[0582] As information devices, smart devices with voice assistants are commonly used. Food and drink instructions are provided to the user through voice or display. For example, instructions such as "Next, slice the tomatoes thinly" are provided audibly through a smart speaker.

[0583] As a concrete example, imagine a system where, upon returning home from work, a robot says, "Welcome back. You seem tired today, so I suggest a relaxing herb salad," and the server automatically orders the appropriate ingredients. Examples of prompts to consider in this case include, "Please use an AI model that suggests tonight's dinner menu, taking into account the user's emotional state," or "Please guide the user through the process of checking the refrigerator inventory and ordering the necessary ingredients."

[0584] In this way, meal preparation at home becomes more efficient, and a personalized dining experience based on the user's emotional state is realized.

[0585] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0586] Step 1:

[0587] The server periodically collects item information from household status detection devices. It receives item data transmitted from these devices as input. The server analyzes this data to identify the type, quantity, and expiration date of items, and registers this information in a database. This ensures that the latest inventory status is always known.

[0588] Step 2:

[0589] The user sends emotional data to the server via their device. An emotion recognition application running on the device analyzes the user's facial expressions and voice based on data acquired from the camera and microphone. Image and audio data related to emotions are taken as input, and the analyzed emotional state (e.g., stress, relaxation) is provided to the server as output.

[0590] Step 3:

[0591] The server generates food and drink instructions using a generative AI model based on item information and user sentiment data. The input is the data accumulated in the previous two steps. The AI ​​engine uses this data to calculate the optimal menu, taking into account health status and preferences. The output is specific menu information provided to the user.

[0592] Step 4:

[0593] The server identifies missing items and places an automated order. It confirms the items required based on the food and beverage instructions and executes the ordering process through the online store's API. The input is a list of required ingredients based on the menu determined in step 3, and the output is the confirmed order information.

[0594] Step 5:

[0595] The user receives cooking instructions via a terminal. Eating instructions sent from the server are displayed on the terminal. Cooking steps are guided as specific instructions using a voice output device such as a smart speaker. The input is eating instructions from the server, and the output is visual or audio instructions received by the user to perform the cooking.

[0596] Step 6:

[0597] After a meal, the user sends feedback to the server via their device. This feedback is then used to inform future dining instructions. The input is user feedback data, and the output is the server updating its stored data for future use.

[0598] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0599] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0600] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0601] [Fourth Embodiment]

[0602] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0603] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0604] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0605] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0606] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0607] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0608] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0609] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0610] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0611] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0612] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0613] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0614] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0615] The system of this invention utilizes an AI agent designed to assist with meal preparation at home. This system functions primarily through interaction between a server, a terminal, and the user. Its specific operation is described below.

[0616] The server periodically collects food information from status detection devices installed in home refrigerators and pantries. This information includes the type, quantity, and expiration date of the food. For example, the server periodically checks the remaining amount and expiration date of milk in the refrigerator.

[0617] Based on the collected food information, the server creates a menu that takes into account the child's age and nutritional needs. This menu also includes an overall nutritional calculation to ensure nutritional balance. For example, a plan might be made to include spinach, which is rich in iron, during certain periods.

[0618] Next, the server compares the created menu with the current food inventory to identify any missing items. Based on this information, the server automatically orders the missing items from an external online store. For example, if the menu includes tomato soup and there is a shortage of tomatoes, the server will order tomatoes.

[0619] Furthermore, the server generates a recipe based on the created menu and sends it to the device. The device then guides the user visually or audibly with this recipe information via a smartphone or smart speaker. For example, a smart speaker might instruct the user to "First, put olive oil in the pot."

[0620] Users can provide feedback through their device during or after cooking. This allows the server to collect this feedback and use it to improve future menus and suggestions. For example, if a user inputs "Please use fewer spices next time," that information will be used in the next menu.

[0621] Thus, the system of this invention helps dual-income families efficiently prepare nutritionally balanced meals, significantly reducing the time and effort spent at home.

[0622] The following describes the processing flow.

[0623] Step 1:

[0624] The server obtains food information from household status detection devices. This includes the type, quantity, and expiration date of food items in the refrigerator and pantry. For example, the server recognizes that there is 200ml of milk remaining.

[0625] Step 2:

[0626] The server analyzes the collected food information and generates a nutritionally balanced menu that takes into account the child's age and allergy information. For example, if iron is needed, sautéed spinach will be added to the menu.

[0627] Step 3:

[0628] The server creates a list of necessary ingredients based on the generated menu and compares it with the current inventory to identify any missing foods. For example, it might find that two tomatoes are missing.

[0629] Step 4:

[0630] The server automatically places orders for identified missing ingredients via an external online store API. For example, it might order 5 tomatoes online.

[0631] Step 5:

[0632] The server generates a recipe corresponding to the menu and sends it to the terminal. This recipe is then organized for later use in the guide.

[0633] Step 6:

[0634] The device provides the user with recipe information received from the server, either visually or audibly. For example, a smart speaker might start a voice guidance saying, "Season the chicken with salt and pepper."

[0635] Step 7:

[0636] Users provide feedback via their device after cooking and eating. This feedback is used to improve future menus. For example, comments such as "It was too spicy" are recorded.

[0637] (Example 1)

[0638] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0639] In modern households, the increasing time and effort required for meal preparation is a challenge. Furthermore, creating nutritionally balanced menus and appropriately supplementing any deficiencies in necessary foods is not easy. Traditional methods make these tasks cumbersome, resulting in difficulty maintaining a healthy diet. Therefore, there is a need for systems that support meal preparation in an efficient and automated way.

[0640] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0641] In this invention, the server includes means for receiving food information from a status detection device installed in a home storage device, means for generating nutritional guidelines considering the user's profile information, means for generating a menu using the nutritional guidelines and food information, and ordering any missing food items through an external supply source, and means for generating cooking procedures based on the menu and guiding the user through an output device. This automates the process of efficient and healthy meal preparation, making it easier for users to manage their diet.

[0642] "Storage equipment" refers to specialized equipment used to preserve food and maintain its quality, and includes refrigerators and pantries.

[0643] A "condition detection device" is a technology used to detect the type, quantity, and expiration date of food within a storage device, and includes RFID sensors and smart cameras.

[0644] "Food information" refers to data obtained from a condition detection device, such as the type of food, remaining quantity, and expiration date.

[0645] "User profile information" refers to information about the user and their family, such as age, health status, and nutritional requirements.

[0646] "Nutritional guidelines" refer to standards that suggest optimal nutritional intake based on the user's profile information.

[0647] A "menu" is a list that shows the combination of dishes and ingredients planned for each meal.

[0648] "Unsold food" refers to ingredients that are necessary to fulfill a menu but are not sufficiently available in the storage device.

[0649] "External sources" refer to online stores or food suppliers that can provide unmet food needs.

[0650] "Cooking instructions" refer to a guide that outlines the instructions and steps for completing a dish based on a specific menu.

[0651] An "output device" is a device used to communicate cooking instructions to the user, and examples include smartphones and smart speakers.

[0652] The embodiment of this invention is based on a storage device, a status detection device, a server, a terminal, and user interaction within the home. The aim of this system is to automate food management in each household and to provide nutritionally balanced meals in a planned manner.

[0653] The server first collects food information from condition detection devices installed in home storage systems. Using hardware such as RFID sensors and smart cameras, it detects the type, quantity, and expiration date of food, and stores this information in a database system (e.g., MySQL). Based on this information, the server utilizes AI algorithms (e.g., scikit-learn), taking into account the user's profile, to generate nutritional guidelines. For example, growing children might be recommended a menu high in protein.

[0654] The server then combines nutritional guidelines and food information to generate meal plans. During this process, it also uses a nutrition database (e.g., the USDA food database) to optimize the overall nutritional value. Based on the generated meal plans, the server identifies any missing foods and places orders with external suppliers (e.g., online stores).

[0655] The device receives cooking instructions generated from the server and provides users with visual or audio guidance. Smartphones and smart speakers are used to guide users through specific steps, such as "add olive oil to the pot." Users can send feedback to the server via the device during and after cooking. This feedback is then used to generate future nutritional guidelines and menus.

[0656] A concrete example of sending a prompt to a generative AI model is, "Please generate a nutritionally balanced weekly meal plan and specific recipes based on the latest food inventory and family nutrition profile." This prompt allows the system to provide suggestions tailored to the family's needs and support efficient meal preparation.

[0657] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0658] Step 1:

[0659] The server collects food information obtained from status detection devices. This information is captured by RFID sensors and smart cameras. Specific inputs include food type, remaining quantity, and expiration date, which are stored in a database system. This allows for monitoring of the food situation within the household.

[0660] Step 2:

[0661] The server uses collected food information to match it with user profile information and generate nutritional guidelines. Input includes profile information such as age, health status, and allergy information, and an AI algorithm is used to generate nutritional guidelines optimized for each household as output. In this process, a baseline menu is determined based on health data.

[0662] Step 3:

[0663] The server generates menus based on nutritional guidelines and food information. This step involves referencing a nutrition database and calculating nutritional values. Specifically, it outputs a menu list that takes into account seasonal ingredients and necessary vitamins and minerals. This ensures that menus are suggested that meet the user's nutritional needs.

[0664] Step 4:

[0665] The server compares the generated menu with current inventory to identify any missing food items. Input data includes a list of required ingredients based on the menu and inventory information. By comparing these, the server outputs a list of insufficient ingredients. This information enables efficient ingredient management.

[0666] Step 5:

[0667] The server automatically orders any missing food items from external suppliers. Specifically, it makes requests through an ordering platform API, enabling the automatic procurement of unmet food needs. Order data is generated and sent to the external service, resulting in the replenishment of the food.

[0668] Step 6:

[0669] The terminal provides cooking instructions to the user based on information received from the server. Input data includes detailed recipes for the menu, and output provides visual or audio instructions. This allows the user to proceed with cooking by following the guide.

[0670] Step 7:

[0671] Users send feedback on their cooking experience to the server via their device. This feedback consists of comments and ratings from users, which are accumulated and used by the AI ​​model to improve future menu generation. This feedback system encourages continuous improvement.

[0672] (Application Example 1)

[0673] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0674] In modern households, preparing efficient and nutritionally balanced meals is difficult, and many families face time constraints and considerable effort. This problem is particularly pronounced in dual-income and busy families. Furthermore, it is difficult to satisfy the individual preferences and nutritional needs of each family, making solutions to these challenges essential.

[0675] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0676] In this invention, the server includes means for acquiring food information from a plurality of sensors installed in a home storage device, means for identifying missing ingredients based on the food information, and means for generating cooking instructions related to the missing ingredients. This makes it possible for even busy households to easily prepare nutritionally balanced meals.

[0677] A "sensor" is a device installed in a home storage system to acquire food information.

[0678] "Food information" refers to data regarding the type, quantity, and expiration date of stored food items.

[0679] "Ingredients that are not yet available" are ingredients that are necessary for cooking based on nutritional balance but are not currently on hand.

[0680] "Cooking instructions" refer to the cooking procedures and guides provided to users based on the generated menu.

[0681] "Feedback" refers to evaluations and opinions provided by users based on their past cooking experiences.

[0682] The "means of automation" refer to a system that orders necessary ingredients through an external purchasing system.

[0683] An "output device" is a device that provides information or instructions to users visually or audibly.

[0684] "Individual user information" refers to data about the user's age, preferences, health status, etc.

[0685] A "nutritional plan" is a nutritionally balanced menu created taking into account the individual information of the user.

[0686] A "cooking support device" is a device used to assist and provide instructions during the cooking process at home.

[0687] This invention's system supports the efficient and nutritionally balanced preparation of meals within the home. Specifically, it uses multiple sensors installed in the home's storage device to acquire food information such as the type, quantity, and expiration date of the food. The sensors are responsible for transmitting the food information to a server in real time.

[0688] The server analyzes current inventory levels based on collected food information and identifies any missing ingredients. This analysis process uses an AI algorithm to generate menus. Furthermore, it creates cooking instructions based on the generated menus and develops detailed cooking guides that reflect nutritional balance to be considered during cooking and the individual preferences of the user.

[0689] If a robot is present as a cooking assistance device, the server sends cooking instructions to the robot and guides the user through the cooking process using voice or a display. A robot equipped with voice recognition capabilities receives feedback from the user, stores this information, and uses it to generate menus for the next time.

[0690] For example, if the server suggests "stir-fried vegetables" and "soup," a sensor detects a shortage of carrots and automatically orders additional carrots using the ordering system. During cooking, the robot provides instructions such as, "Please use one tablespoon of sesame oil when stir-frying." Based on the feedback, the amount of carrots can be adjusted next time.

[0691] Using a generative AI model, the following prompt statements can be used.

[0692] "Based on today's refrigerator inventory, please suggest a dinner menu that meets the following conditions: serves 4 people, can be prepared in 30 minutes or less, and uses ingredients rich in iron."

[0693] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0694] Step 1:

[0695] The server acquires food information from multiple sensors installed in home storage devices. Input includes real-time data from the sensors, including food type, quantity, and expiration date. The server processes this data to generate output that clearly shows the current inventory status.

[0696] Step 2:

[0697] The server uses an AI algorithm to generate nutritionally balanced menus based on acquired food information. Inputs include information on available food items and the user's individual nutritional needs, and the output is a suggested menu. At this stage, an analysis is performed to identify any missing ingredients in the menu.

[0698] Step 3:

[0699] The server operates an automated ordering system based on information about unfulfilled ingredients. The input is a list of missing ingredients. Based on this, it sends order requests to an external purchasing system via a database and API, and arranges for the supply of the necessary ingredients.

[0700] Step 4:

[0701] The server creates detailed cooking instructions based on the generated menu and transmits them to the robot, which is a cooking support device. The input is the menu details, and the output is specific cooking instructions for the user. The robot assists with the cooking process through voice and a display.

[0702] Step 5:

[0703] The user cooks according to voice guidance and display information from the cooking assistance device, and sends feedback received during the process to the server via a terminal. This feedback is sent to the server as input data and used to improve the next menu and cooking instructions.

[0704] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0705] This invention is a system that includes an AI agent that works in conjunction with in-home storage devices to streamline meal preparation for dual-income households. It also incorporates an emotion engine that recognizes the user's emotions and optimizes eating and drinking instructions based on those emotions.

[0706] The server periodically receives food information from status detection devices installed in home refrigerators and pantries. This food information includes the type, quantity, and expiration date of the food. For example, the server might retrieve information indicating that there are six eggs remaining in the refrigerator.

[0707] Based on the collected food information, the server creates menus that take nutritional guidelines into consideration. In this process, algorithms are used that take into account the user's individual health status and preferences. Furthermore, the user's emotional state is also analyzed, and the menu is fine-tuned accordingly. For example, if a user has recently been feeling stressed, the menu will be adjusted to include foods with relaxing effects.

[0708] The server compares current food inventory with the planned menu, identifies any missing ingredients, and automatically places orders. For example, if it determines that there isn't enough lettuce for a salad, the server automatically orders lettuce from a partnered online store.

[0709] The emotion engine analyzes the user's emotions through voice input and facial recognition via the camera. This analysis is sent to a server and used to generate instructions for the next meal. For example, if the user smiles frequently during meals, the instructions will be optimized to maintain a similar menu configuration for the next meal.

[0710] The terminal provides the user with cooking instructions based on the menu received from the server. Specific guidance, such as "Next, slice the tomatoes thinly," is provided via voice through a smart speaker.

[0711] Users can provide feedback via their device after a meal, including emotional responses regarding their food preferences. This feedback is used to improve future menus and analyze emotional responses.

[0712] In this way, this system effectively supports meal preparation at home and enables a personalized dining experience that responds to the user's emotions.

[0713] The following describes the processing flow.

[0714] Step 1:

[0715] The server receives food information from status detection devices installed in refrigerators and pantries within the home. This information includes the type, quantity, and expiration date of the food. For example, the server confirms that there are two packs of yogurt and that the expiration date is in three days.

[0716] Step 2:

[0717] The server generates nutritionally balanced menus based on collected food information. This process also considers the user's health status, allergy information, and past eating history. For example, if a user is prone to vitamin deficiencies, the server might suggest a menu that includes a salad.

[0718] Step 3:

[0719] The emotion engine identifies the user's current emotional state through the device. This uses voice input and facial recognition via the camera to determine the user's stress level and interests. For example, if a user is often smiling, that emotion is registered as "relaxed."

[0720] Step 4:

[0721] The server adjusts the menu based on the analysis results of the emotion engine. If the user is in a relaxed state, it will either maintain the planned dishes in the existing menu or add ingredients that are known to have a relaxing effect. For example, it might suggest adding chamomile tea.

[0722] Step 5:

[0723] The server re-checks inventory levels based on the created menu and identifies any missing ingredients. Based on this, it automatically orders the missing food items via an external online store API. For example, if croutons for a salad are missing, it will order croutons from an online store.

[0724] Step 6:

[0725] The terminal provides the user with menus and cooking instructions received from the server. Through a smartphone app or smart speaker, it provides cooking guidance visually or audibly, such as "We're going to boil the pasta now, I'll set an 8-minute timer."

[0726] Step 7:

[0727] Users submit feedback about their cooked meals via their devices. This feedback includes emotional satisfaction with specific dishes. For example, they might send comments to the server such as, "This soup was too salty."

[0728] Step 8:

[0729] The server uses user feedback and emotion engine data to inform the next menu. This ensures that users continuously receive the most satisfying dining experience.

[0730] (Example 2)

[0731] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0732] In dual-income households, daily meal preparation must be done efficiently within a limited timeframe, requiring careful food management within the home, maintenance of health, and optimization of meals according to individual emotional states. To address this challenge, a system is needed that comprehensively utilizes food information, individual user information, and emotional states to proactively support meal preparation, reduce food waste, and provide nutritionally balanced meals.

[0733] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0734] In this invention, the server includes means for receiving food information from multiple state detection devices installed in a home storage device, means for identifying unfulfilled food items, and means for collecting and analyzing the user's emotions. This makes it possible to suggest meals that take into account the user's health condition and emotions.

[0735] A "condition detection device" is a sensor system installed in household storage equipment that periodically detects information such as the type, quantity, and expiration date of food.

[0736] "Food information" refers to data on the type, quantity, and expiration date of food items acquired by a condition detection device, and is used for managing food ingredients in the home.

[0737] "Unavailable food items" refer to food items that the server determines are not currently present in the household and need to be procured when creating a menu.

[0738] "Dining instructions" refer to instructions generated by AI, taking into account the user's health condition, preferences, emotions, etc., that include specific meal content and necessary cooking procedures.

[0739] A "means of collecting and analyzing emotions" refers to a system that captures a user's voice input and facial expressions and analyzes their emotional state; it is sometimes called an emotion engine.

[0740] "Nutritional guidelines" are standards and guidelines established to suggest a balanced diet based on the user's health condition, preferences, and even their current emotional state.

[0741] This invention is a system that includes an AI agent to streamline meal preparation for dual-income households. A server receives food information from a status detection device installed in the household's storage system and creates a menu based on this information. The server uses IoT devices to collect data such as the type, quantity, and expiration date of food, and then processes it. Specifically, it utilizes a generative AI model to generate a menu that takes nutritional guidelines into consideration based on the collected data. This process takes into account not only the user's health status and preferences, but also the collected emotional data.

[0742] The device collects user emotional data using voice input and a camera, and analyzes it using an emotion engine. The analysis results are sent to a server and used to provide guidance for future meals and drinks. If the user's emotional state indicates stress, the server suggests a menu that includes foods with relaxing effects. The server also checks for missing foods and automatically places orders with partner online stores.

[0743] Cooking instructions are provided to the user via a device and guided by voice through a smart speaker. For example, specific instructions such as "Next, slice the tomatoes thinly" are provided by voice. After the meal, the user provides feedback via the device, which is used to improve the menu for the next meal.

[0744] As a concrete example, there is a server that receives information that "six eggs remain" in the refrigerator. If the user feels "tired recently," the server suggests providing a relaxing meal. The following prompt is entered into the generative AI model: "Describe in detail a system that creates a menu considering nutritional guidelines based on data from a home refrigerator and pantry, and personalizes the dining experience using data obtained from emotion recognition."

[0745] This system can improve users' time efficiency, contribute to optimizing meal content, and reduce the burden of meal preparation for dual-income households by providing nutritionally balanced menus.

[0746] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0747] Step 1:

[0748] The server receives food information from a status detection device installed in the home storage unit. Specifically, the status detection device detects the type, quantity, and expiration date of the food and sends this data to the server. Based on this information, the server builds a list of the food currently in the storage unit.

[0749] Step 2:

[0750] The server uses a generative AI model to create menus based on collected food information. It receives food data and user health and preference information as input, performs data calculations, and outputs optimal menu suggestions. Specifically, the model generates menus while considering nutritional balance.

[0751] Step 3:

[0752] The device collects and analyzes the user's emotions. Specifically, it detects the user's facial expressions using a camera on the device and recognizes their emotional state through voice input. This input data is sent to an emotion engine, which generates a report on the user's emotional state and outputs it to the server.

[0753] Step 4:

[0754] The server optimizes the suggested menu based on the emotional state report. Specifically, it uses the input emotional data to add ingredients with relaxing effects or make adjustments to suit specific emotions. The adjusted menu is output as the following instructions.

[0755] Step 5:

[0756] The server identifies ingredients that are missing from the household based on the suggested menu and automatically places an order with the online store. It receives a comparison of the suggested menu's ingredient list with the household's inventory as input, identifies the missing ingredients, and outputs an order request.

[0757] Step 6:

[0758] The terminal provides the user with the adjusted menu and cooking instructions received from the server via voice. Specifically, the smart speaker assists the user in cooking by giving instructions such as, "Next, slice the tomatoes thinly."

[0759] Step 7:

[0760] Users provide feedback via their device after their meal. This feedback includes emotional responses such as satisfaction with the meal and taste preferences, and this data is sent to the server to be used in creating future menus.

[0761] (Application Example 2)

[0762] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0763] For dual-income households, preparing daily meals is a time-consuming and laborious task. In this environment, efficiently preparing personalized meals based on the user's emotions and health condition is difficult. A system is needed to solve this problem and enable users to prepare meals comfortably.

[0764] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0765] In this invention, the server includes means for receiving item information from multiple situation detection devices installed in a storage device within the home, means for generating food and drink instructions related to unfulfilled items, and means for recognizing the user's emotions and reflecting them in the next meal instructions. This enables personalized meal suggestions and automatic ordering that take into account the user's emotional state.

[0766] "Household storage devices" refer to devices used to store and preserve items within the home, such as refrigerators and pantries.

[0767] A "situation detection device" refers to a device capable of detecting information such as the type, quantity, quality, or expiration date of an item.

[0768] "Item information" refers to data about items stored in the home, acquired by a situation detection device.

[0769] "Items not yet fulfilled" refers to items that need to be purchased additionally to meet pre-set conditions or the user's preferences.

[0770] "Food and beverage instructions" refer to specific meal menus and instructions regarding the preparation of ingredients, generated based on product information.

[0771] "Information device" refers to a device used to display or notify information visually or audibly.

[0772] "User's emotions" refers to the specific psychological state analyzed through voice input and image processing.

[0773] To realize this invention, multiple situation detection devices placed within the home will periodically acquire information about the items in refrigerators and pantries. Temperature sensors and RFID scanners, among others, will be used for this purpose. A server will receive this information from these devices and store it in a database. This database will store detailed information such as the type, quantity, and expiration date of the items.

[0774] The server is equipped with sentiment analysis software to recognize the user's emotions. This software uses the camera and microphone on the user's device (e.g., smartphone or consumer robot) to analyze facial expressions and voice tone. External services such as Google's Cloud Vision API and Microsoft's Emotion API are used for sentiment analysis.

[0775] The server generates food and drink instructions using an AI engine based on item information and emotional data. This AI engine considers the user's health condition and preferences, and proposes the optimal menu using a generated AI model. The server also utilizes machine learning algorithms to improve the accuracy of the menu based on past data and feedback.

[0776] As information devices, smart devices with voice assistants are commonly used. Food and drink instructions are provided to the user through voice or display. For example, instructions such as "Next, slice the tomatoes thinly" are provided audibly through a smart speaker.

[0777] As a concrete example, imagine a system where, upon returning home from work, a robot says, "Welcome back. You seem tired today, so I suggest a relaxing herb salad," and the server automatically orders the appropriate ingredients. Examples of prompts to consider in this case include, "Please use an AI model that suggests tonight's dinner menu, taking into account the user's emotional state," or "Please guide the user through the process of checking the refrigerator inventory and ordering the necessary ingredients."

[0778] In this way, meal preparation at home becomes more efficient, and a personalized dining experience based on the user's emotional state is realized.

[0779] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0780] Step 1:

[0781] The server periodically collects item information from household status detection devices. It receives item data transmitted from these devices as input. The server analyzes this data to identify the type, quantity, and expiration date of items, and registers this information in a database. This ensures that the latest inventory status is always known.

[0782] Step 2:

[0783] The user sends emotional data to the server via their device. An emotion recognition application running on the device analyzes the user's facial expressions and voice based on data acquired from the camera and microphone. Image and audio data related to emotions are taken as input, and the analyzed emotional state (e.g., stress, relaxation) is provided to the server as output.

[0784] Step 3:

[0785] The server generates food and drink instructions using a generative AI model based on item information and user sentiment data. The input is the data accumulated in the previous two steps. The AI ​​engine uses this data to calculate the optimal menu, taking into account health status and preferences. The output is specific menu information provided to the user.

[0786] Step 4:

[0787] The server identifies missing items and places an automated order. It confirms the items required based on the food and beverage instructions and executes the ordering process through the online store's API. The input is a list of required ingredients based on the menu determined in step 3, and the output is the confirmed order information.

[0788] Step 5:

[0789] The user receives cooking instructions via a terminal. Eating instructions sent from the server are displayed on the terminal. Cooking steps are guided as specific instructions using a voice output device such as a smart speaker. The input is eating instructions from the server, and the output is visual or audio instructions received by the user to perform the cooking.

[0790] Step 6:

[0791] After a meal, the user sends feedback to the server via their device. This feedback is then used to inform future dining instructions. The input is user feedback data, and the output is the server updating its stored data for future use.

[0792] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0793] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0794] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0795] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0796] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0797] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0798] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0799] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0800] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0801] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0802] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0803] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0804] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0805] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0806] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0807] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0808] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0809] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0810] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0811] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0812] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0813] The following is further disclosed regarding the embodiments described above.

[0814] (Claim 1)

[0815] A means for receiving food information from multiple status detection devices installed in a household storage device,

[0816] A means for identifying unfulfilled food items based on the aforementioned food information,

[0817] means for generating a food and drink instruction related to the aforementioned unfulfilled food,

[0818] A means for placing an order to supply food based on the aforementioned food and beverage instructions,

[0819] The means of distributing the aforementioned food and drink instructions to an output device and guiding the user visually or audibly,

[0820] A system that includes this.

[0821] (Claim 2)

[0822] The system according to claim 1, further comprising means for generating nutritional guidelines that take into account the individual information of the user.

[0823] (Claim 3)

[0824] The system according to claim 1, further comprising means for collecting feedback from users and reflecting it in future dining instructions.

[0825] "Example 1"

[0826] (Claim 1)

[0827] A means for receiving food information from multiple status detection devices installed in a household storage device,

[0828] A means for generating nutritional guidelines based on the aforementioned food information and considering the user's profile information,

[0829] A means for generating a menu using the aforementioned nutritional guidelines and food information, and for identifying unmet food requirements based on the menu,

[0830] A means of ordering the aforementioned unfulfilled food through an external supply source,

[0831] A means for generating cooking procedures based on the aforementioned menu and guiding the user visually or audibly through an output device,

[0832] A system that includes this.

[0833] (Claim 2)

[0834] The system according to claim 1, further comprising means for collecting user feedback and reflecting it in the generation of future nutritional guidelines and menus.

[0835] (Claim 3)

[0836] The system according to claim 1, further comprising means for sending prompt messages to a generating AI model using collected information to obtain improved menus and nutritional guidelines.

[0837] "Application Example 1"

[0838] (Claim 1)

[0839] A means of acquiring food information from multiple sensors installed in a household storage device,

[0840] A means for identifying missing ingredients based on the aforementioned food information,

[0841] means for generating cooking instructions related to the missing ingredients,

[0842] A means for automating ordering to supply ingredients in accordance with the aforementioned cooking instructions,

[0843] Means for providing the aforementioned cooking instructions to the user visually or audibly via an output device,

[0844] A means of accumulating user feedback and reflecting it in future cooking instructions,

[0845] A system that includes this.

[0846] (Claim 2)

[0847] The system according to claim 1, further comprising means for formulating a nutrition plan using individual user information.

[0848] (Claim 3)

[0849] The system according to claim 1, further comprising a robot as a cooking support device and means for providing instructions to smoothly carry out the cooking process.

[0850] "Example 2 of combining an emotion engine"

[0851] (Claim 1)

[0852] A means for receiving food information from multiple status detection devices installed in a household storage device,

[0853] A means for identifying unfulfilled food items based on the aforementioned food information,

[0854] means for generating a food and drink instruction related to the aforementioned unfulfilled food,

[0855] A means for placing an order to supply food based on the aforementioned food and beverage instructions,

[0856] The means of distributing the aforementioned food and drink instructions to an output device and guiding the user visually or audibly,

[0857] A means of collecting and analyzing user emotions,

[0858] A means for optimizing food and drink instructions based on the analyzed emotions,

[0859] A system that includes this.

[0860] (Claim 2)

[0861] The system according to claim 1, further comprising means for generating nutritional guidelines that take into account the individual information of the user.

[0862] (Claim 3)

[0863] The system according to claim 1, further comprising means for collecting feedback from users and reflecting it in future dining instructions.

[0864] "Application example 2 when combining with an emotional engine"

[0865] (Claim 1)

[0866] A means for receiving item information from multiple status detection devices installed in a storage device within the home,

[0867] A means for identifying unfulfilled items based on the aforementioned item information,

[0868] Means for generating food and drink instructions related to the aforementioned unfulfilled items,

[0869] Means for registering the supply of goods based on the aforementioned food and beverage instructions,

[0870] A means for distributing the aforementioned food and drink instructions to an information device and notifying the user visually or audibly,

[0871] A means of recognizing the user's emotions and reflecting them in the next meal instructions,

[0872] A system that includes this.

[0873] (Claim 2)

[0874] The system according to claim 1, further comprising means for generating nutritional guidelines that take into account the emotional state of the user.

[0875] (Claim 3)

[0876] The system according to claim 1, further comprising means for providing information necessary to assist cooking tasks within the home. [Explanation of Symbols]

[0877] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for receiving food information from multiple status detection devices installed in a household storage device, A means for identifying unfulfilled food items based on the aforementioned food information, means for generating a food and drink instruction related to the aforementioned unfulfilled food, A means for placing an order to supply food based on the aforementioned food and beverage instructions, The means of distributing the aforementioned food and drink instructions to an output device and guiding the user visually or audibly, A system that includes this.

2. The system according to claim 1, further comprising means for generating nutritional guidelines that take into account the individual information of the user.

3. The system according to claim 1, further comprising means for collecting feedback from users and reflecting it in future dining instructions.

Images