System

Abstract

A system is provided.SOLUTION: A system comprising: means for inputting household food information; means for obtaining inventory information of a local retail store; means for suggesting an optimal menu based on the household food information and the inventory information of the retail store; and means for ordering food to be used in the suggested menu from the retail store.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In modern households, planning daily menus amid busy daily lives is a major burden. Furthermore, food waste has become a social issue, creating a need for more efficient food management within the home. The purpose of this invention is to solve these problems and provide a system for managing ingredients within the home and proposing efficient menus. [Means for solving the problem]

[0005] The present invention is a system that includes a means for inputting household food ingredient information, a means for acquiring inventory information from local retailers, a means for proposing an optimal menu based on the household food ingredient information and the retailer's inventory information, and a means for ordering ingredients for the proposed menu from the retailer. Furthermore, by providing a means for managing the inventory status and expiration date of household stockpiles and a means for notifying the user of ingredients approaching their expiration date, food waste can be reduced. Furthermore, by providing a means for inputting the user's household composition, allergy information, and food preferences, the system can propose an optimal menu based on this information, thereby enabling menu proposals tailored to each household. Furthermore, by periodically acquiring inventory information from supermarkets and greengrocers and identifying and proposing ingredients that are prioritized for sale, the system also contributes to reducing food waste. It is also possible to input household food ingredient information using a mobile device such as a smartphone or tablet, view inventory information from local retailers and proposed menus, and order ingredients.

[0006] "Household food ingredient information" is information about the types, quantities, expiration dates, etc. of food ingredients present in the household.

[0007] "Local retailers" are stores such as supermarkets and greengrocers that sell food and daily necessities in a specific area.

[0008] "Inventory information" refers to information about the type, quantity, and status of products (such as food ingredients) currently held by a retail store.

[0009] A "menu" is a plan of meal contents for a certain period of time, or a plan for such a plan.

[0010] A "means for suggesting" is a method or device that has the function of selecting and presenting the optimal menu and ingredients based on specific conditions.

[0011] An "ordering means" is a method or device that allows a user to submit a request to purchase a particular product or service.

[0012] "Stockpiles" are food and daily necessities that are purchased and stored in the home with the understanding that they will be stored for a long period of time.

[0013] A "use-by date" is the date by which food or daily necessities should be consumed, or the time when that date expires.

[0014] A "notification means" is a method or device that has the function of informing a user of specific information or warnings.

[0015] "Household composition" is information about the number of people living in a household and their relationships (e.g., adults, children, pets, etc.).

[0016] "Allergy Information" means information about food allergies that a user or their family members have.

[0017] "Food preferences" refers to information about the preferences or dislikes of a particular food ingredient or dish that a user or their family members have.

[0018] "Means for periodic acquisition" refers to a method or device that has the function of collecting the latest information at regular time intervals.

[0019] "Food ingredients that retailers want to sell preferentially" are food ingredients that retailers want to sell preferentially for specific reasons (e.g., excess inventory, preserving freshness).

[0020] "Mobile devices such as smartphones and tablets" refer to portable electronic devices that can input, display, and communicate information through applications. [Brief explanation of the drawings]

[0021] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3]FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0022] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0023] First, the terms used in the following description will be explained.

[0024] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0025] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0026] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0027] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0028] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0029] [First embodiment]

[0030] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0031] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0032] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0033] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0034] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0035] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0036] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0037] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0038] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0039] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0040] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0041] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0042] This invention is a system that automates food ingredient management and menu suggestions for the home, and also enables efficient food purchasing by linking with local retailers. This system integrates information on home ingredients and inventory information from local retailers, suggests optimal menus for users, and supports them in ordering ingredients.

[0043] System configuration

[0044] The system consists of a user terminal, a server, a database of local retailers, and a notification means.

[0045] System Operation

[0046] 1. Setting up your user profile

[0047] Users open the application on their smartphones, tablets, or other devices and enter their family members, allergy information, and food preferences. This data is then sent from the device to a server and stored in a database.

[0048] 2. Enter and update existing ingredients

[0049] Users input the ingredients (item, quantity, expiration date) in their refrigerator or pantry into the app, and this information is sent from the device to the server and stored in a database.

[0050] 3. Collaboration with local retailers

[0051] The server periodically retrieves inventory information from local retailers, often automatically through an API, and stores and updates the database.

[0052] 4. Menu suggestions

[0053] Users request menu suggestions through the application, and the server uses AI algorithms to generate optimal menus based on the user profile, existing ingredient information, and local retailer inventory information. These menu suggestions are then sent to the device and displayed to the user.

[0054] 5. Order ingredients online

[0055] Based on the proposed menu, users can order the necessary ingredients online through the application. The order information is sent from the device to a server, which then forwards it to the local retailer's system. The retailer receives the order and prepares the ingredients.

[0056] 6. Stockpile management and expiration date notification

[0057] The server periodically scans the database to identify ingredients that are nearing their expiration date, and users are notified when the expiration date approaches, which is displayed on their device, allowing them to consume the ingredients at the appropriate time.

[0058] Specific examples

[0059] A family has tomatoes, chicken, and pasta in the refrigerator. The user opens the app and enters these ingredients. The family also has a fish allergy. Meanwhile, the local supermarket has a large stock of fresh basil.

[0060] When a user requests "Suggest a dinner menu," the server will suggest a menu based on the following information:

[0061] Homemade tomatoes, chicken, and pasta

[0062] Having a fish allergy

[0063] Fresh basil from the supermarket

[0064] Based on this, the server will suggest "Tomato and Chicken Basil Pasta" and encourage the purchase of basil. Also, if there is milk nearing its expiration date, the user will receive a notification on their device saying, "The milk's expiration date is coming up in three days."

[0065] In this way, this system provides food management within the home and efficient menu suggestions, contributing to reducing food waste and easing the burden of housework.

[0066] The processing flow will be explained below.

[0067] Program processing flow

[0068] 1. Setting up your user profile

[0069] Step 1: Enter your profile information

[0070] Users open the application on their smartphone or tablet and enter information about their household composition, allergies, and food preferences.

[0071] Step 2: Submit your input

[0072] The terminal transmits the input profile information to the server.

[0073] Step 3: Save your information

[0074] The server stores the received profile information in a database.

[0075] 2. Enter and update existing ingredients

[0076] Step 1: Enter ingredient information

[0077] Users enter the ingredients (item, quantity, expiration date) in their refrigerator or pantry into the app.

[0078] Step 2: Submit your input

[0079] The terminal transmits the input ingredient information to the server.

[0080] Step 3: Save the ingredients

[0081] The server stores the received ingredient information in a database.

[0082] 3. Collaboration with local retailers

[0083] Step 1: Get inventory information

[0084] The server periodically retrieves inventory information from local retailers via API or data feed.

[0085] Step 2: Save inventory information

[0086] The server stores the acquired inventory information in a database.

[0087] Step 3: View inventory information

[0088] The terminal displays real-time inventory information for users from local retailers.

[0089] 4. Menu suggestions

[0090] Step 1: Request a menu suggestion

[0091] The user requests meal suggestions through the application.

[0092] Step 2: Get the data

[0093] The server retrieves user profiles, existing ingredient information, and local retailer inventory information from a database.

[0094] Step 3: Generate a menu

[0095] The server uses AI algorithms to generate optimal menus based on the data it obtains.

[0096] Step 4: Submit and view your menu

[0097] The server sends the generated menu to the terminal.

[0098] The device displays the sent menu to the user.

[0099] 5. Order ingredients online

[0100] Step 1: Create an order

[0101] Users can order the necessary ingredients online based on the suggested menu.

[0102] Step 2: Submit your order

[0103] The terminal transmits the order details to the server.

[0104] Step 3: Transfer your order

[0105] The server forwards the order to the local retailer's system.

[0106] Step 4: Processing the order

[0107] Local retailers receive orders and prepare the ingredients.

[0108] 6. Stockpile management and expiration date notification

[0109] Step 1: Check the expiration date

[0110] The server periodically scans the database to identify ingredients that are nearing their expiration date.

[0111] Step 2: Generate notification content

[0112] The server creates a notification about food items that are nearing their expiration date.

[0113] Step 3: Send notifications

[0114] The server sends the generated notification content to the terminal.

[0115] Step 4: View notifications

[0116] The device will display expiration date notifications to the user.

[0117] Example 1

[0118] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0119] In modern households, managing and purchasing ingredients, as well as creating menus, requires a lot of time and effort, which is a major challenge. People often overlook expiration dates on ingredients, resulting in food waste. Furthermore, there is a lack of systems that efficiently link household ingredients with inventory information from local retailers, making it difficult to efficiently purchase the ingredients needed. This increases the burden of daily housework and ultimately leads to a decline in quality of life.

[0120] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0121] In this invention, the server includes a means for inputting household food ingredient information, a means for periodically obtaining inventory information from local retailers via an API, a means for proposing optimal menus using a generative AI model based on the household food ingredient information and the local retailer's inventory information, and a means for online ordering of ingredients used in the proposed menu from the retailer. This automates household food ingredient management and menu proposals, enabling efficient food purchasing. Furthermore, by managing and notifying expiration dates, food waste can be reduced, reducing the burden of housework and improving quality of life.

[0122] "Household food ingredient information" is detailed information such as the type, quantity, and expiration date of food ingredients used in the home.

[0123] "Local retailers" refer to stores that sell food and daily necessities in a specific area.

[0124] "Inventory information" refers to information about the types, quantities, prices, and availability of products available at local retail stores.

[0125] "API" stands for Application Program Interface, a means by which different software applications communicate with each other.

[0126] A "generative AI model" is an algorithm or system that uses artificial intelligence techniques to analyze data and generate new information or suggestions.

[0127] A "menu" refers to the combination of foods and dishes for a particular meal, usually suggested for breakfast, lunch, and dinner.

[0128] "Online ordering" refers to the act of purchasing goods over the Internet, particularly in an e-commerce environment.

[0129] "Best before date" indicates the last date that a food product can be safely consumed and still retain its quality.

[0130] A "notification" is a message or signal intended to inform a user of certain information or events.

[0131] This invention is a system that allows efficient management of home ingredients, menu planning, and online ordering of ingredients. To implement this system, a user terminal, a server, a database of local retailers, and a notification means are required.

[0132] First, users open the application on their smartphone, tablet, or other device and enter information about their family, allergies, and food preferences. This information is then sent from the device to a server and stored in a database.

[0133] Next, users enter the items, quantities, and expiration dates of ingredients in their refrigerator or pantry into the app, which then sends this information from the device to a server and stores it in a database.

[0134] The server periodically retrieves inventory information from local retailers via API and stores it in a database. This process is automated, ensuring that inventory information is always up-to-date.

[0135] When a user requests a menu suggestion through the app, the server uses a generative AI model to generate an optimal menu based on the user profile, existing ingredient information, and local retailer inventory information. This suggestion is then sent to the device and displayed to the user.

[0136] The user orders the ingredients needed based on the proposed menu online. The order information is sent from the terminal to a server, which then forwards it to the local retailer's system. The retailer receives the order and prepares the ingredients.

[0137] The server periodically scans the database to identify ingredients that are nearing their expiration date, and users are notified when the expiration date approaches, which is displayed on their device, allowing them to consume the ingredients at the appropriate time.

[0138] As a concrete example, suppose a household has tomatoes, chicken, and pasta in the refrigerator, and the family consists of two adults, one child, and a person with a fish allergy. If a user from this household, who has fresh basil in stock at a local retailer, requests "Suggest a dinner menu," the server generates a menu for "Tomato and Chicken Basil Pasta" based on the following information and encourages the user to purchase basil. If there is milk nearing its expiration date, a notification stating "The milk's expiration date is in three days" will be displayed on the user's device.

[0139] An example prompt might look like this:

[0140] "If a user opens a food management app, inputs the tomatoes, chicken, and pasta in their refrigerator, and specifies that a family member has a fish allergy, and requests the app to suggest a dinner menu, the server will suggest tomato and chicken basil pasta and encourage them to purchase fresh basil. Also, if there is milk nearing its expiration date, a notification will appear on the user's device stating, 'Your milk will expire in three days.'"

[0141] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0142] Step 1:

[0143] The user opens the application on a device such as a smartphone or tablet. First, the user profile setting screen is displayed. The user enters the family composition (e.g., two adults, one child), allergy information (e.g., fish allergy), and food preferences (e.g., prefers Italian food). The device converts this input information into JSON format and sends it to the server. The server receives this JSON data and stores it in the user profile table in the database. In this step, the input data is private information such as family composition and allergy information, and the output data means that the user profile is saved in the database.

[0144] Step 2:

[0145] The user enters the ingredients (e.g., tomatoes, chicken, pasta), quantity (e.g., 3 tomatoes, 2 chicken pieces), and expiration date (e.g., 2023-10-10) of ingredients in the household on the app's ingredient management screen. The device acquires this information, converts it back into JSON format, and sends it to the server. The server receives this JSON data, saves it in the ingredient table of the database, and compares it with existing information to update it. The input data for this step are detailed information about each ingredient, and the output data is the updated information for the ingredient table.

[0146] Step 3:

[0147] The server runs a scheduled job every day at 3 AM, which sends a request to the local retailer's API endpoint to retrieve inventory information. The retrieved inventory information is returned to the server in JSON format. The server receives this information and updates an inventory table in the database. The input data for this step is the retailer's inventory information, and the output data is the updated inventory table contents.

[0148] Step 4:

[0149] The user taps the "Suggest a Meal" button in the application. The device sends this request to the server. Based on the received request, the server queries the user profile, existing ingredient information, and inventory information from local retailers, and uses an AI algorithm to generate an optimal meal plan. The generated meal plan is sent to the device in JSON format. The device parses this data and displays the suggested meal plan to the user. The input data for this step are the user request and related ingredient information, and the output data is the suggested meal plan.

[0150] Step 5:

[0151] Based on the proposed menu, the user uses the application to order the ingredients they need online. For example, they tap the "Buy basil" button. The device sends this purchase request to the server. The server then forwards the request to the local retailer's order system. The local retailer confirms the order and prepares the ingredients. The input data for this step is the user's purchase request, and the output data indicates that the order has been accepted and preparation has begun.

[0152] Step 6:

[0153] The server runs a scheduled job every day at 2 PM that scans the ingredient table in the database to identify ingredients that are close to their expiration date. The server generates a notification for ingredients with an expiration date within three days and sends it to the user's device. The device displays this notification to the user as a push notification. The input data for this step is the expiration date information in the ingredient table, and the output data is the notification information displayed to the user.

[0154] Through these steps, we can create a system that efficiently manages household ingredients, proposes menus, and orders ingredients online. This system will be very useful in reducing food waste and easing the burden of housework.

[0155] (Application example 1)

[0156] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0157] Modern households often spend a great deal of time and effort managing ingredients and planning daily meals, creating a need for efficient ingredient use and reduced food waste. Finding optimal ways to purchase ingredients and quickly supply them is also a challenge. Effective menu suggestions tailored to each household's needs and management of expiration dates are also desirable. To solve these problems, a system is needed that integrates information about ingredients within the home, inventory information from local retailers, and food delivery services, and provides unified management, suggestions, ordering, and delivery.

[0158] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0159] In this invention, the server includes a means for inputting household ingredient information, a means for acquiring inventory information from local retailers, a means for proposing an optimal menu based on the household ingredient information and the retailer's inventory information, a means for ordering ingredients for the proposed menu from the retailer, and a means for coordinating with a food delivery service to deliver the ordered ingredients to the user. This allows users to efficiently manage ingredients in their home, propose optimal menus, and quickly purchase and deliver the necessary ingredients. It also contributes to reducing food waste and easing the burden of housework.

[0160] "Household food information" refers to detailed data such as the items, quantities, and expiration dates of food and ingredients that are normally stored in the home.

[0161] "Local retail store inventory information" refers to detailed data such as the availability, prices, and expiration dates of food ingredients and products held by local retail stores.

[0162] "Menu suggestions" refers to the act of presenting the most suitable dishes and meal menus based on the user's available ingredients and individual needs (allergy information, food preferences, etc.).

[0163] "Ordering ingredients" refers to the process by which a user specifies the ingredients needed based on a suggested menu and purchases these ingredients from a retailer.

[0164] "Food delivery service" refers to a business that includes delivering ingredients ordered by users to designated locations.

[0165] "Food inventory status" refers to the current amount and expiration date of a particular food ingredient at a home or storage facility.

[0166] "Expiration date notification" refers to a function that notifies users when the expiration date of stored food ingredients is approaching.

[0167] A "generative AI model" refers to an artificial intelligence algorithm that automatically generates optimal prompts and suggestions based on data and information generated from user input.

[0168] A "prompt" is a guide or instruction generated to encourage a user to take a particular action or response.

[0169] This invention is a system that automates household food ingredient management and meal planning, and also enables efficient food purchasing and delivery through food delivery services. This system integrates information on household ingredients, inventory information from local retailers, and information from food delivery services, suggests optimal meals for users, and supports the ordering and delivery of ingredients.

[0170] System configuration

[0171] The system consists of a user terminal, a server, a database of local retailers, a food delivery service, and a notification method.

[0172] Hardware and Software Configuration

[0173] Hardware

[0174] Smartphone (iOS, Android)

[0175] Cloud services (AWS, Google Cloud, Microsoft Azure)

[0176] software

[0177] Food management application: Python (Flask)

[0178] Database: MongoDB

[0179] Integration with local store API: RESTful API

[0180] Processing Description

[0181] User profile settings

[0182] Users open the application on their smartphone and enter their family information, allergy information, and food preferences. This data is sent to the server in JSON format and stored in a database.

[0183] Entering and updating existing ingredients

[0184] Users input the ingredients (item, quantity, expiration date) in their refrigerator or pantry into the app, and this information is sent from the device to the server and stored in a database.

[0185] Collaboration with local retailers

[0186] The server periodically retrieves inventory information from local retailers via API, which is then stored and updated in a database.

[0187] Menu suggestions

[0188] When a user requests a menu suggestion, the server uses an AI algorithm to suggest the optimal menu based on the user profile, existing ingredient information, and retail store inventory information, and sends it to the device.

[0189] Online food ordering and delivery

[0190] Based on the proposed menu, users can order the necessary ingredients online through the application. The order information is transferred from the server to the local retailer's system, which receives the order, prepares the ingredients, and then works with a food delivery service to deliver the ingredients to the user.

[0191] Expiration date notification

[0192] The server periodically scans the database to identify ingredients that are nearing their expiration date, and when this happens, a notification is sent to the user and displayed on their device.

[0193] Specific examples

[0194] A family has tomatoes, chicken, and pasta in the refrigerator. The user opens the app and enters these ingredients. The family also includes a fish allergy. Meanwhile, the local supermarket has a large stock of fresh basil.

[0195] When a user requests "Suggest a dinner menu," the server will suggest a menu based on the following information:

[0196] Homemade tomatoes, chicken, and pasta

[0197] Having a fish allergy

[0198] Fresh basil from the supermarket

[0199] Based on this, the server will suggest "Tomato and Chicken Basil Pasta" and encourage the purchase of basil. Also, if there is milk nearing its expiration date, the user will receive a notification on their device saying, "The milk's expiration date is coming up in three days."

[0200] An example of a prompt sentence in this case is:

[0201] "Dear user, based on the ingredients in your fridge, I'd like to suggest 'Tomato and Chicken Basil Pasta'. You need basil, but you can buy it at your local supermarket. Would you like to order basil?"

[0202] is displayed.

[0203] In this way, this system can provide food management within the home and efficient menu suggestions, contributing to reducing food waste and easing the burden of housework.

[0204] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0205] Step 1:

[0206] User profile settings

[0207] Users open the smartphone application and input their family composition, allergy information, and food preferences. This data is sent to the server in JSON format upon input, and the server stores the received data in a database. The input data includes the number of family members, types of allergies, and preferred food ingredients. The output is accurately stored in the database.

[0208] Step 2:

[0209] Entering and updating existing ingredients

[0210] The user inputs the item, quantity, and expiration date of each ingredient in the refrigerator or pantry into the application. This data is also sent to the server in JSON format, and after the server receives it, it is stored in the database. The input data includes the ingredient name, quantity, and expiration date. The current ingredient information is reflected in the database as output.

[0211] Step 3:

[0212] Collaboration with local retailers

[0213] The server periodically retrieves inventory information from local retailers via a specified API. The retrieved data includes detailed information such as stock items, prices, and expiration dates, and is also received in JSON format and stored in a database. An API key is required to retrieve inventory information, and the latest information can be retrieved automatically by setting up requests to be sent at regular intervals. The latest inventory information is updated in the database as an output.

[0214] Step 4:

[0215] Menu suggestions

[0216] When a user requests a menu suggestion through the application, the request is sent to the server. The server uses a generative AI model to generate an optimal menu based on the user profile in the database, existing ingredient information, and local retailer inventory information. Input data includes the user's preferences and allergies, currently owned ingredients, and local retailer inventory information. The suggested menu is sent as output to the user's device.

[0217] Step 5:

[0218] Online food ordering and delivery

[0219] The user selects the ingredients they need based on the suggested menu and places an order online through the application. The order information is sent in JSON format to the server, which then forwards it to the local retailer's system. The server then notifies the retailer and coordinates with the food delivery service to deliver the ordered ingredients to the destination. Input data includes the name of the ingredients to be ordered, the quantity, and delivery address information. The output is a record of the order history in a database, and the ingredients are delivered to the user.

[0220] Step 6:

[0221] Expiration date notification

[0222] The server periodically scans the database to identify ingredients that are nearing their expiration date. When the expiration date approaches, the information is notified to the user's device. The input data is the current ingredient information and its expiration date, and a database query is used to extract ingredients that are approaching their expiration date. The output is a notification sent to the user's device, displaying the message "The expiration date for XX is coming up in 3 days."

[0223] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0224] This system automates home ingredient management and menu suggestions, and also works with local retailers to efficiently purchase ingredients. It also recognizes the user's emotions and adjusts the content of suggestions and notifications based on those emotions, providing a more personalized service.

[0225] System configuration

[0226] The system consists of a user's terminal, a server, a database of local retailers, a notification method, and an emotion engine.

[0227] System Operation

[0228] 1. Setting up your user profile

[0229] Users open the application on their smartphone or tablet and enter information about their family, allergies, and food preferences. This data is then sent from the device to a server and stored in a database.

[0230] 2. Enter and update existing ingredients

[0231] Users input the ingredients (item, quantity, expiration date) in their refrigerator or pantry into the app, and this information is sent from the device to the server and stored in a database.

[0232] 3. Collaboration with local retailers

[0233] The server periodically retrieves inventory information from local retailers via APIs and data feeds, which is then stored and updated in a database.

[0234] 4. Menu suggestions

[0235] Users request menu suggestions through the application, and the server uses AI algorithms to generate optimal menus based on the user profile, existing ingredient information, and local retailer inventory information. The generated menus are then sent to the device and displayed to the user.

[0236] 5. Order ingredients online

[0237] Based on the proposed menu, users can order the necessary ingredients online through the application. The order information is sent from the device to a server, which then forwards it to the local retailer's system. The retailer receives the order and prepares the ingredients.

[0238] 6. Stockpile management and expiration date notification

[0239] The server periodically scans the database to identify ingredients that are nearing their expiration date, and users are notified when the expiration date approaches, which is displayed on their device, allowing them to consume the ingredients at the appropriate time.

[0240] Introducing the Emotion Engine

[0241] 7. Emotion recognition

[0242] Through applications, the device collects user input and emotions using sensors such as the camera and microphone, including facial expression recognition, voice recognition, and analysis of input patterns.

[0243] 8. Transmission and Processing of Emotional Data

[0244] The device sends the collected emotional data to a server, which uses an emotion engine to analyze this data and determine the user's current emotional state.

[0245] 9. Emotion-Based Regulation

[0246] The server adjusts menu suggestions and notification content based on the emotional data recognized by the emotion engine. For example, if the user is feeling stressed, it will suggest easy-to-prepare meals. It can also adjust the tone and content of notifications about food items nearing their expiration date based on the user's emotions.

[0247] Specific examples

[0248] A family has tomatoes, chicken, and pasta in the refrigerator. The user opens the app and enters these ingredients. The family also has a fish allergy. Meanwhile, the local supermarket has a large stock of fresh basil.

[0249] When a user requests "Suggest a dinner menu," the server will suggest a menu based on the following information:

[0250] Homemade tomatoes, chicken, and pasta

[0251] Having a fish allergy

[0252] Fresh basil from the supermarket

[0253] Based on this, the server suggests "Tomato and Chicken Basil Pasta" and encourages the purchase of basil. At the same time, if the server detects stress levels from the user's facial expressions and tone of voice, it can also suggest recipes with shorter cooking times. Also, if the user has milk nearing its expiration date, the user will receive a notification on their device stating, "Your milk will expire in three days." This notification is also displayed in a tone that corresponds to the user's emotional state.

[0254] In this way, by incorporating emotion recognition functionality, this system can provide more personalized services, helping to reduce food waste and ease the burden of housework, as well as managing ingredients in the home and suggesting efficient menus.

[0255] The processing flow will be explained below.

[0256] Program processing flow

[0257] 1. Setting up your user profile

[0258] Step 1: Enter your profile information

[0259] Users open the application on their smartphone or tablet and enter information about their household composition, allergies, and food preferences.

[0260] Step 2: Submit your input

[0261] The terminal transmits the input profile information to the server.

[0262] Step 3: Save your information

[0263] The server stores the received profile information in a database.

[0264] 2. Enter and update existing ingredients

[0265] Step 1: Enter ingredient information

[0266] Users enter the ingredients (item, quantity, expiration date) in their refrigerator or pantry into the app.

[0267] Step 2: Submit your input

[0268] The terminal transmits the input ingredient information to the server.

[0269] Step 3: Save the ingredients

[0270] The server stores the received ingredient information in a database.

[0271] 3. Collaboration with local retailers

[0272] Step 1: Get inventory information

[0273] The server periodically retrieves inventory information from local retailers via APIs and data feeds.

[0274] Step 2: Save inventory information

[0275] The server stores the acquired inventory information in a database.

[0276] Step 3: View inventory information

[0277] The terminal displays real-time inventory information for users from local retailers.

[0278] 4. Menu suggestions

[0279] Step 1: Request a menu suggestion

[0280] The user requests meal suggestions through the application.

[0281] Step 2: Get the data

[0282] The server retrieves user profiles, existing ingredient information, and local retailer inventory information from a database.

[0283] Step 3: Generate a menu

[0284] The server uses AI algorithms to generate optimal menus based on the data it obtains.

[0285] Step 4: Submit and view your menu

[0286] The server sends the generated menu to the terminal.

[0287] The device displays the sent menu to the user.

[0288] 5. Order ingredients online

[0289] Step 1: Create an order

[0290] Users can order the necessary ingredients online based on the suggested menu.

[0291] Step 2: Submit your order

[0292] The terminal transmits the order details to the server.

[0293] Step 3: Transfer your order

[0294] The server forwards the order to the local retailer's system.

[0295] Step 4: Processing the order

[0296] Local retailers receive orders and prepare the ingredients.

[0297] 6. Stockpile management and expiration date notification

[0298] Step 1: Check the expiration date

[0299] The server periodically scans the database to identify ingredients that are nearing their expiration date.

[0300] Step 2: Generate notification content

[0301] The server creates a notification about food items that are nearing their expiration date.

[0302] Step 3: Send notifications

[0303] The server sends the generated notification content to the terminal.

[0304] Step 4: View notifications

[0305] The device will display expiration date notifications to the user.

[0306] 7. Introducing the Emotion Engine

[0307] Step 1: Collecting emotion data

[0308] The device collects emotional data through applications, using user input and sensors such as the camera and microphone, including facial expression recognition, voice recognition, and analysis of input patterns.

[0309] Step 2: Sending emotion data

[0310] The device transmits the collected emotion data to the server.

[0311] Step 3: Analyze the sentiment data

[0312] The server uses an emotion engine to analyze the emotion data and determine the user's current emotional state.

[0313] Step 4: Emotional Adjustment

[0314] The server adjusts the menu suggestions and notifications based on the emotion data recognized by the emotion engine. For example, if the user is feeling stressed, it will suggest simple and easy meals.

[0315] Specific examples

[0316] A family has tomatoes, chicken, and pasta in the refrigerator. The user opens the app and enters these ingredients. They also specify a "fish allergy" in their household information. Meanwhile, the local supermarket has a large stock of fresh basil.

[0317] Example processing steps:

[0318] Step 1: Request a menu

[0319] A user requests, "Suggest a dinner menu."

[0320] Step 2: Get the data

[0321] The server retrieves user profiles, existing food information, and local retailer inventory information.

[0322] Step 3: Generate a menu

[0323] The server uses an AI algorithm to suggest "tomato and chicken basil pasta," taking into account the user's emotional state (e.g., stress) and choosing recipes with short cooking times.

[0324] Step 4: View the proposal

[0325] The device displays the suggested menu to the user, and includes keywords that correspond to emotions (e.g., "easy" and "fun").

[0326] Step 5: Create an order

[0327] A user orders basil online.

[0328] Step 6: Submit your order

[0329] The terminal sends the order details to the server.

[0330] Step 7: Processing the order

[0331] The server forwards the order to a local retailer, which prepares the ingredients.

[0332] Step 8: Expiration date notification

[0333] The server identifies milk that is about to expire and creates a notification saying, "Milk will expire in 3 days."

[0334] The device will display notifications in a friendly tone based on emotion recognition.

[0335] In this way, this system achieves ingredient management, efficient menu suggestions, and even personalized responses through emotion recognition.

[0336] Example 2

[0337] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0338] In modern households, food ingredient management, menu suggestions, and necessary ingredient purchasing are often inefficient. In addition, food ingredients that are close to their expiration date are not adequately managed or utilized, resulting in food waste. Furthermore, systems ignore the user's psychological state and emotions when cooking at home, and do not provide personalized service. There is a need to provide a system that solves these problems, improves the efficiency of food ingredient management at home, and reduces the burden on users.

[0339] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a means for inputting household ingredient information, a means for acquiring inventory information from local retailers, and a means for proposing an optimal menu based on the household ingredient information and the retailer's inventory information. This enables coordination between home ingredient management and local retailers. It also includes a means for collecting user emotions and a means for adjusting the content of suggestions and notifications based on the emotions. This enables personalized menu suggestions and notifications based on the user's psychological state, making home ingredient management more efficient and user-friendly. In addition, the server includes a means for managing the inventory status and expiration dates of household stockpiles and a means for notifying the user of ingredients that are approaching their expiration dates, thereby contributing to the reduction of food waste.

[0340] "Household food information" refers to information such as the type, quantity, and expiration date of food items owned by the user.

[0341] "Local retail store inventory information" refers to information such as the product types, stock quantities, prices, and availability of products held by local retail stores.

[0342] The "optimal menu" is a menu of dishes suggested based on information about ingredients in the household, inventory information at local retailers, the user's household composition, allergy information, and food preferences.

[0343] "Home stockpiles inventory status" is information that indicates the current quantity and status of food ingredients and consumables owned by the user.

[0344] The "expiration date" is information that indicates the period during which food ingredients can be safely consumed.

[0345] "User emotion" refers to data that represents the user's psychological state or mood, including facial expressions, voice tone, input patterns, etc.

[0346] "Means for adjusting the content of suggestions and notifications based on emotions" refers to a function that analyzes a user's emotional data, generates appropriate suggestions and notifications based on the results, and provides them to the user.

[0347] MODE FOR CARRYING OUT THE INVENTION

[0348] This system automates home ingredient management and menu suggestions, and also connects with local retailers to efficiently purchase ingredients. It also recognizes the user's emotions and adjusts the content of suggestions and notifications based on those emotions, providing a more personalized service.

[0349] System configuration

[0350] The system consists of a user's terminal, a server, a database of local retailers, a notification method, and an emotion engine.

[0351] System Operation

[0352] This system uses the following hardware and software to perform specific data processing and calculations.

[0353] User devices: Devices such as smartphones and tablets running iOS or Android.

[0354] Server: Uses cloud servers that host databases and AI algorithms, including database management systems such as MySQL and AI libraries such as TensorFlow.

[0355] Emotion engine: An AI engine for analyzing user emotions, using Google Cloud Vision API, Amazon Rekognition, IBM Watson, etc.

[0356] User profile settings

[0357] Users open the application on their smartphone or tablet and enter information about their family members, allergies, and food preferences, which is then sent from the device to a server and stored in a database.

[0358] Entering and updating existing ingredients

[0359] Users input the ingredients (item, quantity, expiration date) in their refrigerator or pantry into the app. This can be done manually or by using a barcode scanner. This information is sent from the device to the server and stored in a database.

[0360] Collaboration with local retailers

[0361] The server periodically retrieves inventory information from local retailers via APIs and data feeds, which is then stored and updated in a database.

[0362] Menu suggestions

[0363] Users request menu suggestions through the application, and the server applies AI algorithms to generate optimal menus based on the user profile, existing ingredient information, and local retailer inventory information. The generated menus are then sent to the device and displayed to the user.

[0364] Online ordering of ingredients

[0365] Based on the proposed menu, users order the necessary ingredients online through the application. The order information is sent from the device to a server, which then forwards it to the local retailer's system. The retailer receives the order and prepares the ingredients. The user can then pick up the ingredients at the designated store or use a delivery service.

[0366] Stockpile management and expiration date notification

[0367] The server periodically scans the database to identify ingredients that are nearing their expiration date, and users are notified when the expiration date approaches, which is displayed on their device, allowing them to consume the ingredients at the appropriate time.

[0368] Introducing the Emotion Engine

[0369] emotion recognition

[0370] Through applications, the device collects user input and emotions using sensors such as the camera and microphone, including facial expression recognition, voice recognition, and analysis of input patterns.

[0371] Sending and processing emotional data

[0372] The device sends the collected emotional data to a server, which uses an emotion engine to analyze this data and determine the user's current emotional state.

[0373] Emotion-Based Adjustment

[0374] The server adjusts menu suggestions and notification content based on the emotional data recognized by the emotion engine. For example, if the user is feeling stressed, it will suggest easy-to-prepare meals. It can also adjust the tone and content of notifications about food items nearing their expiration date based on the user's emotions.

[0375] Examples of specific examples and prompts

[0376] A family has tomatoes, chicken, and pasta in the refrigerator. The user opens the app and enters these ingredients. The family also has a fish allergy. Meanwhile, the local supermarket has a large stock of fresh basil.

[0377] When a user requests "Suggest a dinner menu," the server will suggest a menu based on the following information:

[0378] Homemade tomatoes, chicken, and pasta

[0379] Having a fish allergy

[0380] Fresh basil from the supermarket

[0381] The server suggests "Basil Pasta with Tomatoes and Chicken" and encourages the user to purchase basil. At the same time, if the server detects stress levels from the user's facial expressions and tone of voice, it can also suggest recipes with shorter cooking times. If the user has milk nearing its expiration date, the user will receive a notification on their device stating, "Your milk will expire in three days." This notification is also displayed in a tone that reflects the user's emotional state.

[0382] Prompt Sentence Examples

[0383] "You have tomatoes, chicken, pasta in your household, and a fish allergy. Your local supermarket has fresh basil in stock. Suggest dinner options. Also, consider adjustments if the user is stressed."

[0384] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0385] Specific processing steps of the program

[0386] Step 1: Setting up a user profile

[0387] Users open the application on their smartphone or tablet and enter their family composition, allergy information, and food preferences. This data is entered and sent to the server, which stores the received data in a database. This allows for personalized menu suggestions.

[0388] Input: Family composition, allergy information, food preferences

[0389] Output: User profile information stored in the database

[0390] Step 2: Enter and update existing ingredients

[0391] Users input the food items, quantities, and expiration dates in their refrigerators or pantries into the application. The input information is sent from the device to the server, which then stores it in a database and manages it as existing food information, enabling accurate inventory management.

[0392] Input: Item, quantity, expiration date

[0393] Output: Existing ingredients stored in the database

[0394] Step 3: Partner with local retailers

[0395] The server periodically retrieves inventory information from local retailers via APIs or data feeds. This data includes items, prices, and stock quantities. The server analyzes the information received and stores it in a database. This ensures that the latest inventory information is always available in the system.

[0396] Input: Local retailer inventory information (item, price, quantity in stock)

[0397] Output: Retail store inventory information stored in a database

[0398] Step 4: Menu suggestions

[0399] Users request menu suggestions through the application. The server uses an AI algorithm to generate the optimal menu based on the user profile, existing ingredient information, and retail store inventory information. This menu information is sent to the device and displayed to the user, allowing the user to check the suggested menu.

[0400] Input: User profile, existing ingredient information, retail store inventory information, menu request

[0401] Output: The optimal menu is generated and sent to the device.

[0402] Step 5: Order ingredients online

[0403] Based on the proposed menu, users order the ingredients they need online through the application. The order information is sent from the device to a server. The server then forwards this information to the local retailer's system, which receives the order and prepares the ingredients. The user can then pick up the ingredients at the designated store or use a delivery service.

[0404] Input: Order Information

[0405] Output: Order information sent to local retailer, ingredients received by user

[0406] Step 6: Stockpile management and expiration date notification

[0407] The server periodically scans the database and checks the expiration dates of registered ingredients. When the expiration date approaches, it sends a notification to the user. This notification is displayed on the device. This allows the user to consume the ingredients appropriately based on the expiration date set.

[0408] Input: Existing ingredient information

[0409] Output: Expiration date notification

[0410] Step 7: Emotion Recognition

[0411] The device collects user input and emotions through sensors such as cameras and microphones. The collected emotion data is sent from the device to a server. This includes facial expression recognition, voice recognition, and input pattern analysis.

[0412] Input: User's facial expressions, voice, and input data

[0413] Output: Emotion data sent to the server

[0414] Step 8: Send and process emotion data

[0415] The server analyzes the received emotional data using an emotion engine to identify the user's emotional state. The analysis results are stored on the server and used for further processing.

[0416] Input: Emotion data

[0417] Output: Analysis results

[0418] Step 9: Emotional Adjustment

[0419] The server then adjusts the menu suggestions and notification content based on the emotion data generated by the emotion engine. For example, if the user is feeling stressed, it will suggest easy-to-prepare meals. It also adjusts the tone and content of notifications about food items that are nearing their expiration date.

[0420] Input: Analysis results, menu suggestion request, expiration date information

[0421] Output: Adjusted menu suggestions, notification content

[0422] In this way, each processing step is explained in detail, and the inputs and outputs, as well as the data processing and calculations based on them, are clearly indicated, making the function and operation of the entire system clear.

[0423] (Application example 2)

[0424] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0425] Currently, there are many systems that efficiently manage ingredients and suggest menus for households, but few systems offer personalized suggestions based on the user's emotional state. Furthermore, it is also challenging to realize efficient ingredient purchasing through collaboration with local retailers. Furthermore, there is a need to support these functions on new devices such as virtual stores and smart glasses.

[0426] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a means for inputting food ingredient information for the household, a means for acquiring inventory information for local retailers, a means for proposing an optimal menu based on the food ingredient information for the household and the inventory information for the local retailers, a means for ordering the ingredients to be used in the proposed menu from the local retailers, a means for recognizing the user's emotions, and a means for adjusting the contents of the proposed menu based on the recognized user emotions. This makes it possible to efficiently manage food ingredients in the household and propose menus, while providing personalized services according to the user's emotional state and enabling efficient food purchasing through collaboration with local retailers.

[0427] The "means for inputting household food information" is an interface for inputting the items, quantities, and expiration dates of food items in the refrigerator or pantry in the home into the system.

[0428] The "means for obtaining inventory information from local retail stores" is a method for obtaining the inventory status of ingredients and products held by local retail stores by linking with the system.

[0429] The "means of proposing the optimal menu" is a process that generates the optimal menu for the user, taking into consideration household composition, allergy information, and food preferences, based on information about ingredients in the household and inventory information from local retailers.

[0430] The "means for ordering ingredients to be used in the proposed menu from the retail store" is a mechanism for ordering the necessary ingredients from a local retail store based on the menu proposed by the system.

[0431] "Means for recognizing user emotions" refers to technology that analyzes the user's facial expressions, tone of voice, input patterns, etc. to identify the user's emotional state.

[0432] The "means for adjusting the content of the suggested menu based on the recognized user's emotions" is an algorithm that changes the difficulty and content of the suggested menu depending on the recognized user's emotions.

[0433] "Means for inputting household composition, allergy information, and food preferences" refers to an interface that allows users to input information about their household composition, allergy information, and food preferences into the system.

[0434] "Means for managing expiration dates" refers to a method for tracking and managing expiration dates of food ingredients in the home.

[0435] "Means for notifying about ingredients approaching their expiration date" refers to the process by which the system identifies ingredients that are approaching their expiration date and sends a notification to the user.

[0436] The system for implementing this invention includes a wearable device such as smart glasses, a smartphone, a local server, and a database of local retailers. The operation of the entire system is configured as follows.

[0437] 1. Setting up your user profile

[0438] Users open the smartphone app and enter personal information such as household composition, allergy information, food preferences, etc. This data is sent from the device to a server and stored in a database.

[0439] 2. Input and management of ingredients

[0440] Users use smart glasses to scan ingredients in their refrigerators or pantries. The smart glasses' camera recognizes the ingredients and can also read QR codes and barcodes. This ingredient information is sent to a server and stored in a database. The server also manages expiration dates and sends notifications to users when ingredients are nearing their expiration date.

[0441] 3. Collaboration with local retailers

[0442] The server uses API integrations and data feeds to periodically retrieve inventory information from local retailers, which is then stored and updated in a database.

[0443] 4. Menu suggestions

[0444] Users can request menu suggestions using smart glasses or smartphones. The server uses AI algorithms to generate optimal menus based on user profiles, ingredient information, and local retailer inventory information. The generated menus are then displayed on the user's device.

[0445] 5. Order ingredients online

[0446] Based on the proposed menu, users can use the application to order the necessary ingredients from local retailers online. The order information is transferred to the local retailer's system via a server, and the retailer receives the order and prepares the ingredients.

[0447] 6. Emotion recognition function

[0448] The smart glasses use sensors (camera, microphone) built into them to analyze the user's facial expressions and tone of voice to recognize their emotions, and then use emotion-recognition AI to adjust the suggested menu and notification tone based on the user's emotional state.

[0449] For example, if it senses that the user is tired, it will suggest easy-to-prepare meals. The following prompts are fed into the generative AI model:

[0450] "Please suggest some simple recipes based on the user's current emotional state. Emotion recognition data is as follows: {emotion data}"

[0451] Specific examples

[0452] The user uses smart glasses in front of the refrigerator to scan "tomato," "chicken," and "pasta." The camera in the smart glasses recognizes these ingredients and sends the information to the server. When the user requests a dinner menu, the server considers the user's household composition, allergy information, and food preferences, and suggests "tomato and chicken basil pasta" based on local retailer inventory information (such as fresh basil). If the server detects that the user is tired, it prioritizes recipes with shorter cooking times.

[0453] This invention contributes to reducing food waste and the burden of housework by streamlining food management within the home and providing more personalized services based on the user's emotions.

[0454] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0455] Step 1:

[0456] Users open the smartphone app and enter their user profile, including household composition, allergy information, food preferences, etc. The entered data is sent from the device to the server, which then stores it in a database.

[0457] Step 2:

[0458] The user uses smart glasses to scan ingredients in the refrigerator or pantry. The camera in the smart glasses recognizes the ingredients and reads the QR code or barcode. This recognized ingredient information is sent to the server, which stores it in a database. During this process, information such as the ingredient type, quantity, and expiration date is processed.

[0459] Step 3:

[0460] The server periodically retrieves inventory information from local retailers via API. The retrieved inventory information is stored and updated in a database. The data used here includes product availability and price information for retailers.

[0461] Step 4:

[0462] The user requests menu suggestions using smart glasses or a smartphone. The server uses an AI algorithm to generate an optimal menu based on the user profile, existing ingredient information, and inventory information from local retailers. This generated menu is then displayed on the user's device. Menu generation involves a recipe generation process using a generative AI model.

[0463] Step 5:

[0464] Based on the proposed menu, users can order the necessary ingredients online through the application. The order information is sent from the terminal to the server, which then forwards this data to the local retailer's system. The retailer receives the order and prepares the ingredients.

[0465] Step 6:

[0466] The server uses sensors (camera, microphone) built into the smart glasses to analyze the user's facial expressions and tone of voice to recognize their emotions. This emotional data is then analyzed by emotion recognition AI to determine the user's current emotional state.

[0467] Step 7:

[0468] Based on the recognized user's emotional data, the server generates a prompt for the AI ​​model, saying, "Please suggest a simple dish based on the user's current emotional state." This prompt is input into the AI ​​model, which then generates an optimal menu based on the user's emotional state.

[0469] Step 8:

[0470] The server sends the generated menu to the user's device. For example, if the server detects that the user is tired, it will suggest a menu that is easy to prepare. For example, it might suggest "easy basil pasta with tomatoes and chicken."

[0471] Step 9:

[0472] The server manages expiration dates for food items in the home and sends notifications to the user about food items that are nearing their expiration date. The tone and content of the notifications are adjusted according to the user's emotional state.

[0473] The above processing steps will improve the efficiency of home food management and menu suggestions, and provide personalized services according to the user's emotional state. It will also enable efficient food purchasing through collaboration with local retailers.

[0474] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0475] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0476] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0477] [Second embodiment]

[0478] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0479] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0480] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0481] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0482] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0483] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0484] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0485] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0486] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0487] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0488] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0489] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0490] This invention is a system that automates food ingredient management and menu suggestions for the home, and also enables efficient food purchasing by linking with local retailers. This system integrates information on home ingredients and inventory information from local retailers, suggests optimal menus for users, and supports them in ordering ingredients.

[0491] System configuration

[0492] The system consists of a user terminal, a server, a database of local retailers, and a notification means.

[0493] System Operation

[0494] 1. Setting up your user profile

[0495] Users open the application on their smartphones, tablets, or other devices and enter their family members, allergy information, and food preferences. This data is then sent from the device to a server and stored in a database.

[0496] 2. Enter and update existing ingredients

[0497] Users input the ingredients (item, quantity, expiration date) in their refrigerator or pantry into the app, and this information is sent from the device to the server and stored in a database.

[0498] 3. Collaboration with local retailers

[0499] The server periodically retrieves inventory information from local retailers, often automatically through an API, and stores and updates the database.

[0500] 4. Menu suggestions

[0501] Users request menu suggestions through the application, and the server uses AI algorithms to generate optimal menus based on the user profile, existing ingredient information, and local retailer inventory information. These menu suggestions are then sent to the device and displayed to the user.

[0502] 5. Order ingredients online

[0503] Based on the proposed menu, users can order the necessary ingredients online through the application. The order information is sent from the device to a server, which then forwards it to the local retailer's system. The retailer receives the order and prepares the ingredients.

[0504] 6. Stockpile management and expiration date notification

[0505] The server periodically scans the database to identify ingredients that are nearing their expiration date, and users are notified when the expiration date approaches, which is displayed on their device, allowing them to consume the ingredients at the appropriate time.

[0506] Specific examples

[0507] A family has tomatoes, chicken, and pasta in the refrigerator. The user opens the app and enters these ingredients. The family also has a fish allergy. Meanwhile, the local supermarket has a large stock of fresh basil.

[0508] When a user requests "Suggest a dinner menu," the server will suggest a menu based on the following information:

[0509] Homemade tomatoes, chicken, and pasta

[0510] Having a fish allergy

[0511] Fresh basil from the supermarket

[0512] Based on this, the server will suggest "Tomato and Chicken Basil Pasta" and encourage the purchase of basil. Also, if there is milk nearing its expiration date, the user will receive a notification on their device saying, "The milk's expiration date is coming up in three days."

[0513] In this way, this system provides food management within the home and efficient menu suggestions, contributing to reducing food waste and easing the burden of housework.

[0514] The processing flow will be explained below.

[0515] Program processing flow

[0516] 1. Setting up your user profile

[0517] Step 1: Enter your profile information

[0518] Users open the application on their smartphone or tablet and enter information about their household composition, allergies, and food preferences.

[0519] Step 2: Submit your input

[0520] The terminal transmits the input profile information to the server.

[0521] Step 3: Save your information

[0522] The server stores the received profile information in a database.

[0523] 2. Enter and update existing ingredients

[0524] Step 1: Enter ingredient information

[0525] Users enter the ingredients (item, quantity, expiration date) in their refrigerator or pantry into the app.

[0526] Step 2: Submit your input

[0527] The terminal transmits the input ingredient information to the server.

[0528] Step 3: Save the ingredients

[0529] The server stores the received ingredient information in a database.

[0530] 3. Collaboration with local retailers

[0531] Step 1: Get inventory information

[0532] The server periodically retrieves inventory information from local retailers via API or data feed.

[0533] Step 2: Save inventory information

[0534] The server stores the acquired inventory information in a database.

[0535] Step 3: View inventory information

[0536] The terminal displays real-time inventory information for users from local retailers.

[0537] 4. Menu suggestions

[0538] Step 1: Request a menu suggestion

[0539] The user requests meal suggestions through the application.

[0540] Step 2: Get the data

[0541] The server retrieves user profiles, existing ingredient information, and local retailer inventory information from a database.

[0542] Step 3: Generate a menu

[0543] The server uses AI algorithms to generate optimal menus based on the data it obtains.

[0544] Step 4: Submit and view your menu

[0545] The server sends the generated menu to the terminal.

[0546] The device displays the sent menu to the user.

[0547] 5. Order ingredients online

[0548] Step 1: Create an order

[0549] Users can order the necessary ingredients online based on the suggested menu.

[0550] Step 2: Submit your order

[0551] The terminal transmits the order details to the server.

[0552] Step 3: Transfer your order

[0553] The server forwards the order to the local retailer's system.

[0554] Step 4: Processing the order

[0555] Local retailers receive orders and prepare the ingredients.

[0556] 6. Stockpile management and expiration date notification

[0557] Step 1: Check the expiration date

[0558] The server periodically scans the database to identify ingredients that are nearing their expiration date.

[0559] Step 2: Generate notification content

[0560] The server creates a notification about food items that are nearing their expiration date.

[0561] Step 3: Send notifications

[0562] The server sends the generated notification content to the terminal.

[0563] Step 4: View notifications

[0564] The device will display expiration date notifications to the user.

[0565] Example 1

[0566] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0567] In modern households, managing and purchasing ingredients, as well as creating menus, requires a lot of time and effort, which is a major challenge. People often overlook expiration dates on ingredients, resulting in food waste. Furthermore, there is a lack of systems that efficiently link household ingredients with inventory information from local retailers, making it difficult to efficiently purchase the ingredients needed. This increases the burden of daily housework and ultimately leads to a decline in quality of life.

[0568] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0569] In this invention, the server includes a means for inputting household food ingredient information, a means for periodically obtaining inventory information from local retailers via an API, a means for proposing optimal menus using a generative AI model based on the household food ingredient information and the local retailer's inventory information, and a means for online ordering of ingredients used in the proposed menu from the retailer. This automates household food ingredient management and menu proposals, enabling efficient food purchasing. Furthermore, by managing and notifying expiration dates, food waste can be reduced, reducing the burden of housework and improving quality of life.

[0570] "Household food ingredient information" is detailed information such as the type, quantity, and expiration date of food ingredients used in the home.

[0571] "Local retailers" refer to stores that sell food and daily necessities in a specific area.

[0572] "Inventory information" refers to information about the types, quantities, prices, and availability of products available at local retail stores.

[0573] "API" stands for Application Program Interface, a means by which different software applications communicate with each other.

[0574] A "generative AI model" is an algorithm or system that uses artificial intelligence techniques to analyze data and generate new information or suggestions.

[0575] A "menu" refers to the combination of foods and dishes for a particular meal, usually suggested for breakfast, lunch, and dinner.

[0576] "Online ordering" refers to the act of purchasing goods over the Internet, particularly in an e-commerce environment.

[0577] "Best before date" indicates the last date that a food product can be safely consumed and still retain its quality.

[0578] A "notification" is a message or signal intended to inform a user of certain information or events.

[0579] This invention is a system that allows efficient management of home ingredients, menu planning, and online ordering of ingredients. To implement this system, a user terminal, a server, a database of local retailers, and a notification means are required.

[0580] First, users open the application on their smartphone, tablet, or other device and enter information about their family, allergies, and food preferences. This information is then sent from the device to a server and stored in a database.

[0581] Next, users enter the items, quantities, and expiration dates of ingredients in their refrigerator or pantry into the app, which then sends this information from the device to a server and stores it in a database.

[0582] The server periodically retrieves inventory information from local retailers via API and stores it in a database. This process is automated, ensuring that inventory information is always up-to-date.

[0583] When a user requests a menu suggestion through the app, the server uses a generative AI model to generate an optimal menu based on the user profile, existing ingredient information, and local retailer inventory information. This suggestion is then sent to the device and displayed to the user.

[0584] The user orders the ingredients needed based on the proposed menu online. The order information is sent from the terminal to a server, which then forwards it to the local retailer's system. The retailer receives the order and prepares the ingredients.

[0585] The server periodically scans the database to identify ingredients that are nearing their expiration date, and users are notified when the expiration date approaches, which is displayed on their device, allowing them to consume the ingredients at the appropriate time.

[0586] As a concrete example, suppose a household has tomatoes, chicken, and pasta in the refrigerator, and the family consists of two adults, one child, and a person with a fish allergy. If a user from this household, who has fresh basil in stock at a local retailer, requests "Suggest a dinner menu," the server generates a menu for "Tomato and Chicken Basil Pasta" based on the following information and encourages the user to purchase basil. If there is milk nearing its expiration date, a notification stating "The milk's expiration date is in three days" will be displayed on the user's device.

[0587] An example prompt might look like this:

[0588] "If a user opens a food management app, inputs the tomatoes, chicken, and pasta in their refrigerator, and indicates that a family member has a fish allergy, and requests the app to suggest a dinner menu, the server will suggest a tomato, chicken, and basil pasta dish and encourage them to purchase fresh basil. Also, if there is milk nearing its expiration date, a notification will appear on the user's device stating, 'Your milk will expire in three days.'"

[0589] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0590] Step 1:

[0591] The user opens the application on a device such as a smartphone or tablet. First, the user profile setting screen is displayed. The user enters the family composition (e.g., two adults, one child), allergy information (e.g., fish allergy), and food preferences (e.g., prefers Italian food). The device converts this input information into JSON format and sends it to the server. The server receives this JSON data and stores it in the user profile table in the database. In this step, the input data is private information such as family composition and allergy information, and the output data means that the user profile is saved in the database.

[0592] Step 2:

[0593] The user enters the ingredients (e.g., tomatoes, chicken, pasta), quantity (e.g., 3 tomatoes, 2 chicken pieces), and expiration date (e.g., 2023-10-10) of ingredients in the household on the app's ingredient management screen. The device acquires this information, converts it back into JSON format, and sends it to the server. The server receives this JSON data, saves it in the ingredient table of the database, and compares it with existing information to update it. The input data for this step are detailed information about each ingredient, and the output data is the updated information for the ingredient table.

[0594] Step 3:

[0595] The server runs a scheduled job every day at 3 AM, which sends a request to the local retailer's API endpoint to retrieve inventory information. The retrieved inventory information is returned to the server in JSON format. The server receives this information and updates an inventory table in the database. The input data for this step is the retailer's inventory information, and the output data is the updated inventory table contents.

[0596] Step 4:

[0597] The user taps the "Suggest a Meal" button in the application. The device sends this request to the server. Based on the received request, the server queries the user profile, existing ingredient information, and inventory information from local retailers, and uses an AI algorithm to generate an optimal meal plan. The generated meal plan is sent to the device in JSON format. The device parses this data and displays the suggested meal plan to the user. The input data for this step are the user request and related ingredient information, and the output data is the suggested meal plan.

[0598] Step 5:

[0599] Based on the proposed menu, the user uses the application to order the ingredients they need online. For example, they tap the "Buy basil" button. The device sends this purchase request to the server. The server then forwards the request to the local retailer's order system. The local retailer confirms the order and prepares the ingredients. The input data for this step is the user's purchase request, and the output data indicates that the order has been accepted and preparation has begun.

[0600] Step 6:

[0601] The server runs a scheduled job every day at 2 PM that scans the ingredient table in the database to identify ingredients that are close to their expiration date. The server generates a notification for ingredients with an expiration date within three days and sends it to the user's device. The device displays this notification to the user as a push notification. The input data for this step is the expiration date information in the ingredient table, and the output data is the notification information displayed to the user.

[0602] Through these steps, we can create a system that efficiently manages household ingredients, proposes menus, and orders ingredients online. This system will be very useful in reducing food waste and easing the burden of housework.

[0603] (Application example 1)

[0604] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0605] Modern households often spend a great deal of time and effort managing ingredients and planning daily meals, creating a need for efficient ingredient use and reduced food waste. Finding optimal ways to purchase ingredients and quickly supply them is also a challenge. Effective menu suggestions tailored to each household's needs and management of expiration dates are also desirable. To solve these problems, a system is needed that integrates information about ingredients within the home, inventory information from local retailers, and food delivery services, and provides unified management, suggestions, ordering, and delivery.

[0606] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0607] In this invention, the server includes a means for inputting household ingredient information, a means for acquiring inventory information from local retailers, a means for proposing an optimal menu based on the household ingredient information and the retailer's inventory information, a means for ordering ingredients for the proposed menu from the retailer, and a means for coordinating with a food delivery service to deliver the ordered ingredients to the user. This allows users to efficiently manage ingredients in their home, propose optimal menus, and quickly purchase and deliver the necessary ingredients. It also contributes to reducing food waste and easing the burden of housework.

[0608] "Household food information" refers to detailed data such as the items, quantities, and expiration dates of food and ingredients that are normally stored in the home.

[0609] "Local retail store inventory information" refers to detailed data such as the availability, prices, and expiration dates of food ingredients and products held by local retail stores.

[0610] "Menu suggestions" refers to the act of presenting the most suitable dishes and meal menus based on the user's available ingredients and individual needs (allergy information, food preferences, etc.).

[0611] "Ordering ingredients" refers to the process by which a user specifies the ingredients needed based on a suggested menu and purchases these ingredients from a retailer.

[0612] "Food delivery service" refers to a business that includes delivering ingredients ordered by users to designated locations.

[0613] "Food inventory status" refers to the current amount and expiration date of a particular food ingredient at a home or storage facility.

[0614] "Expiration date notification" refers to a function that notifies users when the expiration date of stored food ingredients is approaching.

[0615] A "generative AI model" refers to an artificial intelligence algorithm that automatically generates optimal prompts and suggestions based on data and information generated from user input.

[0616] A "prompt" is a guide or instruction generated to encourage a user to take a particular action or response.

[0617] This invention is a system that automates household food ingredient management and meal planning, and also enables efficient food purchasing and delivery through food delivery services. This system integrates information on household ingredients, inventory information from local retailers, and information from food delivery services, suggests optimal meals for users, and supports the ordering and delivery of ingredients.

[0618] System configuration

[0619] The system consists of a user terminal, a server, a database of local retailers, a food delivery service, and a notification method.

[0620] Hardware and Software Configuration

[0621] Hardware

[0622] Smartphone (iOS, Android)

[0623] Cloud services (AWS, Google Cloud, Microsoft Azure)

[0624] software

[0625] Food management application: Python (Flask)

[0626] Database: MongoDB

[0627] Integration with local store API: RESTful API

[0628] Processing Description

[0629] User profile settings

[0630] Users open the application on their smartphone and enter their family information, allergy information, and food preferences. This data is sent to the server in JSON format and stored in a database.

[0631] Entering and updating existing ingredients

[0632] Users input the ingredients (item, quantity, expiration date) in their refrigerator or pantry into the app, and this information is sent from the device to the server and stored in a database.

[0633] Collaboration with local retailers

[0634] The server periodically retrieves inventory information from local retailers via API, which is then stored and updated in a database.

[0635] Menu suggestions

[0636] When a user requests a menu suggestion, the server uses an AI algorithm to suggest the optimal menu based on the user profile, existing ingredient information, and retail store inventory information, and sends it to the device.

[0637] Online food ordering and delivery

[0638] Based on the proposed menu, users can order the necessary ingredients online through the application. The order information is transferred from the server to the local retailer's system, which receives the order, prepares the ingredients, and then works with a food delivery service to deliver the ingredients to the user.

[0639] Expiration date notification

[0640] The server periodically scans the database to identify ingredients that are nearing their expiration date, and when this happens, a notification is sent to the user and displayed on their device.

[0641] Specific examples

[0642] A family has tomatoes, chicken, and pasta in the refrigerator. The user opens the app and enters these ingredients. The family also includes a fish allergy. Meanwhile, the local supermarket has a large stock of fresh basil.

[0643] When a user requests "Suggest a dinner menu," the server will suggest a menu based on the following information:

[0644] Homemade tomatoes, chicken, and pasta

[0645] Having a fish allergy

[0646] Fresh basil from the supermarket

[0647] Based on this, the server will suggest "Tomato and Chicken Basil Pasta" and encourage the purchase of basil. Also, if there is milk nearing its expiration date, the user will receive a notification on their device saying, "The milk's expiration date is coming up in three days."

[0648] An example of a prompt sentence in this case is:

[0649] "Dear user, based on the ingredients in your fridge, I'd like to suggest 'Tomato and Chicken Basil Pasta'. You need basil, but you can buy it at your local supermarket. Would you like to order basil?"

[0650] is displayed.

[0651] In this way, this system can provide food management within the home and efficient menu suggestions, contributing to reducing food waste and easing the burden of housework.

[0652] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0653] Step 1:

[0654] User profile settings

[0655] Users open the smartphone application and input their family composition, allergy information, and food preferences. This data is sent to the server in JSON format upon input, and the server stores the received data in a database. The input data includes the number of family members, types of allergies, and preferred food ingredients. The output is accurately stored in the database.

[0656] Step 2:

[0657] Entering and updating existing ingredients

[0658] The user inputs the item, quantity, and expiration date of each ingredient in the refrigerator or pantry into the application. This data is also sent to the server in JSON format, and after the server receives it, it is stored in the database. The input data includes the ingredient name, quantity, and expiration date. The current ingredient information is reflected in the database as output.

[0659] Step 3:

[0660] Collaboration with local retailers

[0661] The server periodically retrieves inventory information from local retailers via a specified API. The retrieved data includes detailed information such as stock items, prices, and expiration dates, and is also received in JSON format and stored in a database. An API key is required to retrieve inventory information, and the latest information can be retrieved automatically by setting up requests to be sent at regular intervals. The latest inventory information is updated in the database as an output.

[0662] Step 4:

[0663] Menu suggestions

[0664] When a user requests a menu suggestion through the application, the request is sent to the server. The server uses a generative AI model to generate an optimal menu based on the user profile in the database, existing ingredient information, and local retailer inventory information. Input data includes the user's preferences and allergies, currently owned ingredients, and local retailer inventory information. The suggested menu is sent as output to the user's device.

[0665] Step 5:

[0666] Online food ordering and delivery

[0667] The user selects the ingredients they need based on the suggested menu and places an order online through the application. The order information is sent in JSON format to the server, which then forwards it to the local retailer's system. The server then notifies the retailer and coordinates with the food delivery service to deliver the ordered ingredients to the destination. Input data includes the name of the ingredients to be ordered, the quantity, and delivery address information. The output is a record of the order history in a database, and the ingredients are delivered to the user.

[0668] Step 6:

[0669] Expiration date notification

[0670] The server periodically scans the database to identify ingredients that are nearing their expiration date. When the expiration date approaches, the information is notified to the user's device. The input data is the current ingredient information and its expiration date, and a database query is used to extract ingredients that are approaching their expiration date. The output is a notification sent to the user's device, displaying the message "The expiration date for XX is coming up in 3 days."

[0671] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0672] This system automates home ingredient management and menu suggestions, and also works with local retailers to efficiently purchase ingredients. It also recognizes the user's emotions and adjusts the content of suggestions and notifications based on those emotions, providing a more personalized service.

[0673] System configuration

[0674] The system consists of a user's terminal, a server, a database of local retailers, a notification method, and an emotion engine.

[0675] System Operation

[0676] 1. Setting up your user profile

[0677] Users open the application on their smartphone or tablet and enter information about their family, allergies, and food preferences. This data is then sent from the device to a server and stored in a database.

[0678] 2. Enter and update existing ingredients

[0679] Users input the ingredients (item, quantity, expiration date) in their refrigerator or pantry into the app, and this information is sent from the device to the server and stored in a database.

[0680] 3. Collaboration with local retailers

[0681] The server periodically retrieves inventory information from local retailers via APIs and data feeds, which is then stored and updated in a database.

[0682] 4. Menu suggestions

[0683] Users request menu suggestions through the application, and the server uses AI algorithms to generate optimal menus based on the user profile, existing ingredient information, and local retailer inventory information. The generated menus are then sent to the device and displayed to the user.

[0684] 5. Order ingredients online

[0685] Based on the proposed menu, users can order the necessary ingredients online through the application. The order information is sent from the device to a server, which then forwards it to the local retailer's system. The retailer receives the order and prepares the ingredients.

[0686] 6. Stockpile management and expiration date notification

[0687] The server periodically scans the database to identify ingredients that are nearing their expiration date, and users are notified when the expiration date approaches, which is displayed on their device, allowing them to consume the ingredients at the appropriate time.

[0688] Introducing the Emotion Engine

[0689] 7. Emotion recognition

[0690] Through applications, the device collects user input and emotions using sensors such as the camera and microphone, including facial expression recognition, voice recognition, and analysis of input patterns.

[0691] 8. Transmission and Processing of Emotional Data

[0692] The device sends the collected emotional data to a server, which uses an emotion engine to analyze this data and determine the user's current emotional state.

[0693] 9. Emotion-Based Regulation

[0694] The server adjusts menu suggestions and notification content based on the emotional data recognized by the emotion engine. For example, if the user is feeling stressed, it will suggest easy-to-prepare meals. It can also adjust the tone and content of notifications about food items nearing their expiration date based on the user's emotions.

[0695] Specific examples

[0696] A family has tomatoes, chicken, and pasta in the refrigerator. The user opens the app and enters these ingredients. The family also has a fish allergy. Meanwhile, the local supermarket has a large stock of fresh basil.

[0697] When a user requests "Suggest a dinner menu," the server will suggest a menu based on the following information:

[0698] Homemade tomatoes, chicken, and pasta

[0699] Having a fish allergy

[0700] Fresh basil from the supermarket

[0701] Based on this, the server suggests "Tomato and Chicken Basil Pasta" and encourages the purchase of basil. At the same time, if the server detects stress levels from the user's facial expressions and tone of voice, it can also suggest recipes with shorter cooking times. Also, if the user has milk nearing its expiration date, the user will receive a notification on their device stating, "Your milk will expire in three days." This notification is also displayed in a tone that corresponds to the user's emotional state.

[0702] In this way, by incorporating emotion recognition functionality, this system can provide more personalized services, helping to reduce food waste and ease the burden of housework, as well as managing ingredients in the home and suggesting efficient menus.

[0703] The processing flow will be explained below.

[0704] Program processing flow

[0705] 1. Setting up your user profile

[0706] Step 1: Enter your profile information

[0707] Users open the application on their smartphone or tablet and enter information about their household composition, allergies, and food preferences.

[0708] Step 2: Submit your input

[0709] The terminal transmits the input profile information to the server.

[0710] Step 3: Save your information

[0711] The server stores the received profile information in a database.

[0712] 2. Enter and update existing ingredients

[0713] Step 1: Enter ingredient information

[0714] Users enter the ingredients (item, quantity, expiration date) in their refrigerator or pantry into the app.

[0715] Step 2: Submit your input

[0716] The terminal transmits the input ingredient information to the server.

[0717] Step 3: Save the ingredients

[0718] The server stores the received ingredient information in a database.

[0719] 3. Collaboration with local retailers

[0720] Step 1: Get inventory information

[0721] The server periodically retrieves inventory information from local retailers via APIs and data feeds.

[0722] Step 2: Save inventory information

[0723] The server stores the acquired inventory information in a database.

[0724] Step 3: View inventory information

[0725] The terminal displays real-time inventory information for users from local retailers.

[0726] 4. Menu suggestions

[0727] Step 1: Request a menu suggestion

[0728] The user requests meal suggestions through the application.

[0729] Step 2: Get the data

[0730] The server retrieves user profiles, existing ingredient information, and local retailer inventory information from a database.

[0731] Step 3: Generate a menu

[0732] The server uses AI algorithms to generate optimal menus based on the data it obtains.

[0733] Step 4: Submit and view your menu

[0734] The server sends the generated menu to the terminal.

[0735] The device displays the sent menu to the user.

[0736] 5. Order ingredients online

[0737] Step 1: Create an order

[0738] Users can order the necessary ingredients online based on the suggested menu.

[0739] Step 2: Submit your order

[0740] The terminal transmits the order details to the server.

[0741] Step 3: Transfer your order

[0742] The server forwards the order to the local retailer's system.

[0743] Step 4: Processing the order

[0744] Local retailers receive orders and prepare the ingredients.

[0745] 6. Stockpile management and expiration date notification

[0746] Step 1: Check the expiration date

[0747] The server periodically scans the database to identify ingredients that are nearing their expiration date.

[0748] Step 2: Generate notification content

[0749] The server creates a notification about food items that are nearing their expiration date.

[0750] Step 3: Send notifications

[0751] The server sends the generated notification content to the terminal.

[0752] Step 4: View notifications

[0753] The device will display expiration date notifications to the user.

[0754] 7. Introducing the Emotion Engine

[0755] Step 1: Collecting emotion data

[0756] The device collects emotional data through applications, using user input and sensors such as the camera and microphone, including facial expression recognition, voice recognition, and analysis of input patterns.

[0757] Step 2: Sending emotion data

[0758] The device transmits the collected emotion data to the server.

[0759] Step 3: Analyze the sentiment data

[0760] The server uses an emotion engine to analyze the emotion data and determine the user's current emotional state.

[0761] Step 4: Emotional Adjustment

[0762] The server adjusts the menu suggestions and notifications based on the emotion data recognized by the emotion engine. For example, if the user is feeling stressed, it will suggest simple and easy meals.

[0763] Specific examples

[0764] A family has tomatoes, chicken, and pasta in the refrigerator. The user opens the app and enters these ingredients. They also specify a "fish allergy" in their household information. Meanwhile, the local supermarket has a large stock of fresh basil.

[0765] Example processing steps:

[0766] Step 1: Request a menu

[0767] A user requests, "Suggest a dinner menu."

[0768] Step 2: Get the data

[0769] The server retrieves user profiles, existing food information, and local retailer inventory information.

[0770] Step 3: Generate a menu

[0771] The server uses an AI algorithm to suggest "tomato and chicken basil pasta," taking into account the user's emotional state (e.g., stress) and choosing recipes with short cooking times.

[0772] Step 4: View the proposal

[0773] The device displays the suggested menu to the user, and includes keywords that correspond to emotions (e.g., "easy" and "fun").

[0774] Step 5: Create an order

[0775] A user orders basil online.

[0776] Step 6: Submit your order

[0777] The terminal sends the order details to the server.

[0778] Step 7: Processing the order

[0779] The server forwards the order to a local retailer, which prepares the ingredients.

[0780] Step 8: Expiration date notification

[0781] The server identifies milk that is about to expire and creates a notification saying, "Milk will expire in 3 days."

[0782] The device will display notifications in a friendly tone based on emotion recognition.

[0783] In this way, this system achieves ingredient management, efficient menu suggestions, and even personalized responses through emotion recognition.

[0784] Example 2

[0785] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0786] In modern households, food ingredient management, menu suggestions, and necessary ingredient purchasing are often inefficient. In addition, food ingredients that are close to their expiration date are not adequately managed or utilized, resulting in food waste. Furthermore, systems ignore the user's psychological state and emotions when cooking at home, and do not provide personalized service. There is a need to provide a system that solves these problems, improves the efficiency of food ingredient management at home, and reduces the burden on users.

[0787] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a means for inputting household ingredient information, a means for acquiring inventory information from local retailers, and a means for proposing an optimal menu based on the household ingredient information and the retailer's inventory information. This enables coordination between home ingredient management and local retailers. It also includes a means for collecting user emotions and a means for adjusting the content of suggestions and notifications based on the emotions. This enables personalized menu suggestions and notifications based on the user's psychological state, making home ingredient management more efficient and user-friendly. In addition, the server includes a means for managing the inventory status and expiration dates of household stockpiles and a means for notifying the user of ingredients that are approaching their expiration dates, thereby contributing to the reduction of food waste.

[0788] "Household food information" refers to information such as the type, quantity, and expiration date of food items owned by the user.

[0789] "Local retail store inventory information" refers to information such as the product types, stock quantities, prices, and availability of products held by local retail stores.

[0790] The "optimal menu" is a menu of dishes suggested based on information about ingredients in the household, inventory information at local retailers, the user's household composition, allergy information, and food preferences.

[0791] "Home stockpiles inventory status" is information that indicates the current quantity and status of food ingredients and consumables owned by the user.

[0792] The "expiration date" is information that indicates the period during which food ingredients can be safely consumed.

[0793] "User emotion" refers to data that represents the user's psychological state or mood, including facial expressions, voice tone, input patterns, etc.

[0794] "Means for adjusting the content of suggestions and notifications based on emotions" refers to a function that analyzes a user's emotional data, generates appropriate suggestions and notifications based on the results, and provides them to the user.

[0795] MODE FOR CARRYING OUT THE INVENTION

[0796] This system automates home ingredient management and menu suggestions, and also connects with local retailers to efficiently purchase ingredients. It also recognizes the user's emotions and adjusts the content of suggestions and notifications based on those emotions, providing a more personalized service.

[0797] System configuration

[0798] The system consists of a user's terminal, a server, a database of local retailers, a notification method, and an emotion engine.

[0799] System Operation

[0800] This system uses the following hardware and software to perform specific data processing and calculations.

[0801] User devices: Devices such as smartphones and tablets running iOS or Android.

[0802] Server: Uses cloud servers that host databases and AI algorithms, including database management systems such as MySQL and AI libraries such as TensorFlow.

[0803] Emotion engine: An AI engine for analyzing user emotions, using Google Cloud Vision API, Amazon Rekognition, IBM Watson, etc.

[0804] User profile settings

[0805] Users open the application on their smartphone or tablet and enter information about their family members, allergies, and food preferences, which is then sent from the device to a server and stored in a database.

[0806] Entering and updating existing ingredients

[0807] Users input the ingredients (item, quantity, expiration date) in their refrigerator or pantry into the app. This can be done manually or by using a barcode scanner. This information is sent from the device to the server and stored in a database.

[0808] Collaboration with local retailers

[0809] The server periodically retrieves inventory information from local retailers via APIs and data feeds, which is then stored and updated in a database.

[0810] Menu suggestions

[0811] Users request menu suggestions through the application, and the server applies AI algorithms to generate optimal menus based on the user profile, existing ingredient information, and local retailer inventory information. The generated menus are then sent to the device and displayed to the user.

[0812] Online ordering of ingredients

[0813] Based on the proposed menu, users order the necessary ingredients online through the application. The order information is sent from the device to a server, which then forwards it to the local retailer's system. The retailer receives the order and prepares the ingredients. The user can then pick up the ingredients at the designated store or use a delivery service.

[0814] Stockpile management and expiration date notification

[0815] The server periodically scans the database to identify ingredients that are nearing their expiration date, and users are notified when the expiration date approaches, which is displayed on their device, allowing them to consume the ingredients at the appropriate time.

[0816] Introducing the Emotion Engine

[0817] emotion recognition

[0818] Through applications, the device collects user input and emotions using sensors such as the camera and microphone, including facial expression recognition, voice recognition, and analysis of input patterns.

[0819] Sending and processing emotional data

[0820] The device sends the collected emotional data to a server, which uses an emotion engine to analyze this data and determine the user's current emotional state.

[0821] Emotion-Based Adjustment

[0822] The server adjusts menu suggestions and notification content based on the emotional data recognized by the emotion engine. For example, if the user is feeling stressed, it will suggest easy-to-prepare meals. It can also adjust the tone and content of notifications about food items nearing their expiration date based on the user's emotions.

[0823] Examples of specific examples and prompts

[0824] A family has tomatoes, chicken, and pasta in the refrigerator. The user opens the app and enters these ingredients. The family also has a fish allergy. Meanwhile, the local supermarket has a large stock of fresh basil.

[0825] When a user requests "Suggest a dinner menu," the server will suggest a menu based on the following information:

[0826] Homemade tomatoes, chicken, and pasta

[0827] Having a fish allergy

[0828] Fresh basil from the supermarket

[0829] The server suggests "Basil Pasta with Tomatoes and Chicken" and encourages the user to purchase basil. At the same time, if the server detects stress levels from the user's facial expressions and tone of voice, it can also suggest recipes with shorter cooking times. If the user has milk nearing its expiration date, the user will receive a notification on their device stating, "Your milk will expire in three days." This notification is also displayed in a tone that reflects the user's emotional state.

[0830] Prompt Sentence Examples

[0831] "You have tomatoes, chicken, pasta in your household, and a fish allergy. Your local supermarket has fresh basil in stock. Suggest dinner options. Also, consider adjustments if the user is stressed."

[0832] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0833] Specific processing steps of the program

[0834] Step 1: Setting up a user profile

[0835] Users open the application on their smartphone or tablet and enter their family composition, allergy information, and food preferences. This data is entered and sent to the server, which stores the received data in a database. This allows for personalized menu suggestions.

[0836] Input: Family composition, allergy information, food preferences

[0837] Output: User profile information stored in the database

[0838] Step 2: Enter and update existing ingredients

[0839] Users input the food items, quantities, and expiration dates in their refrigerators or pantries into the application. The input information is sent from the device to the server, which then stores it in a database and manages it as existing food information, enabling accurate inventory management.

[0840] Input: Item, quantity, expiration date

[0841] Output: Existing ingredients stored in the database

[0842] Step 3: Partner with local retailers

[0843] The server periodically retrieves inventory information from local retailers via APIs or data feeds. This data includes items, prices, and stock quantities. The server analyzes the information received and stores it in a database. This ensures that the latest inventory information is always available in the system.

[0844] Input: Local retailer inventory information (item, price, quantity in stock)

[0845] Output: Retail store inventory information stored in a database

[0846] Step 4: Menu suggestions

[0847] Users request menu suggestions through the application. The server uses an AI algorithm to generate the optimal menu based on the user profile, existing ingredient information, and retail store inventory information. This menu information is sent to the device and displayed to the user, allowing the user to check the suggested menu.

[0848] Input: User profile, existing ingredient information, retail store inventory information, menu request

[0849] Output: The optimal menu is generated and sent to the device.

[0850] Step 5: Order ingredients online

[0851] Based on the proposed menu, users order the ingredients they need online through the application. The order information is sent from the device to a server. The server then forwards this information to the local retailer's system, which receives the order and prepares the ingredients. The user can then pick up the ingredients at the designated store or use a delivery service.

[0852] Input: Order Information

[0853] Output: Order information sent to local retailer, ingredients received by user

[0854] Step 6: Stockpile management and expiration date notification

[0855] The server periodically scans the database and checks the expiration dates of registered ingredients. When the expiration date approaches, it sends a notification to the user. This notification is displayed on the device. This allows the user to consume the ingredients appropriately based on the expiration date set.

[0856] Input: Existing ingredient information

[0857] Output: Expiration date notification

[0858] Step 7: Emotion Recognition

[0859] The device collects user input and emotions through sensors such as cameras and microphones. The collected emotion data is sent from the device to a server. This includes facial expression recognition, voice recognition, and input pattern analysis.

[0860] Input: User's facial expressions, voice, and input data

[0861] Output: Emotion data sent to the server

[0862] Step 8: Send and process emotion data

[0863] The server analyzes the received emotional data using an emotion engine to identify the user's emotional state. The analysis results are stored on the server and used for further processing.

[0864] Input: Emotion data

[0865] Output: Analysis results

[0866] Step 9: Emotional Adjustment

[0867] The server then adjusts the menu suggestions and notification content based on the emotion data generated by the emotion engine. For example, if the user is feeling stressed, it will suggest easy-to-prepare meals. It also adjusts the tone and content of notifications about food items that are nearing their expiration date.

[0868] Input: Analysis results, menu suggestion request, expiration date information

[0869] Output: Adjusted menu suggestions, notification content

[0870] In this way, each processing step is explained in detail, and the inputs and outputs, as well as the data processing and calculations based on them, are clearly indicated, making the function and operation of the entire system clear.

[0871] (Application example 2)

[0872] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0873] Currently, there are many systems that efficiently manage ingredients and suggest menus for households, but few systems offer personalized suggestions based on the user's emotional state. Furthermore, it is also challenging to realize efficient ingredient purchasing through collaboration with local retailers. Furthermore, there is a need to support these functions on new devices such as virtual stores and smart glasses.

[0874] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a means for inputting food ingredient information for the household, a means for acquiring inventory information for local retailers, a means for proposing an optimal menu based on the food ingredient information for the household and the inventory information for the local retailers, a means for ordering the ingredients to be used in the proposed menu from the local retailers, a means for recognizing the user's emotions, and a means for adjusting the contents of the proposed menu based on the recognized user emotions. This makes it possible to efficiently manage food ingredients in the household and propose menus, while providing personalized services according to the user's emotional state and enabling efficient food purchasing through collaboration with local retailers.

[0875] The "means for inputting household food information" is an interface for inputting the items, quantities, and expiration dates of food items in the refrigerator or pantry in the home into the system.

[0876] The "means for obtaining inventory information from local retail stores" is a method for obtaining the inventory status of ingredients and products held by local retail stores by linking with the system.

[0877] The "means of proposing the optimal menu" is a process that generates the optimal menu for the user, taking into consideration household composition, allergy information, and food preferences, based on information about ingredients in the household and inventory information from local retailers.

[0878] The "means for ordering ingredients to be used in the proposed menu from the retail store" is a mechanism for ordering the necessary ingredients from a local retail store based on the menu proposed by the system.

[0879] "Means for recognizing user emotions" refers to technology that analyzes the user's facial expressions, tone of voice, input patterns, etc. to identify the user's emotional state.

[0880] The "means for adjusting the content of the suggested menu based on the recognized user's emotions" is an algorithm that changes the difficulty and content of the suggested menu depending on the recognized user's emotions.

[0881] "Means for inputting household composition, allergy information, and food preferences" refers to an interface that allows users to input information about their household composition, allergy information, and food preferences into the system.

[0882] "Means for managing expiration dates" refers to a method for tracking and managing expiration dates of food ingredients in the home.

[0883] "Means for notifying about ingredients approaching their expiration date" refers to the process by which the system identifies ingredients that are approaching their expiration date and sends a notification to the user.

[0884] The system for implementing this invention includes a wearable device such as smart glasses, a smartphone, a local server, and a database of local retailers. The operation of the entire system is configured as follows.

[0885] 1. Setting up your user profile

[0886] Users open the smartphone app and enter personal information such as household composition, allergy information, food preferences, etc. This data is sent from the device to a server and stored in a database.

[0887] 2. Input and management of ingredients

[0888] Users use smart glasses to scan ingredients in their refrigerators or pantries. The smart glasses' camera recognizes the ingredients and can also read QR codes and barcodes. This ingredient information is sent to a server and stored in a database. The server also manages expiration dates and sends notifications to users when ingredients are nearing their expiration date.

[0889] 3. Collaboration with local retailers

[0890] The server uses API integrations and data feeds to periodically retrieve inventory information from local retailers, which is then stored and updated in a database.

[0891] 4. Menu suggestions

[0892] Users can request menu suggestions using smart glasses or smartphones. The server uses AI algorithms to generate optimal menus based on user profiles, ingredient information, and local retailer inventory information. The generated menus are then displayed on the user's device.

[0893] 5. Order ingredients online

[0894] Based on the proposed menu, users can use the application to order the necessary ingredients from local retailers online. The order information is transferred to the local retailer's system via a server, and the retailer receives the order and prepares the ingredients.

[0895] 6. Emotion recognition function

[0896] The smart glasses use sensors (camera, microphone) built into them to analyze the user's facial expressions and tone of voice to recognize their emotions, and then use emotion-recognition AI to adjust the suggested menu and notification tone based on the user's emotional state.

[0897] For example, if it senses that the user is tired, it will suggest easy-to-prepare meals. The following prompts are fed into the generative AI model:

[0898] "Please suggest some simple recipes based on the user's current emotional state. Emotion recognition data is as follows: {emotion data}"

[0899] Specific examples

[0900] The user uses smart glasses in front of the refrigerator to scan "tomato," "chicken," and "pasta." The camera in the smart glasses recognizes these ingredients and sends the information to the server. When the user requests a dinner menu, the server considers the user's household composition, allergy information, and food preferences, and suggests "tomato and chicken basil pasta" based on local retailer inventory information (such as fresh basil). If the server detects that the user is tired, it prioritizes recipes with shorter cooking times.

[0901] This invention contributes to reducing food waste and the burden of housework by streamlining food management within the home and providing more personalized services based on the user's emotions.

[0902] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0903] Step 1:

[0904] Users open the smartphone app and enter their user profile, including household composition, allergy information, food preferences, etc. The entered data is sent from the device to the server, which then stores it in a database.

[0905] Step 2:

[0906] The user uses smart glasses to scan ingredients in the refrigerator or pantry. The camera in the smart glasses recognizes the ingredients and reads the QR code or barcode. This recognized ingredient information is sent to the server, which stores it in a database. During this process, information such as the ingredient type, quantity, and expiration date is processed.

[0907] Step 3:

[0908] The server periodically retrieves inventory information from local retailers via API. The retrieved inventory information is stored and updated in a database. The data used here includes product availability and price information for retailers.

[0909] Step 4:

[0910] The user requests menu suggestions using smart glasses or a smartphone. The server uses an AI algorithm to generate an optimal menu based on the user profile, existing ingredient information, and inventory information from local retailers. This generated menu is then displayed on the user's device. Menu generation involves a recipe generation process using a generative AI model.

[0911] Step 5:

[0912] Based on the proposed menu, users can order the necessary ingredients online through the application. The order information is sent from the terminal to the server, which then forwards this data to the local retailer's system. The retailer receives the order and prepares the ingredients.

[0913] Step 6:

[0914] The server uses sensors (camera, microphone) built into the smart glasses to analyze the user's facial expressions and tone of voice to recognize their emotions. This emotional data is then analyzed by emotion recognition AI to determine the user's current emotional state.

[0915] Step 7:

[0916] Based on the recognized user's emotional data, the server generates a prompt for the AI ​​model, saying, "Please suggest a simple dish based on the user's current emotional state." This prompt is input into the AI ​​model, which then generates an optimal menu based on the user's emotional state.

[0917] Step 8:

[0918] The server sends the generated menu to the user's device. For example, if the server detects that the user is tired, it will suggest a menu that is easy to prepare. For example, it might suggest "easy basil pasta with tomatoes and chicken."

[0919] Step 9:

[0920] The server manages expiration dates for food items in the home and sends notifications to the user about food items that are nearing their expiration date. The tone and content of the notifications are adjusted according to the user's emotional state.

[0921] The above processing steps will improve the efficiency of home food management and menu suggestions, and provide personalized services according to the user's emotional state. It will also enable efficient food purchasing through collaboration with local retailers.

[0922] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0923] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0924] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0925] [Third embodiment]

[0926] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0927] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0928] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0929] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0930] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0931] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0932] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0933] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0934] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0935] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0936] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0937] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0938] This invention is a system that automates food ingredient management and menu suggestions for the home, and also enables efficient food purchasing by linking with local retailers. This system integrates information on home ingredients and inventory information from local retailers, suggests optimal menus for users, and supports them in ordering ingredients.

[0939] System configuration

[0940] The system consists of a user terminal, a server, a database of local retailers, and a notification means.

[0941] System Operation

[0942] 1. Setting up your user profile

[0943] Users open the application on their smartphones, tablets, or other devices and enter their family members, allergy information, and food preferences. This data is then sent from the device to a server and stored in a database.

[0944] 2. Enter and update existing ingredients

[0945] Users input the ingredients (item, quantity, expiration date) in their refrigerator or pantry into the app, and this information is sent from the device to the server and stored in a database.

[0946] 3. Collaboration with local retailers

[0947] The server periodically retrieves inventory information from local retailers, often automatically through an API, and stores and updates the database.

[0948] 4. Menu suggestions

[0949] Users request menu suggestions through the application, and the server uses AI algorithms to generate optimal menus based on the user profile, existing ingredient information, and local retailer inventory information. These menu suggestions are then sent to the device and displayed to the user.

[0950] 5. Order ingredients online

[0951] Based on the proposed menu, users can order the necessary ingredients online through the application. The order information is sent from the device to a server, which then forwards it to the local retailer's system. The retailer receives the order and prepares the ingredients.

[0952] 6. Stockpile management and expiration date notification

[0953] The server periodically scans the database to identify ingredients that are nearing their expiration date, and users are notified when the expiration date approaches, which is displayed on their device, allowing them to consume the ingredients at the appropriate time.

[0954] Specific examples

[0955] A family has tomatoes, chicken, and pasta in the refrigerator. The user opens the app and enters these ingredients. The family also has a fish allergy. Meanwhile, the local supermarket has a large stock of fresh basil.

[0956] When a user requests "Suggest a dinner menu," the server will suggest a menu based on the following information:

[0957] Homemade tomatoes, chicken, and pasta

[0958] Having a fish allergy

[0959] Fresh basil from the supermarket

[0960] Based on this, the server will suggest "Tomato and Chicken Basil Pasta" and encourage the purchase of basil. Also, if there is milk nearing its expiration date, the user will receive a notification on their device saying, "The milk's expiration date is coming up in three days."

[0961] In this way, this system provides food management within the home and efficient menu suggestions, contributing to reducing food waste and easing the burden of housework.

[0962] The processing flow will be explained below.

[0963] Program processing flow

[0964] 1. Setting up your user profile

[0965] Step 1: Enter your profile information

[0966] Users open the application on their smartphone or tablet and enter information about their household composition, allergies, and food preferences.

[0967] Step 2: Submit your input

[0968] The terminal transmits the input profile information to the server.

[0969] Step 3: Save your information

[0970] The server stores the received profile information in a database.

[0971] 2. Enter and update existing ingredients

[0972] Step 1: Enter ingredient information

[0973] Users enter the ingredients (item, quantity, expiration date) in their refrigerator or pantry into the app.

[0974] Step 2: Submit your input

[0975] The terminal transmits the input ingredient information to the server.

[0976] Step 3: Save the ingredients

[0977] The server stores the received ingredient information in a database.

[0978] 3. Collaboration with local retailers

[0979] Step 1: Get inventory information

[0980] The server periodically retrieves inventory information from local retailers via API or data feed.

[0981] Step 2: Save inventory information

[0982] The server stores the acquired inventory information in a database.

[0983] Step 3: View inventory information

[0984] The terminal displays real-time inventory information for users from local retailers.

[0985] 4. Menu suggestions

[0986] Step 1: Request a menu suggestion

[0987] The user requests meal suggestions through the application.

[0988] Step 2: Get the data

[0989] The server retrieves user profiles, existing ingredient information, and local retailer inventory information from a database.

[0990] Step 3: Generate a menu

[0991] The server uses AI algorithms to generate optimal menus based on the data it obtains.

[0992] Step 4: Submit and view your menu

[0993] The server sends the generated menu to the terminal.

[0994] The device displays the sent menu to the user.

[0995] 5. Order ingredients online

[0996] Step 1: Create an order

[0997] Users can order the necessary ingredients online based on the suggested menu.

[0998] Step 2: Submit your order

[0999] The terminal transmits the order details to the server.

[1000] Step 3: Transfer your order

[1001] The server forwards the order to the local retailer's system.

[1002] Step 4: Processing the order

[1003] Local retailers receive orders and prepare the ingredients.

[1004] 6. Stockpile management and expiration date notification

[1005] Step 1: Check the expiration date

[1006] The server periodically scans the database to identify ingredients that are nearing their expiration date.

[1007] Step 2: Generate notification content

[1008] The server creates a notification about food items that are nearing their expiration date.

[1009] Step 3: Send notifications

[1010] The server sends the generated notification content to the terminal.

[1011] Step 4: View notifications

[1012] The device will display expiration date notifications to the user.

[1013] Example 1

[1014] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1015] In modern households, managing and purchasing ingredients, as well as creating menus, requires a lot of time and effort, which is a major challenge. People often overlook expiration dates on ingredients, resulting in food waste. Furthermore, there is a lack of systems that efficiently link household ingredients with inventory information from local retailers, making it difficult to efficiently purchase the ingredients needed. This increases the burden of daily housework and ultimately leads to a decline in quality of life.

[1016] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1017] In this invention, the server includes a means for inputting household food ingredient information, a means for periodically obtaining inventory information from local retailers via an API, a means for proposing optimal menus using a generative AI model based on the household food ingredient information and the local retailer's inventory information, and a means for online ordering of ingredients used in the proposed menu from the retailer. This automates household food ingredient management and menu proposals, enabling efficient food purchasing. Furthermore, by managing and notifying expiration dates, food waste can be reduced, reducing the burden of housework and improving quality of life.

[1018] "Household food ingredient information" is detailed information such as the type, quantity, and expiration date of food ingredients used in the home.

[1019] "Local retailers" refer to stores that sell food and daily necessities in a specific area.

[1020] "Inventory information" refers to information about the types, quantities, prices, and availability of products available at local retail stores.

[1021] "API" stands for Application Program Interface, a means by which different software applications communicate with each other.

[1022] A "generative AI model" is an algorithm or system that uses artificial intelligence techniques to analyze data and generate new information or suggestions.

[1023] A "menu" refers to the combination of foods and dishes for a particular meal, usually suggested for breakfast, lunch, and dinner.

[1024] "Online ordering" refers to the act of purchasing goods over the Internet, particularly in an e-commerce environment.

[1025] "Best before date" indicates the last date that a food product can be safely consumed and still retain its quality.

[1026] A "notification" is a message or signal intended to inform a user of certain information or events.

[1027] This invention is a system that allows efficient management of home ingredients, menu planning, and online ordering of ingredients. To implement this system, a user terminal, a server, a database of local retailers, and a notification means are required.

[1028] First, users open the application on their smartphone, tablet, or other device and enter information about their family, allergies, and food preferences. This information is then sent from the device to a server and stored in a database.

[1029] Next, users enter the items, quantities, and expiration dates of ingredients in their refrigerator or pantry into the app, which then sends this information from the device to a server and stores it in a database.

[1030] The server periodically retrieves inventory information from local retailers via API and stores it in a database. This process is automated, ensuring that inventory information is always up-to-date.

[1031] When a user requests a menu suggestion through the app, the server uses a generative AI model to generate an optimal menu based on the user profile, existing ingredient information, and local retailer inventory information. This suggestion is then sent to the device and displayed to the user.

[1032] The user orders the ingredients needed based on the proposed menu online. The order information is sent from the terminal to a server, which then forwards it to the local retailer's system. The retailer receives the order and prepares the ingredients.

[1033] The server periodically scans the database to identify ingredients that are nearing their expiration date, and users are notified when the expiration date approaches, which is displayed on their device, allowing them to consume the ingredients at the appropriate time.

[1034] As a concrete example, suppose a household has tomatoes, chicken, and pasta in the refrigerator, and the family consists of two adults, one child, and a person with a fish allergy. If a user from this household, who has fresh basil in stock at a local retailer, requests "Suggest a dinner menu," the server generates a menu for "Tomato and Chicken Basil Pasta" based on the following information and encourages the user to purchase basil. If there is milk nearing its expiration date, a notification stating "The milk's expiration date is in three days" will be displayed on the user's device.

[1035] An example prompt might look like this:

[1036] "If a user opens a food management app, inputs the tomatoes, chicken, and pasta in their refrigerator, and indicates that a family member has a fish allergy, and requests the app to suggest a dinner menu, the server will suggest a tomato, chicken, and basil pasta dish and encourage them to purchase fresh basil. Also, if there is milk nearing its expiration date, a notification will appear on the user's device stating, 'Your milk will expire in three days.'"

[1037] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1038] Step 1:

[1039] The user opens the application on a device such as a smartphone or tablet. First, the user profile setting screen is displayed. The user enters the family composition (e.g., two adults, one child), allergy information (e.g., fish allergy), and food preferences (e.g., prefers Italian food). The device converts this input information into JSON format and sends it to the server. The server receives this JSON data and stores it in the user profile table in the database. In this step, the input data is private information such as family composition and allergy information, and the output data means that the user profile is saved in the database.

[1040] Step 2:

[1041] The user enters the ingredients (e.g., tomatoes, chicken, pasta), quantity (e.g., 3 tomatoes, 2 chicken pieces), and expiration date (e.g., 2023-10-10) of ingredients in the household on the app's ingredient management screen. The device acquires this information, converts it back into JSON format, and sends it to the server. The server receives this JSON data, saves it in the ingredient table of the database, and compares it with existing information to update it. The input data for this step are detailed information about each ingredient, and the output data is the updated information for the ingredient table.

[1042] Step 3:

[1043] The server runs a scheduled job every day at 3 AM, which sends a request to the local retailer's API endpoint to retrieve inventory information. The retrieved inventory information is returned to the server in JSON format. The server receives this information and updates an inventory table in the database. The input data for this step is the retailer's inventory information, and the output data is the updated inventory table contents.

[1044] Step 4:

[1045] The user taps the "Suggest a Meal" button in the application. The device sends this request to the server. Based on the received request, the server queries the user profile, existing ingredient information, and inventory information from local retailers, and uses an AI algorithm to generate an optimal meal plan. The generated meal plan is sent to the device in JSON format. The device parses this data and displays the suggested meal plan to the user. The input data for this step are the user request and related ingredient information, and the output data is the suggested meal plan.

[1046] Step 5:

[1047] Based on the proposed menu, the user uses the application to order the ingredients they need online. For example, they tap the "Buy basil" button. The device sends this purchase request to the server. The server then forwards the request to the local retailer's order system. The local retailer confirms the order and prepares the ingredients. The input data for this step is the user's purchase request, and the output data indicates that the order has been accepted and preparation has begun.

[1048] Step 6:

[1049] The server runs a scheduled job every day at 2 PM that scans the ingredient table in the database to identify ingredients that are close to their expiration date. The server generates a notification for ingredients with an expiration date within three days and sends it to the user's device. The device displays this notification to the user as a push notification. The input data for this step is the expiration date information in the ingredient table, and the output data is the notification information displayed to the user.

[1050] Through these steps, we can create a system that efficiently manages household ingredients, proposes menus, and orders ingredients online. This system will be very useful in reducing food waste and easing the burden of housework.

[1051] (Application example 1)

[1052] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1053] Modern households often spend a great deal of time and effort managing ingredients and planning daily meals, creating a need for efficient ingredient use and reduced food waste. Finding optimal ways to purchase ingredients and quickly supply them is also a challenge. Effective menu suggestions tailored to each household's needs and management of expiration dates are also desirable. To solve these problems, a system is needed that integrates information about ingredients within the home, inventory information from local retailers, and food delivery services, and provides unified management, suggestions, ordering, and delivery.

[1054] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1055] In this invention, the server includes a means for inputting household ingredient information, a means for acquiring inventory information from local retailers, a means for proposing an optimal menu based on the household ingredient information and the retailer's inventory information, a means for ordering ingredients for the proposed menu from the retailer, and a means for coordinating with a food delivery service to deliver the ordered ingredients to the user. This allows users to efficiently manage ingredients in their home, propose optimal menus, and quickly purchase and deliver the necessary ingredients. It also contributes to reducing food waste and easing the burden of housework.

[1056] "Household food information" refers to detailed data such as the items, quantities, and expiration dates of food and ingredients that are normally stored in the home.

[1057] "Local retail store inventory information" refers to detailed data such as the availability, prices, and expiration dates of food ingredients and products held by local retail stores.

[1058] "Menu suggestions" refers to the act of presenting the most suitable dishes and meal menus based on the user's available ingredients and individual needs (allergy information, food preferences, etc.).

[1059] "Ordering ingredients" refers to the process by which a user specifies the ingredients needed based on a suggested menu and purchases these ingredients from a retailer.

[1060] "Food delivery service" refers to a business that includes delivering ingredients ordered by users to designated locations.

[1061] "Food inventory status" refers to the current amount and expiration date of a particular food ingredient at a home or storage facility.

[1062] "Expiration date notification" refers to a function that notifies users when the expiration date of stored food ingredients is approaching.

[1063] A "generative AI model" refers to an artificial intelligence algorithm that automatically generates optimal prompts and suggestions based on data and information generated from user input.

[1064] A "prompt" is a guide or instruction generated to encourage a user to take a particular action or response.

[1065] This invention is a system that automates household food ingredient management and meal planning, and also enables efficient food purchasing and delivery through food delivery services. This system integrates information on household ingredients, inventory information from local retailers, and information from food delivery services, suggests optimal meals for users, and supports the ordering and delivery of ingredients.

[1066] System configuration

[1067] The system consists of a user terminal, a server, a database of local retailers, a food delivery service, and a notification method.

[1068] Hardware and Software Configuration

[1069] Hardware

[1070] Smartphone (iOS, Android)

[1071] Cloud services (AWS, Google Cloud, Microsoft Azure)

[1072] software

[1073] Food management application: Python (Flask)

[1074] Database: MongoDB

[1075] Integration with local store API: RESTful API

[1076] Processing Description

[1077] User profile settings

[1078] Users open the application on their smartphone and enter their family information, allergy information, and food preferences. This data is sent to the server in JSON format and stored in a database.

[1079] Entering and updating existing ingredients

[1080] Users input the ingredients (item, quantity, expiration date) in their refrigerator or pantry into the app, and this information is sent from the device to the server and stored in a database.

[1081] Collaboration with local retailers

[1082] The server periodically retrieves inventory information from local retailers via API, which is then stored and updated in a database.

[1083] Menu suggestions

[1084] When a user requests a menu suggestion, the server uses an AI algorithm to suggest the optimal menu based on the user profile, existing ingredient information, and retail store inventory information, and sends it to the device.

[1085] Online food ordering and delivery

[1086] Based on the proposed menu, users can order the necessary ingredients online through the application. The order information is transferred from the server to the local retailer's system, which receives the order, prepares the ingredients, and then works with a food delivery service to deliver the ingredients to the user.

[1087] Expiration date notification

[1088] The server periodically scans the database to identify ingredients that are nearing their expiration date, and when this happens, a notification is sent to the user and displayed on their device.

[1089] Specific examples

[1090] A family has tomatoes, chicken, and pasta in the refrigerator. The user opens the app and enters these ingredients. The family also includes a fish allergy. Meanwhile, the local supermarket has a large stock of fresh basil.

[1091] When a user requests "Suggest a dinner menu," the server will suggest a menu based on the following information:

[1092] Homemade tomatoes, chicken, and pasta

[1093] Having a fish allergy

[1094] Fresh basil from the supermarket

[1095] Based on this, the server will suggest "Tomato and Chicken Basil Pasta" and encourage the purchase of basil. Also, if there is milk nearing its expiration date, the user will receive a notification on their device saying, "The milk's expiration date is coming up in three days."

[1096] An example of a prompt sentence in this case is:

[1097] "Dear user, based on the ingredients in your fridge, I'd like to suggest 'Tomato and Chicken Basil Pasta'. You need basil, but you can buy it at your local supermarket. Would you like to order basil?"

[1098] is displayed.

[1099] In this way, this system can provide food management within the home and efficient menu suggestions, contributing to reducing food waste and easing the burden of housework.

[1100] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1101] Step 1:

[1102] User profile settings

[1103] Users open the smartphone application and input their family composition, allergy information, and food preferences. This data is sent to the server in JSON format upon input, and the server stores the received data in a database. The input data includes the number of family members, types of allergies, and preferred food ingredients. The output is accurately stored in the database.

[1104] Step 2:

[1105] Entering and updating existing ingredients

[1106] The user inputs the item, quantity, and expiration date of each ingredient in the refrigerator or pantry into the application. This data is also sent to the server in JSON format, and after the server receives it, it is stored in the database. The input data includes the ingredient name, quantity, and expiration date. The current ingredient information is reflected in the database as output.

[1107] Step 3:

[1108] Collaboration with local retailers

[1109] The server periodically retrieves inventory information from local retailers via a specified API. The retrieved data includes detailed information such as stock items, prices, and expiration dates, and is also received in JSON format and stored in a database. An API key is required to retrieve inventory information, and the latest information can be retrieved automatically by setting up requests to be sent at regular intervals. The latest inventory information is updated in the database as an output.

[1110] Step 4:

[1111] Menu suggestions

[1112] When a user requests a menu suggestion through the application, the request is sent to the server. The server uses a generative AI model to generate an optimal menu based on the user profile in the database, existing ingredient information, and local retailer inventory information. Input data includes the user's preferences and allergies, currently owned ingredients, and local retailer inventory information. The suggested menu is sent as output to the user's device.

[1113] Step 5:

[1114] Online food ordering and delivery

[1115] The user selects the ingredients they need based on the suggested menu and places an order online through the application. The order information is sent in JSON format to the server, which then forwards it to the local retailer's system. The server then notifies the retailer and coordinates with the food delivery service to deliver the ordered ingredients to the destination. Input data includes the name of the ingredients to be ordered, the quantity, and delivery address information. The output is a record of the order history in a database, and the ingredients are delivered to the user.

[1116] Step 6:

[1117] Expiration date notification

[1118] The server periodically scans the database to identify ingredients that are nearing their expiration date. When the expiration date approaches, the information is notified to the user's device. The input data is the current ingredient information and its expiration date, and a database query is used to extract ingredients that are approaching their expiration date. The output is a notification sent to the user's device, displaying the message "The expiration date for XX is coming up in 3 days."

[1119] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1120] This system automates home ingredient management and menu suggestions, and also works with local retailers to efficiently purchase ingredients. It also recognizes the user's emotions and adjusts the content of suggestions and notifications based on those emotions, providing a more personalized service.

[1121] System configuration

[1122] The system consists of a user's terminal, a server, a database of local retailers, a notification method, and an emotion engine.

[1123] System Operation

[1124] 1. Setting up your user profile

[1125] Users open the application on their smartphone or tablet and enter information about their family, allergies, and food preferences. This data is then sent from the device to a server and stored in a database.

[1126] 2. Enter and update existing ingredients

[1127] Users input the ingredients (item, quantity, expiration date) in their refrigerator or pantry into the app, and this information is sent from the device to the server and stored in a database.

[1128] 3. Collaboration with local retailers

[1129] The server periodically retrieves inventory information from local retailers via APIs and data feeds, which is then stored and updated in a database.

[1130] 4. Menu suggestions

[1131] Users request menu suggestions through the application, and the server uses AI algorithms to generate optimal menus based on the user profile, existing ingredient information, and local retailer inventory information. The generated menus are then sent to the device and displayed to the user.

[1132] 5. Order ingredients online

[1133] Based on the proposed menu, users can order the necessary ingredients online through the application. The order information is sent from the device to a server, which then forwards it to the local retailer's system. The retailer receives the order and prepares the ingredients.

[1134] 6. Stockpile management and expiration date notification

[1135] The server periodically scans the database to identify ingredients that are nearing their expiration date, and users are notified when the expiration date approaches, which is displayed on their device, allowing them to consume the ingredients at the appropriate time.

[1136] Introducing the Emotion Engine

[1137] 7. Emotion recognition

[1138] Through applications, the device collects user input and emotions using sensors such as the camera and microphone, including facial expression recognition, voice recognition, and analysis of input patterns.

[1139] 8. Transmission and Processing of Emotional Data

[1140] The device sends the collected emotional data to a server, which uses an emotion engine to analyze this data and determine the user's current emotional state.

[1141] 9. Emotion-Based Regulation

[1142] The server adjusts menu suggestions and notification content based on the emotional data recognized by the emotion engine. For example, if the user is feeling stressed, it will suggest easy-to-prepare meals. It can also adjust the tone and content of notifications about food items nearing their expiration date based on the user's emotions.

[1143] Specific examples

[1144] A family has tomatoes, chicken, and pasta in the refrigerator. The user opens the app and enters these ingredients. The family also has a fish allergy. Meanwhile, the local supermarket has a large stock of fresh basil.

[1145] When a user requests "Suggest a dinner menu," the server will suggest a menu based on the following information:

[1146] Homemade tomatoes, chicken, and pasta

[1147] Having a fish allergy

[1148] Fresh basil from the supermarket

[1149] Based on this, the server suggests "Tomato and Chicken Basil Pasta" and encourages the purchase of basil. At the same time, if the server detects stress levels from the user's facial expressions and tone of voice, it can also suggest recipes with shorter cooking times. Also, if the user has milk nearing its expiration date, the user will receive a notification on their device stating, "Your milk will expire in three days." This notification is also displayed in a tone that corresponds to the user's emotional state.

[1150] In this way, by incorporating emotion recognition functionality, this system can provide more personalized services, helping to reduce food waste and ease the burden of housework, as well as managing ingredients in the home and suggesting efficient menus.

[1151] The processing flow will be explained below.

[1152] Program processing flow

[1153] 1. Setting up your user profile

[1154] Step 1: Enter your profile information

[1155] Users open the application on their smartphone or tablet and enter information about their household composition, allergies, and food preferences.

[1156] Step 2: Submit your input

[1157] The terminal transmits the input profile information to the server.

[1158] Step 3: Save your information

[1159] The server stores the received profile information in a database.

[1160] 2. Enter and update existing ingredients

[1161] Step 1: Enter ingredient information

[1162] Users enter the ingredients (item, quantity, expiration date) in their refrigerator or pantry into the app.

[1163] Step 2: Submit your input

[1164] The terminal transmits the input ingredient information to the server.

[1165] Step 3: Save the ingredients

[1166] The server stores the received ingredient information in a database.

[1167] 3. Collaboration with local retailers

[1168] Step 1: Get inventory information

[1169] The server periodically retrieves inventory information from local retailers via APIs and data feeds.

[1170] Step 2: Save inventory information

[1171] The server stores the acquired inventory information in a database.

[1172] Step 3: View inventory information

[1173] The terminal displays real-time inventory information for users from local retailers.

[1174] 4. Menu suggestions

[1175] Step 1: Request a menu suggestion

[1176] The user requests meal suggestions through the application.

[1177] Step 2: Get the data

[1178] The server retrieves user profiles, existing ingredient information, and local retailer inventory information from a database.

[1179] Step 3: Generate a menu

[1180] The server uses AI algorithms to generate optimal menus based on the data it obtains.

[1181] Step 4: Submit and view your menu

[1182] The server sends the generated menu to the terminal.

[1183] The device displays the sent menu to the user.

[1184] 5. Order ingredients online

[1185] Step 1: Create an order

[1186] Users can order the necessary ingredients online based on the suggested menu.

[1187] Step 2: Submit your order

[1188] The terminal transmits the order details to the server.

[1189] Step 3: Transfer your order

[1190] The server forwards the order to the local retailer's system.

[1191] Step 4: Processing the order

[1192] Local retailers receive orders and prepare the ingredients.

[1193] 6. Stockpile management and expiration date notification

[1194] Step 1: Check the expiration date

[1195] The server periodically scans the database to identify ingredients that are nearing their expiration date.

[1196] Step 2: Generate notification content

[1197] The server creates a notification about food items that are nearing their expiration date.

[1198] Step 3: Send notifications

[1199] The server sends the generated notification content to the terminal.

[1200] Step 4: View notifications

[1201] The device will display expiration date notifications to the user.

[1202] 7. Introducing the Emotion Engine

[1203] Step 1: Collecting emotion data

[1204] The device collects emotional data through applications, using user input and sensors such as the camera and microphone, including facial expression recognition, voice recognition, and analysis of input patterns.

[1205] Step 2: Sending emotion data

[1206] The device transmits the collected emotion data to the server.

[1207] Step 3: Analyze the sentiment data

[1208] The server uses an emotion engine to analyze the emotion data and determine the user's current emotional state.

[1209] Step 4: Emotional Adjustment

[1210] The server adjusts the menu suggestions and notifications based on the emotion data recognized by the emotion engine. For example, if the user is feeling stressed, it will suggest simple and easy meals.

[1211] Specific examples

[1212] A family has tomatoes, chicken, and pasta in the refrigerator. The user opens the app and enters these ingredients. They also specify a "fish allergy" in their household information. Meanwhile, the local supermarket has a large stock of fresh basil.

[1213] Example processing steps:

[1214] Step 1: Request a menu

[1215] A user requests, "Suggest a dinner menu."

[1216] Step 2: Get the data

[1217] The server retrieves user profiles, existing food information, and local retailer inventory information.

[1218] Step 3: Generate a menu

[1219] The server uses an AI algorithm to suggest "tomato and chicken basil pasta," taking into account the user's emotional state (e.g., stress) and choosing recipes with short cooking times.

[1220] Step 4: View the proposal

[1221] The device displays the suggested menu to the user, and includes keywords that correspond to emotions (e.g., "easy" and "fun").

[1222] Step 5: Create an order

[1223] A user orders basil online.

[1224] Step 6: Submit your order

[1225] The terminal sends the order details to the server.

[1226] Step 7: Processing the order

[1227] The server forwards the order to a local retailer, which prepares the ingredients.

[1228] Step 8: Expiration date notification

[1229] The server identifies milk that is about to expire and creates a notification saying, "Milk will expire in 3 days."

[1230] The device will display notifications in a friendly tone based on emotion recognition.

[1231] In this way, this system achieves ingredient management, efficient menu suggestions, and even personalized responses through emotion recognition.

[1232] Example 2

[1233] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1234] In modern households, food ingredient management, menu suggestions, and necessary ingredient purchasing are often inefficient. In addition, food ingredients that are close to their expiration date are not adequately managed or utilized, resulting in food waste. Furthermore, systems ignore the user's psychological state and emotions when cooking at home, and do not provide personalized service. There is a need to provide a system that solves these problems, improves the efficiency of food ingredient management at home, and reduces the burden on users.

[1235] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a means for inputting household ingredient information, a means for acquiring inventory information from local retailers, and a means for proposing an optimal menu based on the household ingredient information and the retailer's inventory information. This enables coordination between home ingredient management and local retailers. It also includes a means for collecting user emotions and a means for adjusting the content of suggestions and notifications based on the emotions. This enables personalized menu suggestions and notifications based on the user's psychological state, making home ingredient management more efficient and user-friendly. In addition, the server includes a means for managing the inventory status and expiration dates of household stockpiles and a means for notifying the user of ingredients that are approaching their expiration dates, thereby contributing to the reduction of food waste.

[1236] "Household food information" refers to information such as the type, quantity, and expiration date of food items owned by the user.

[1237] "Local retail store inventory information" refers to information such as the product types, stock quantities, prices, and availability of products held by local retail stores.

[1238] The "optimal menu" is a menu of dishes suggested based on information about ingredients in the household, inventory information at local retailers, the user's household composition, allergy information, and food preferences.

[1239] "Home stockpiles inventory status" is information that indicates the current quantity and status of food ingredients and consumables owned by the user.

[1240] The "expiration date" is information that indicates the period during which food ingredients can be safely consumed.

[1241] "User emotion" refers to data that represents the user's psychological state or mood, including facial expressions, voice tone, input patterns, etc.

[1242] "Means for adjusting the content of suggestions and notifications based on emotions" refers to a function that analyzes a user's emotional data, generates appropriate suggestions and notifications based on the results, and provides them to the user.

[1243] MODE FOR CARRYING OUT THE INVENTION

[1244] This system automates home ingredient management and menu suggestions, and also connects with local retailers to efficiently purchase ingredients. It also recognizes the user's emotions and adjusts the content of suggestions and notifications based on those emotions, providing a more personalized service.

[1245] System configuration

[1246] The system consists of a user's terminal, a server, a database of local retailers, a notification method, and an emotion engine.

[1247] System Operation

[1248] This system uses the following hardware and software to perform specific data processing and calculations.

[1249] User devices: Devices such as smartphones and tablets running iOS or Android.

[1250] Server: Uses cloud servers that host databases and AI algorithms, including database management systems such as MySQL and AI libraries such as TensorFlow.

[1251] Emotion engine: An AI engine for analyzing user emotions, using Google Cloud Vision API, Amazon Rekognition, IBM Watson, etc.

[1252] User profile settings

[1253] Users open the application on their smartphone or tablet and enter information about their family members, allergies, and food preferences, which is then sent from the device to a server and stored in a database.

[1254] Entering and updating existing ingredients

[1255] Users input the ingredients (item, quantity, expiration date) in their refrigerator or pantry into the app. This can be done manually or by using a barcode scanner. This information is sent from the device to the server and stored in a database.

[1256] Collaboration with local retailers

[1257] The server periodically retrieves inventory information from local retailers via APIs and data feeds, which is then stored and updated in a database.

[1258] Menu suggestions

[1259] Users request menu suggestions through the application, and the server applies AI algorithms to generate optimal menus based on the user profile, existing ingredient information, and local retailer inventory information. The generated menus are then sent to the device and displayed to the user.

[1260] Online ordering of ingredients

[1261] Based on the proposed menu, users order the necessary ingredients online through the application. The order information is sent from the device to a server, which then forwards it to the local retailer's system. The retailer receives the order and prepares the ingredients. The user can then pick up the ingredients at the designated store or use a delivery service.

[1262] Stockpile management and expiration date notification

[1263] The server periodically scans the database to identify ingredients that are nearing their expiration date, and users are notified when the expiration date approaches, which is displayed on their device, allowing them to consume the ingredients at the appropriate time.

[1264] Introducing the Emotion Engine

[1265] emotion recognition

[1266] Through applications, the device collects user input and emotions using sensors such as the camera and microphone, including facial expression recognition, voice recognition, and analysis of input patterns.

[1267] Sending and processing emotional data

[1268] The device sends the collected emotional data to a server, which uses an emotion engine to analyze this data and determine the user's current emotional state.

[1269] Emotion-Based Adjustment

[1270] The server adjusts menu suggestions and notification content based on the emotional data recognized by the emotion engine. For example, if the user is feeling stressed, it will suggest easy-to-prepare meals. It can also adjust the tone and content of notifications about food items nearing their expiration date based on the user's emotions.

[1271] Examples of specific examples and prompts

[1272] A family has tomatoes, chicken, and pasta in the refrigerator. The user opens the app and enters these ingredients. The family also has a fish allergy. Meanwhile, the local supermarket has a large stock of fresh basil.

[1273] When a user requests "Suggest a dinner menu," the server will suggest a menu based on the following information:

[1274] Homemade tomatoes, chicken, and pasta

[1275] Having a fish allergy

[1276] Fresh basil from the supermarket

[1277] The server suggests "Basil Pasta with Tomatoes and Chicken" and encourages the user to purchase basil. At the same time, if the server detects stress levels from the user's facial expressions and tone of voice, it can also suggest recipes with shorter cooking times. If the user has milk nearing its expiration date, the user will receive a notification on their device stating, "Your milk will expire in three days." This notification is also displayed in a tone that reflects the user's emotional state.

[1278] Prompt Sentence Examples

[1279] "You have tomatoes, chicken, pasta in your household, and a fish allergy. Your local supermarket has fresh basil in stock. Suggest dinner options. Also, consider adjustments if the user is stressed."

[1280] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1281] Specific processing steps of the program

[1282] Step 1: Setting up a user profile

[1283] Users open the application on their smartphone or tablet and enter their family composition, allergy information, and food preferences. This data is entered and sent to the server, which stores the received data in a database. This allows for personalized menu suggestions.

[1284] Input: Family composition, allergy information, food preferences

[1285] Output: User profile information stored in the database

[1286] Step 2: Enter and update existing ingredients

[1287] Users input the food items, quantities, and expiration dates in their refrigerators or pantries into the application. The input information is sent from the device to the server, which then stores it in a database and manages it as existing food information, enabling accurate inventory management.

[1288] Input: Item, quantity, expiration date

[1289] Output: Existing ingredients stored in the database

[1290] Step 3: Partner with local retailers

[1291] The server periodically retrieves inventory information from local retailers via APIs or data feeds. This data includes items, prices, and stock quantities. The server analyzes the information received and stores it in a database. This ensures that the latest inventory information is always available in the system.

[1292] Input: Local retailer inventory information (item, price, quantity in stock)

[1293] Output: Retail store inventory information stored in a database

[1294] Step 4: Menu suggestions

[1295] Users request menu suggestions through the application. The server uses an AI algorithm to generate the optimal menu based on the user profile, existing ingredient information, and retail store inventory information. This menu information is sent to the device and displayed to the user, allowing the user to check the suggested menu.

[1296] Input: User profile, existing ingredient information, retail store inventory information, menu request

[1297] Output: The optimal menu is generated and sent to the device.

[1298] Step 5: Order ingredients online

[1299] Based on the proposed menu, users order the ingredients they need online through the application. The order information is sent from the device to a server. The server then forwards this information to the local retailer's system, which receives the order and prepares the ingredients. The user can then pick up the ingredients at the designated store or use a delivery service.

[1300] Input: Order Information

[1301] Output: Order information sent to local retailer, ingredients received by user

[1302] Step 6: Stockpile management and expiration date notification

[1303] The server periodically scans the database and checks the expiration dates of registered ingredients. When the expiration date approaches, it sends a notification to the user. This notification is displayed on the device. This allows the user to consume the ingredients appropriately based on the expiration date set.

[1304] Input: Existing ingredient information

[1305] Output: Expiration date notification

[1306] Step 7: Emotion Recognition

[1307] The device collects user input and emotions through sensors such as cameras and microphones. The collected emotion data is sent from the device to a server. This includes facial expression recognition, voice recognition, and input pattern analysis.

[1308] Input: User's facial expressions, voice, and input data

[1309] Output: Emotion data sent to the server

[1310] Step 8: Send and process emotion data

[1311] The server analyzes the received emotional data using an emotion engine to identify the user's emotional state. The analysis results are stored on the server and used for further processing.

[1312] Input: Emotion data

[1313] Output: Analysis results

[1314] Step 9: Emotional Adjustment

[1315] The server then adjusts the menu suggestions and notification content based on the emotion data generated by the emotion engine. For example, if the user is feeling stressed, it will suggest easy-to-prepare meals. It also adjusts the tone and content of notifications about food items that are nearing their expiration date.

[1316] Input: Analysis results, menu suggestion request, expiration date information

[1317] Output: Adjusted menu suggestions, notification content

[1318] In this way, each processing step is explained in detail, and the inputs and outputs, as well as the data processing and calculations based on them, are clearly indicated, making the function and operation of the entire system clear.

[1319] (Application example 2)

[1320] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1321] Currently, there are many systems that efficiently manage ingredients and suggest menus for households, but few systems offer personalized suggestions based on the user's emotional state. Furthermore, it is also challenging to realize efficient ingredient purchasing through collaboration with local retailers. Furthermore, there is a need to support these functions on new devices such as virtual stores and smart glasses.

[1322] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a means for inputting food ingredient information for the household, a means for acquiring inventory information for local retailers, a means for proposing an optimal menu based on the food ingredient information for the household and the inventory information for the local retailers, a means for ordering the ingredients to be used in the proposed menu from the local retailers, a means for recognizing the user's emotions, and a means for adjusting the contents of the proposed menu based on the recognized user emotions. This makes it possible to efficiently manage food ingredients in the household and propose menus, while providing personalized services according to the user's emotional state and enabling efficient food purchasing through collaboration with local retailers.

[1323] The "means for inputting household food information" is an interface for inputting the items, quantities, and expiration dates of food items in the refrigerator or pantry in the home into the system.

[1324] The "means for obtaining inventory information from local retail stores" is a method for obtaining the inventory status of ingredients and products held by local retail stores by linking with the system.

[1325] The "means of proposing the optimal menu" is a process that generates the optimal menu for the user, taking into consideration household composition, allergy information, and food preferences, based on information about ingredients in the household and inventory information from local retailers.

[1326] The "means for ordering ingredients to be used in the proposed menu from the retail store" is a mechanism for ordering the necessary ingredients from a local retail store based on the menu proposed by the system.

[1327] "Means for recognizing user emotions" refers to technology that analyzes the user's facial expressions, tone of voice, input patterns, etc. to identify the user's emotional state.

[1328] The "means for adjusting the content of the suggested menu based on the recognized user's emotions" is an algorithm that changes the difficulty and content of the suggested menu depending on the recognized user's emotions.

[1329] "Means for inputting household composition, allergy information, and food preferences" refers to an interface that allows users to input information about their household composition, allergy information, and food preferences into the system.

[1330] "Means for managing expiration dates" refers to a method for tracking and managing expiration dates of food ingredients in the home.

[1331] "Means for notifying about ingredients approaching their expiration date" refers to the process by which the system identifies ingredients that are approaching their expiration date and sends a notification to the user.

[1332] The system for implementing this invention includes a wearable device such as smart glasses, a smartphone, a local server, and a database of local retailers. The operation of the entire system is configured as follows.

[1333] 1. Setting up your user profile

[1334] Users open the smartphone app and enter personal information such as household composition, allergy information, food preferences, etc. This data is sent from the device to a server and stored in a database.

[1335] 2. Input and management of ingredients

[1336] Users use smart glasses to scan ingredients in their refrigerators or pantries. The smart glasses' camera recognizes the ingredients and can also read QR codes and barcodes. This ingredient information is sent to a server and stored in a database. The server also manages expiration dates and sends notifications to users when ingredients are nearing their expiration date.

[1337] 3. Collaboration with local retailers

[1338] The server uses API integrations and data feeds to periodically retrieve inventory information from local retailers, which is then stored and updated in a database.

[1339] 4. Menu suggestions

[1340] Users can request menu suggestions using smart glasses or smartphones. The server uses AI algorithms to generate optimal menus based on user profiles, ingredient information, and local retailer inventory information. The generated menus are then displayed on the user's device.

[1341] 5. Order ingredients online

[1342] Based on the proposed menu, users can use the application to order the necessary ingredients from local retailers online. The order information is transferred to the local retailer's system via a server, and the retailer receives the order and prepares the ingredients.

[1343] 6. Emotion recognition function

[1344] The smart glasses use sensors (camera, microphone) built into them to analyze the user's facial expressions and tone of voice to recognize their emotions, and then use emotion-recognition AI to adjust the suggested menu and notification tone based on the user's emotional state.

[1345] For example, if it senses that the user is tired, it will suggest easy-to-prepare meals. The following prompts are fed into the generative AI model:

[1346] "Please suggest some simple recipes based on the user's current emotional state. Emotion recognition data is as follows: {emotion data}"

[1347] Specific examples

[1348] The user uses smart glasses in front of the refrigerator to scan "tomato," "chicken," and "pasta." The camera in the smart glasses recognizes these ingredients and sends the information to the server. When the user requests a dinner menu, the server considers the user's household composition, allergy information, and food preferences, and suggests "tomato and chicken basil pasta" based on local retailer inventory information (such as fresh basil). If the server detects that the user is tired, it prioritizes recipes with shorter cooking times.

[1349] This invention contributes to reducing food waste and the burden of housework by streamlining food management within the home and providing more personalized services based on the user's emotions.

[1350] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1351] Step 1:

[1352] Users open the smartphone app and enter their user profile, including household composition, allergy information, food preferences, etc. The entered data is sent from the device to the server, which then stores it in a database.

[1353] Step 2:

[1354] The user uses smart glasses to scan ingredients in the refrigerator or pantry. The camera in the smart glasses recognizes the ingredients and reads the QR code or barcode. This recognized ingredient information is sent to the server, which stores it in a database. During this process, information such as the ingredient type, quantity, and expiration date is processed.

[1355] Step 3:

[1356] The server periodically retrieves inventory information from local retailers via API. The retrieved inventory information is stored and updated in a database. The data used here includes product availability and price information for retailers.

[1357] Step 4:

[1358] The user requests menu suggestions using smart glasses or a smartphone. The server uses an AI algorithm to generate an optimal menu based on the user profile, existing ingredient information, and inventory information from local retailers. This generated menu is then displayed on the user's device. Menu generation involves a recipe generation process using a generative AI model.

[1359] Step 5:

[1360] Based on the proposed menu, users can order the necessary ingredients online through the application. The order information is sent from the terminal to the server, which then forwards this data to the local retailer's system. The retailer receives the order and prepares the ingredients.

[1361] Step 6:

[1362] The server uses sensors (camera, microphone) built into the smart glasses to analyze the user's facial expressions and tone of voice to recognize their emotions. This emotional data is then analyzed by emotion recognition AI to determine the user's current emotional state.

[1363] Step 7:

[1364] Based on the recognized user's emotional data, the server generates a prompt for the AI ​​model, saying, "Please suggest a simple dish based on the user's current emotional state." This prompt is input into the AI ​​model, which then generates an optimal menu based on the user's emotional state.

[1365] Step 8:

[1366] The server sends the generated menu to the user's device. For example, if the server detects that the user is tired, it will suggest a menu that is easy to prepare. For example, it might suggest "easy basil pasta with tomatoes and chicken."

[1367] Step 9:

[1368] The server manages expiration dates for food items in the home and sends notifications to the user about food items that are nearing their expiration date. The tone and content of the notifications are adjusted according to the user's emotional state.

[1369] The above processing steps will improve the efficiency of home food management and menu suggestions, and provide personalized services according to the user's emotional state. It will also enable efficient food purchasing through collaboration with local retailers.

[1370] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1371] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1372] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1373] [Fourth embodiment]

[1374] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1375] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1376] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1377] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1378] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1379] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1380] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1381] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1382] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1383] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1384] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1385] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1386] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1387] This invention is a system that automates food ingredient management and menu suggestions for the home, and also enables efficient food purchasing by linking with local retailers. This system integrates information on home ingredients and inventory information from local retailers, suggests optimal menus for users, and supports them in ordering ingredients.

[1388] System configuration

[1389] The system consists of a user terminal, a server, a database of local retailers, and a notification means.

[1390] System Operation

[1391] 1. Setting up your user profile

[1392] Users open the application on their smartphones, tablets, or other devices and enter their family members, allergy information, and food preferences. This data is then sent from the device to a server and stored in a database.

[1393] 2. Enter and update existing ingredients

[1394] Users input the ingredients (item, quantity, expiration date) in their refrigerator or pantry into the app, and this information is sent from the device to the server and stored in a database.

[1395] 3. Collaboration with local retailers

[1396] The server periodically retrieves inventory information from local retailers, often automatically through an API, and stores and updates the database.

[1397] 4. Menu suggestions

[1398] Users request menu suggestions through the application, and the server uses AI algorithms to generate optimal menus based on the user profile, existing ingredient information, and local retailer inventory information. These menu suggestions are then sent to the device and displayed to the user.

[1399] 5. Order ingredients online

[1400] Based on the proposed menu, users can order the necessary ingredients online through the application. The order information is sent from the device to a server, which then forwards it to the local retailer's system. The retailer receives the order and prepares the ingredients.

[1401] 6. Stockpile management and expiration date notification

[1402] The server periodically scans the database to identify ingredients that are nearing their expiration date, and users are notified when the expiration date approaches, which is displayed on their device, allowing them to consume the ingredients at the appropriate time.

[1403] Specific examples

[1404] A family has tomatoes, chicken, and pasta in the refrigerator. The user opens the app and enters these ingredients. The family also has a fish allergy. Meanwhile, the local supermarket has a large stock of fresh basil.

[1405] When a user requests "Suggest a dinner menu," the server will suggest a menu based on the following information:

[1406] Homemade tomatoes, chicken, and pasta

[1407] Having a fish allergy

[1408] Fresh basil from the supermarket

[1409] Based on this, the server will suggest "Tomato and Chicken Basil Pasta" and encourage the purchase of basil. Also, if there is milk nearing its expiration date, the user will receive a notification on their device saying, "The milk's expiration date is coming up in three days."

[1410] In this way, this system provides food management within the home and efficient menu suggestions, contributing to reducing food waste and easing the burden of housework.

[1411] The processing flow will be explained below.

[1412] Program processing flow

[1413] 1. Setting up your user profile

[1414] Step 1: Enter your profile information

[1415] Users open the application on their smartphone or tablet and enter information about their household composition, allergies, and food preferences.

[1416] Step 2: Submit your input

[1417] The terminal transmits the input profile information to the server.

[1418] Step 3: Save your information

[1419] The server stores the received profile information in a database.

[1420] 2. Enter and update existing ingredients

[1421] Step 1: Enter ingredient information

[1422] Users enter the ingredients (item, quantity, expiration date) in their refrigerator or pantry into the app.

[1423] Step 2: Submit your input

[1424] The terminal transmits the input ingredient information to the server.

[1425] Step 3: Save the ingredients

[1426] The server stores the received ingredient information in a database.

[1427] 3. Collaboration with local retailers

[1428] Step 1: Get inventory information

[1429] The server periodically retrieves inventory information from local retailers via API or data feed.

[1430] Step 2: Save inventory information

[1431] The server stores the acquired inventory information in a database.

[1432] Step 3: View inventory information

[1433] The terminal displays real-time inventory information for users from local retailers.

[1434] 4. Menu suggestions

[1435] Step 1: Request a menu suggestion

[1436] The user requests meal suggestions through the application.

[1437] Step 2: Get the data

[1438] The server retrieves user profiles, existing ingredient information, and local retailer inventory information from a database.

[1439] Step 3: Generate a menu

[1440] The server uses AI algorithms to generate optimal menus based on the data it obtains.

[1441] Step 4: Submit and view your menu

[1442] The server sends the generated menu to the terminal.

[1443] The device displays the sent menu to the user.

[1444] 5. Order ingredients online

[1445] Step 1: Create an order

[1446] Users can order the necessary ingredients online based on the suggested menu.

[1447] Step 2: Submit your order

[1448] The terminal transmits the order details to the server.

[1449] Step 3: Transfer your order

[1450] The server forwards the order to the local retailer's system.

[1451] Step 4: Processing the order

[1452] Local retailers receive orders and prepare the ingredients.

[1453] 6. Stockpile management and expiration date notification

[1454] Step 1: Check the expiration date

[1455] The server periodically scans the database to identify ingredients that are nearing their expiration date.

[1456] Step 2: Generate notification content

[1457] The server creates a notification about food items that are nearing their expiration date.

[1458] Step 3: Send notifications

[1459] The server sends the generated notification content to the terminal.

[1460] Step 4: View notifications

[1461] The device will display expiration date notifications to the user.

[1462] Example 1

[1463] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1464] In modern households, managing and purchasing ingredients, as well as creating menus, requires a lot of time and effort, which is a major challenge. People often overlook expiration dates on ingredients, resulting in food waste. Furthermore, there is a lack of systems that efficiently link household ingredients with inventory information from local retailers, making it difficult to efficiently purchase the ingredients needed. This increases the burden of daily housework and ultimately leads to a decline in quality of life.

[1465] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1466] In this invention, the server includes a means for inputting household food ingredient information, a means for periodically obtaining inventory information from local retailers via an API, a means for proposing optimal menus using a generative AI model based on the household food ingredient information and the local retailer's inventory information, and a means for online ordering of ingredients used in the proposed menu from the retailer. This automates household food ingredient management and menu proposals, enabling efficient food purchasing. Furthermore, by managing and notifying expiration dates, food waste can be reduced, reducing the burden of housework and improving quality of life.

[1467] "Household food ingredient information" is detailed information such as the type, quantity, and expiration date of food ingredients used in the home.

[1468] "Local retailers" refer to stores that sell food and daily necessities in a specific area.

[1469] "Inventory information" refers to information about the types, quantities, prices, and availability of products available at local retail stores.

[1470] "API" stands for Application Program Interface, a means by which different software applications communicate with each other.

[1471] A "generative AI model" is an algorithm or system that uses artificial intelligence techniques to analyze data and generate new information or suggestions.

[1472] A "menu" refers to the combination of foods and dishes for a particular meal, usually suggested for breakfast, lunch, and dinner.

[1473] "Online ordering" refers to the act of purchasing goods over the Internet, particularly in an e-commerce environment.

[1474] "Best before date" indicates the last date that a food product can be safely consumed and still retain its quality.

[1475] A "notification" is a message or signal intended to inform a user of certain information or events.

[1476] This invention is a system that allows efficient management of home ingredients, menu planning, and online ordering of ingredients. To implement this system, a user terminal, a server, a database of local retailers, and a notification means are required.

[1477] First, users open the application on their smartphone, tablet, or other device and enter information about their family, allergies, and food preferences. This information is then sent from the device to a server and stored in a database.

[1478] Next, users enter the items, quantities, and expiration dates of ingredients in their refrigerator or pantry into the app, which then sends this information from the device to a server and stores it in a database.

[1479] The server periodically retrieves inventory information from local retailers via API and stores it in a database. This process is automated, ensuring that inventory information is always up-to-date.

[1480] When a user requests a menu suggestion through the app, the server uses a generative AI model to generate an optimal menu based on the user profile, existing ingredient information, and local retailer inventory information. This suggestion is then sent to the device and displayed to the user.

[1481] The user orders the ingredients needed based on the proposed menu online. The order information is sent from the terminal to a server, which then forwards it to the local retailer's system. The retailer receives the order and prepares the ingredients.

[1482] The server periodically scans the database to identify ingredients that are nearing their expiration date, and users are notified when the expiration date approaches, which is displayed on their device, allowing them to consume the ingredients at the appropriate time.

[1483] As a concrete example, suppose a household has tomatoes, chicken, and pasta in the refrigerator, and the family consists of two adults, one child, and a person with a fish allergy. If a user from this household, who has fresh basil in stock at a local retailer, requests "Suggest a dinner menu," the server generates a menu for "Tomato and Chicken Basil Pasta" based on the following information and encourages the user to purchase basil. If there is milk nearing its expiration date, a notification stating "The milk's expiration date is in three days" will be displayed on the user's device.

[1484] An example prompt might look like this:

[1485] "If a user opens a food management app, inputs the tomatoes, chicken, and pasta in their refrigerator, and indicates that a family member has a fish allergy, and requests the app to suggest a dinner menu, the server will suggest a tomato, chicken, and basil pasta dish and encourage them to purchase fresh basil. Also, if there is milk nearing its expiration date, a notification will appear on the user's device stating, 'Your milk will expire in three days.'"

[1486] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1487] Step 1:

[1488] The user opens the application on a device such as a smartphone or tablet. First, the user profile setting screen is displayed. The user enters the family composition (e.g., two adults, one child), allergy information (e.g., fish allergy), and food preferences (e.g., prefers Italian food). The device converts this input information into JSON format and sends it to the server. The server receives this JSON data and stores it in the user profile table in the database. In this step, the input data is private information such as family composition and allergy information, and the output data means that the user profile is saved in the database.

[1489] Step 2:

[1490] The user enters the ingredients (e.g., tomatoes, chicken, pasta), quantity (e.g., 3 tomatoes, 2 chicken pieces), and expiration date (e.g., 2023-10-10) of ingredients in the household on the app's ingredient management screen. The device acquires this information, converts it back into JSON format, and sends it to the server. The server receives this JSON data, saves it in the ingredient table of the database, and compares it with existing information to update it. The input data for this step are detailed information about each ingredient, and the output data is the updated information for the ingredient table.

[1491] Step 3:

[1492] The server runs a scheduled job every day at 3 AM, which sends a request to the local retailer's API endpoint to retrieve inventory information. The retrieved inventory information is returned to the server in JSON format. The server receives this information and updates an inventory table in the database. The input data for this step is the retailer's inventory information, and the output data is the updated inventory table contents.

[1493] Step 4:

[1494] The user taps the "Suggest a Meal" button in the application. The device sends this request to the server. Based on the received request, the server queries the user profile, existing ingredient information, and inventory information from local retailers, and uses an AI algorithm to generate an optimal meal plan. The generated meal plan is sent to the device in JSON format. The device parses this data and displays the suggested meal plan to the user. The input data for this step are the user request and related ingredient information, and the output data is the suggested meal plan.

[1495] Step 5:

[1496] Based on the proposed menu, the user uses the application to order the ingredients they need online. For example, they tap the "Buy basil" button. The device sends this purchase request to the server. The server then forwards the request to the local retailer's order system. The local retailer confirms the order and prepares the ingredients. The input data for this step is the user's purchase request, and the output data indicates that the order has been accepted and preparation has begun.

[1497] Step 6:

[1498] The server runs a scheduled job every day at 2 PM that scans the ingredient table in the database to identify ingredients that are close to their expiration date. The server generates a notification for ingredients with an expiration date within three days and sends it to the user's device. The device displays this notification to the user as a push notification. The input data for this step is the expiration date information in the ingredient table, and the output data is the notification information displayed to the user.

[1499] Through these steps, we can create a system that efficiently manages household ingredients, proposes menus, and orders ingredients online. This system will be very useful in reducing food waste and easing the burden of housework.

[1500] (Application example 1)

[1501] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1502] Modern households often spend a great deal of time and effort managing ingredients and planning daily meals, creating a need for efficient ingredient use and reduced food waste. Finding optimal ways to purchase ingredients and quickly supply them is also a challenge. Effective menu suggestions tailored to each household's needs and management of expiration dates are also desirable. To solve these problems, a system is needed that integrates information about ingredients within the home, inventory information from local retailers, and food delivery services, and provides unified management, suggestions, ordering, and delivery.

[1503] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1504] In this invention, the server includes a means for inputting household ingredient information, a means for acquiring inventory information from local retailers, a means for proposing an optimal menu based on the household ingredient information and the retailer's inventory information, a means for ordering ingredients for the proposed menu from the retailer, and a means for coordinating with a food delivery service to deliver the ordered ingredients to the user. This allows users to efficiently manage ingredients in their home, propose optimal menus, and quickly purchase and deliver the necessary ingredients. It also contributes to reducing food waste and easing the burden of housework.

[1505] "Household food information" refers to detailed data such as the items, quantities, and expiration dates of food and ingredients that are normally stored in the home.

[1506] "Local retail store inventory information" refers to detailed data such as the availability, prices, and expiration dates of food ingredients and products held by local retail stores.

[1507] "Menu suggestions" refers to the act of presenting the most suitable dishes and meal menus based on the user's available ingredients and individual needs (allergy information, food preferences, etc.).

[1508] "Ordering ingredients" refers to the process by which a user specifies the ingredients needed based on a suggested menu and purchases these ingredients from a retailer.

[1509] "Food delivery service" refers to a business that includes delivering ingredients ordered by users to designated locations.

[1510] "Food inventory status" refers to the current amount and expiration date of a particular food ingredient at a home or storage facility.

[1511] "Expiration date notification" refers to a function that notifies users when the expiration date of stored food ingredients is approaching.

[1512] A "generative AI model" refers to an artificial intelligence algorithm that automatically generates optimal prompts and suggestions based on data and information generated from user input.

[1513] A "prompt" is a guide or instruction generated to encourage a user to take a particular action or response.

[1514] This invention is a system that automates household food ingredient management and meal planning, and also enables efficient food purchasing and delivery through food delivery services. This system integrates information on household ingredients, inventory information from local retailers, and information from food delivery services, suggests optimal meals for users, and supports the ordering and delivery of ingredients.

[1515] System configuration

[1516] The system consists of a user terminal, a server, a database of local retailers, a food delivery service, and a notification method.

[1517] Hardware and Software Configuration

[1518] Hardware

[1519] Smartphone (iOS, Android)

[1520] Cloud services (AWS, Google Cloud, Microsoft Azure)

[1521] software

[1522] Food management application: Python (Flask)

[1523] Database: MongoDB

[1524] Integration with local store API: RESTful API

[1525] Processing Description

[1526] User profile settings

[1527] Users open the application on their smartphone and enter their family information, allergy information, and food preferences. This data is sent to the server in JSON format and stored in a database.

[1528] Entering and updating existing ingredients

[1529] Users input the ingredients (item, quantity, expiration date) in their refrigerator or pantry into the app, and this information is sent from the device to the server and stored in a database.

[1530] Collaboration with local retailers

[1531] The server periodically retrieves inventory information from local retailers via API, which is then stored and updated in a database.

[1532] Menu suggestions

[1533] When a user requests a menu suggestion, the server uses an AI algorithm to suggest the optimal menu based on the user profile, existing ingredient information, and retail store inventory information, and sends it to the device.

[1534] Online food ordering and delivery

[1535] Based on the proposed menu, users can order the necessary ingredients online through the application. The order information is transferred from the server to the local retailer's system, which receives the order, prepares the ingredients, and then works with a food delivery service to deliver the ingredients to the user.

[1536] Expiration date notification

[1537] The server periodically scans the database to identify ingredients that are nearing their expiration date, and when this happens, a notification is sent to the user and displayed on their device.

[1538] Specific examples

[1539] A family has tomatoes, chicken, and pasta in the refrigerator. The user opens the app and enters these ingredients. The family also includes a fish allergy. Meanwhile, the local supermarket has a large stock of fresh basil.

[1540] When a user requests "Suggest a dinner menu," the server will suggest a menu based on the following information:

[1541] Homemade tomatoes, chicken, and pasta

[1542] Having a fish allergy

[1543] Fresh basil from the supermarket

[1544] Based on this, the server will suggest "Tomato and Chicken Basil Pasta" and encourage the purchase of basil. Also, if there is milk nearing its expiration date, the user will receive a notification on their device saying, "The milk's expiration date is coming up in three days."

[1545] An example of a prompt sentence in this case is:

[1546] "Dear user, based on the ingredients in your fridge, I'd like to suggest 'Tomato and Chicken Basil Pasta'. You need basil, but you can buy it at your local supermarket. Would you like to order basil?"

[1547] is displayed.

[1548] In this way, this system can provide food management within the home and efficient menu suggestions, contributing to reducing food waste and easing the burden of housework.

[1549] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1550] Step 1:

[1551] User profile settings

[1552] Users open the smartphone application and input their family composition, allergy information, and food preferences. This data is sent to the server in JSON format upon input, and the server stores the received data in a database. The input data includes the number of family members, types of allergies, and preferred food ingredients. The output is accurately stored in the database.

[1553] Step 2:

[1554] Entering and updating existing ingredients

[1555] The user inputs the item, quantity, and expiration date of each ingredient in the refrigerator or pantry into the application. This data is also sent to the server in JSON format, and after the server receives it, it is stored in the database. The input data includes the ingredient name, quantity, and expiration date. The current ingredient information is reflected in the database as output.

[1556] Step 3:

[1557] Collaboration with local retailers

[1558] The server periodically retrieves inventory information from local retailers via a specified API. The retrieved data includes detailed information such as stock items, prices, and expiration dates, and is also received in JSON format and stored in a database. An API key is required to retrieve inventory information, and the latest information can be retrieved automatically by setting up requests to be sent at regular intervals. The latest inventory information is updated in the database as an output.

[1559] Step 4:

[1560] Menu suggestions

[1561] When a user requests a menu suggestion through the application, the request is sent to the server. The server uses a generative AI model to generate an optimal menu based on the user profile in the database, existing ingredient information, and local retailer inventory information. Input data includes the user's preferences and allergies, currently owned ingredients, and local retailer inventory information. The suggested menu is sent as output to the user's device.

[1562] Step 5:

[1563] Online food ordering and delivery

[1564] The user selects the ingredients they need based on the suggested menu and places an order online through the application. The order information is sent in JSON format to the server, which then forwards it to the local retailer's system. The server then notifies the retailer and coordinates with the food delivery service to deliver the ordered ingredients to the destination. Input data includes the name of the ingredients to be ordered, the quantity, and delivery address information. The output is a record of the order history in a database, and the ingredients are delivered to the user.

[1565] Step 6:

[1566] Expiration date notification

[1567] The server periodically scans the database to identify ingredients that are nearing their expiration date. When the expiration date approaches, the information is notified to the user's device. The input data is the current ingredient information and its expiration date, and a database query is used to extract ingredients that are approaching their expiration date. The output is a notification sent to the user's device, displaying the message "The expiration date for XX is coming up in 3 days."

[1568] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1569] This system automates home ingredient management and menu suggestions, and also works with local retailers to efficiently purchase ingredients. It also recognizes the user's emotions and adjusts the content of suggestions and notifications based on those emotions, providing a more personalized service.

[1570] System configuration

[1571] The system consists of a user's terminal, a server, a database of local retailers, a notification method, and an emotion engine.

[1572] System Operation

[1573] 1. Setting up your user profile

[1574] Users open the application on their smartphone or tablet and enter information about their family, allergies, and food preferences. This data is then sent from the device to a server and stored in a database.

[1575] 2. Enter and update existing ingredients

[1576] Users input the ingredients (item, quantity, expiration date) in their refrigerator or pantry into the app, and this information is sent from the device to the server and stored in a database.

[1577] 3. Collaboration with local retailers

[1578] The server periodically retrieves inventory information from local retailers via APIs and data feeds, which is then stored and updated in a database.

[1579] 4. Menu suggestions

[1580] Users request menu suggestions through the application, and the server uses AI algorithms to generate optimal menus based on the user profile, existing ingredient information, and local retailer inventory information. The generated menus are then sent to the device and displayed to the user.

[1581] 5. Order ingredients online

[1582] Based on the proposed menu, users can order the necessary ingredients online through the application. The order information is sent from the device to a server, which then forwards it to the local retailer's system. The retailer receives the order and prepares the ingredients.

[1583] 6. Stockpile management and expiration date notification

[1584] The server periodically scans the database to identify ingredients that are nearing their expiration date, and users are notified when the expiration date approaches, which is displayed on their device, allowing them to consume the ingredients at the appropriate time.

[1585] Introducing the Emotion Engine

[1586] 7. Emotion recognition

[1587] Through applications, the device collects user input and emotions using sensors such as the camera and microphone, including facial expression recognition, voice recognition, and analysis of input patterns.

[1588] 8. Transmission and Processing of Emotional Data

[1589] The device sends the collected emotional data to a server, which uses an emotion engine to analyze this data and determine the user's current emotional state.

[1590] 9. Emotion-Based Regulation

[1591] The server adjusts menu suggestions and notification content based on the emotional data recognized by the emotion engine. For example, if the user is feeling stressed, it will suggest easy-to-prepare meals. It can also adjust the tone and content of notifications about food items nearing their expiration date based on the user's emotions.

[1592] Specific examples

[1593] A family has tomatoes, chicken, and pasta in the refrigerator. The user opens the app and enters these ingredients. The family also has a fish allergy. Meanwhile, the local supermarket has a large stock of fresh basil.

[1594] When a user requests "Suggest a dinner menu," the server will suggest a menu based on the following information:

[1595] Homemade tomatoes, chicken, and pasta

[1596] Having a fish allergy

[1597] Fresh basil from the supermarket

[1598] Based on this, the server suggests "Tomato and Chicken Basil Pasta" and encourages the purchase of basil. At the same time, if the server detects stress levels from the user's facial expressions and tone of voice, it can also suggest recipes with shorter cooking times. Also, if the user has milk nearing its expiration date, the user will receive a notification on their device stating, "Your milk will expire in three days." This notification is also displayed in a tone that corresponds to the user's emotional state.

[1599] In this way, by incorporating emotion recognition functionality, this system can provide more personalized services, helping to reduce food waste and ease the burden of housework, as well as managing ingredients in the home and suggesting efficient menus.

[1600] The processing flow will be explained below.

[1601] Program processing flow

[1602] 1. Setting up your user profile

[1603] Step 1: Enter your profile information

[1604] Users open the application on their smartphone or tablet and enter information about their household composition, allergies, and food preferences.

[1605] Step 2: Submit your input

[1606] The terminal transmits the input profile information to the server.

[1607] Step 3: Save your information

[1608] The server stores the received profile information in a database.

[1609] 2. Enter and update existing ingredients

[1610] Step 1: Enter ingredient information

[1611] Users enter the ingredients (item, quantity, expiration date) in their refrigerator or pantry into the app.

[1612] Step 2: Submit your input

[1613] The terminal transmits the input ingredient information to the server.

[1614] Step 3: Save the ingredients

[1615] The server stores the received ingredient information in a database.

[1616] 3. Collaboration with local retailers

[1617] Step 1: Get inventory information

[1618] The server periodically retrieves inventory information from local retailers via APIs and data feeds.

[1619] Step 2: Save inventory information

[1620] The server stores the acquired inventory information in a database.

[1621] Step 3: View inventory information

[1622] The terminal displays real-time inventory information for users from local retailers.

[1623] 4. Menu suggestions

[1624] Step 1: Request a menu suggestion

[1625] The user requests meal suggestions through the application.

[1626] Step 2: Get the data

[1627] The server retrieves user profiles, existing ingredient information, and local retailer inventory information from a database.

[1628] Step 3: Generate a menu

[1629] The server uses AI algorithms to generate optimal menus based on the data it obtains.

[1630] Step 4: Submit and view your menu

[1631] The server sends the generated menu to the terminal.

[1632] The device displays the sent menu to the user.

[1633] 5. Order ingredients online

[1634] Step 1: Create an order

[1635] Users can order the necessary ingredients online based on the suggested menu.

[1636] Step 2: Submit your order

[1637] The terminal transmits the order details to the server.

[1638] Step 3: Transfer your order

[1639] The server forwards the order to the local retailer's system.

[1640] Step 4: Processing the order

[1641] Local retailers receive orders and prepare the ingredients.

[1642] 6. Stockpile management and expiration date notification

[1643] Step 1: Check the expiration date

[1644] The server periodically scans the database to identify ingredients that are nearing their expiration date.

[1645] Step 2: Generate notification content

[1646] The server creates a notification about food items that are nearing their expiration date.

[1647] Step 3: Send notifications

[1648] The server sends the generated notification content to the terminal.

[1649] Step 4: View notifications

[1650] The device will display expiration date notifications to the user.

[1651] 7. Introducing the Emotion Engine

[1652] Step 1: Collecting emotion data

[1653] The device collects emotional data through applications, using user input and sensors such as the camera and microphone, including facial expression recognition, voice recognition, and analysis of input patterns.

[1654] Step 2: Sending emotion data

[1655] The device transmits the collected emotion data to the server.

[1656] Step 3: Analyze the sentiment data

[1657] The server uses an emotion engine to analyze the emotion data and determine the user's current emotional state.

[1658] Step 4: Emotional Adjustment

[1659] The server adjusts the menu suggestions and notifications based on the emotion data recognized by the emotion engine. For example, if the user is feeling stressed, it will suggest simple and easy meals.

[1660] Specific examples

[1661] A family has tomatoes, chicken, and pasta in the refrigerator. The user opens the app and enters these ingredients. They also specify a "fish allergy" in their household information. Meanwhile, the local supermarket has a large stock of fresh basil.

[1662] Example processing steps:

[1663] Step 1: Request a menu

[1664] A user requests, "Suggest a dinner menu."

[1665] Step 2: Get the data

[1666] The server retrieves user profiles, existing food information, and local retailer inventory information.

[1667] Step 3: Generate a menu

[1668] The server uses an AI algorithm to suggest "tomato and chicken basil pasta," taking into account the user's emotional state (e.g., stress) and choosing recipes with short cooking times.

[1669] Step 4: View the proposal

[1670] The device displays the suggested menu to the user, and includes keywords that correspond to emotions (e.g., "easy" and "fun").

[1671] Step 5: Create an order

[1672] A user orders basil online.

[1673] Step 6: Submit your order

[1674] The terminal sends the order details to the server.

[1675] Step 7: Processing the order

[1676] The server forwards the order to a local retailer, which prepares the ingredients.

[1677] Step 8: Expiration date notification

[1678] The server identifies milk that is about to expire and creates a notification saying, "Milk will expire in 3 days."

[1679] The device will display notifications in a friendly tone based on emotion recognition.

[1680] In this way, this system achieves ingredient management, efficient menu suggestions, and even personalized responses through emotion recognition.

[1681] Example 2

[1682] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1683] In modern households, food ingredient management, menu suggestions, and necessary ingredient purchasing are often inefficient. In addition, food ingredients that are close to their expiration date are not adequately managed or utilized, resulting in food waste. Furthermore, systems ignore the user's psychological state and emotions when cooking at home, and do not provide personalized service. There is a need to provide a system that solves these problems, improves the efficiency of food ingredient management at home, and reduces the burden on users.

[1684] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a means for inputting household ingredient information, a means for acquiring inventory information from local retailers, and a means for proposing an optimal menu based on the household ingredient information and the retailer's inventory information. This enables coordination between home ingredient management and local retailers. It also includes a means for collecting user emotions and a means for adjusting the content of suggestions and notifications based on the emotions. This enables personalized menu suggestions and notifications based on the user's psychological state, making home ingredient management more efficient and user-friendly. In addition, the server includes a means for managing the inventory status and expiration dates of household stockpiles and a means for notifying the user of ingredients that are approaching their expiration dates, thereby contributing to the reduction of food waste.

[1685] "Household food information" refers to information such as the type, quantity, and expiration date of food items owned by the user.

[1686] "Local retail store inventory information" refers to information such as the product types, stock quantities, prices, and availability of products held by local retail stores.

[1687] The "optimal menu" is a menu of dishes suggested based on information about ingredients in the household, inventory information at local retailers, the user's household composition, allergy information, and food preferences.

[1688] "Home stockpiles inventory status" is information that indicates the current quantity and status of food ingredients and consumables owned by the user.

[1689] The "expiration date" is information that indicates the period during which food ingredients can be safely consumed.

[1690] "User emotion" refers to data that represents the user's psychological state or mood, including facial expressions, voice tone, input patterns, etc.

[1691] "Means for adjusting the content of suggestions and notifications based on emotions" refers to a function that analyzes a user's emotional data, generates appropriate suggestions and notifications based on the results, and provides them to the user.

[1692] MODE FOR CARRYING OUT THE INVENTION

[1693] This system automates home ingredient management and menu suggestions, and also connects with local retailers to efficiently purchase ingredients. It also recognizes the user's emotions and adjusts the content of suggestions and notifications based on those emotions, providing a more personalized service.

[1694] System configuration

[1695] The system consists of a user's terminal, a server, a database of local retailers, a notification method, and an emotion engine.

[1696] System Operation

[1697] This system uses the following hardware and software to perform specific data processing and calculations.

[1698] User devices: Devices such as smartphones and tablets running iOS or Android.

[1699] Server: Uses cloud servers that host databases and AI algorithms, including database management systems such as MySQL and AI libraries such as TensorFlow.

[1700] Emotion engine: An AI engine for analyzing user emotions, using Google Cloud Vision API, Amazon Rekognition, IBM Watson, etc.

[1701] User profile settings

[1702] Users open the application on their smartphone or tablet and enter information about their family members, allergies, and food preferences, which is then sent from the device to a server and stored in a database.

[1703] Entering and updating existing ingredients

[1704] Users input the ingredients (item, quantity, expiration date) in their refrigerator or pantry into the app. This can be done manually or by using a barcode scanner. This information is sent from the device to the server and stored in a database.

[1705] Collaboration with local retailers

[1706] The server periodically retrieves inventory information from local retailers via APIs and data feeds, which is then stored and updated in a database.

[1707] Menu suggestions

[1708] Users request menu suggestions through the application, and the server applies AI algorithms to generate optimal menus based on the user profile, existing ingredient information, and local retailer inventory information. The generated menus are then sent to the device and displayed to the user.

[1709] Online ordering of ingredients

[1710] Based on the proposed menu, users order the necessary ingredients online through the application. The order information is sent from the device to a server, which then forwards it to the local retailer's system. The retailer receives the order and prepares the ingredients. The user can then pick up the ingredients at the designated store or use a delivery service.

[1711] Stockpile management and expiration date notification

[1712] The server periodically scans the database to identify ingredients that are nearing their expiration date, and users are notified when the expiration date approaches, which is displayed on their device, allowing them to consume the ingredients at the appropriate time.

[1713] Introducing the Emotion Engine

[1714] emotion recognition

[1715] Through applications, the device collects user input and emotions using sensors such as the camera and microphone, including facial expression recognition, voice recognition, and analysis of input patterns.

[1716] Sending and processing emotional data

[1717] The device sends the collected emotional data to a server, which uses an emotion engine to analyze this data and determine the user's current emotional state.

[1718] Emotion-Based Adjustment

[1719] The server adjusts menu suggestions and notification content based on the emotional data recognized by the emotion engine. For example, if the user is feeling stressed, it will suggest easy-to-prepare meals. It can also adjust the tone and content of notifications about food items nearing their expiration date based on the user's emotions.

[1720] Examples of specific examples and prompts

[1721] A family has tomatoes, chicken, and pasta in the refrigerator. The user opens the app and enters these ingredients. The family also has a fish allergy. Meanwhile, the local supermarket has a large stock of fresh basil.

[1722] When a user requests "Suggest a dinner menu," the server will suggest a menu based on the following information:

[1723] Homemade tomatoes, chicken, and pasta

[1724] Having a fish allergy

[1725] Fresh basil from the supermarket

[1726] The server suggests "Basil Pasta with Tomatoes and Chicken" and encourages the user to purchase basil. At the same time, if the server detects stress levels from the user's facial expressions and tone of voice, it can also suggest recipes with shorter cooking times. If the user has milk nearing its expiration date, the user will receive a notification on their device stating, "Your milk will expire in three days." This notification is also displayed in a tone that reflects the user's emotional state.

[1727] Prompt Sentence Examples

[1728] "You have tomatoes, chicken, pasta in your household, and a fish allergy. Your local supermarket has fresh basil in stock. Suggest dinner options. Also, consider adjustments if the user is stressed."

[1729] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1730] Specific processing steps of the program

[1731] Step 1: Setting up a user profile

[1732] Users open the application on their smartphone or tablet and enter their family composition, allergy information, and food preferences. This data is entered and sent to the server, which stores the received data in a database. This allows for personalized menu suggestions.

[1733] Input: Family composition, allergy information, food preferences

[1734] Output: User profile information stored in the database

[1735] Step 2: Enter and update existing ingredients

[1736] Users input the food items, quantities, and expiration dates in their refrigerators or pantries into the application. The input information is sent from the device to the server, which then stores it in a database and manages it as existing food information, enabling accurate inventory management.

[1737] Input: Item, quantity, expiration date

[1738] Output: Existing ingredients stored in the database

[1739] Step 3: Partner with local retailers

[1740] The server periodically retrieves inventory information from local retailers via APIs or data feeds. This data includes items, prices, and stock quantities. The server analyzes the information received and stores it in a database. This ensures that the latest inventory information is always available in the system.

[1741] Input: Local retailer inventory information (item, price, quantity in stock)

[1742] Output: Retail store inventory information stored in a database

[1743] Step 4: Menu suggestions

[1744] Users request menu suggestions through the application. The server uses an AI algorithm to generate the optimal menu based on the user profile, existing ingredient information, and retail store inventory information. This menu information is sent to the device and displayed to the user, allowing the user to check the suggested menu.

[1745] Input: User profile, existing ingredient information, retail store inventory information, menu request

[1746] Output: The optimal menu is generated and sent to the device.

[1747] Step 5: Order ingredients online

[1748] Based on the proposed menu, users order the ingredients they need online through the application. The order information is sent from the device to a server. The server then forwards this information to the local retailer's system, which receives the order and prepares the ingredients. The user can then pick up the ingredients at the designated store or use a delivery service.

[1749] Input: Order Information

[1750] Output: Order information sent to local retailer, ingredients received by user

[1751] Step 6: Stockpile management and expiration date notification

[1752] The server periodically scans the database and checks the expiration dates of registered ingredients. When the expiration date approaches, it sends a notification to the user. This notification is displayed on the device. This allows the user to consume the ingredients appropriately based on the expiration date set.

[1753] Input: Existing ingredient information

[1754] Output: Expiration date notification

[1755] Step 7: Emotion Recognition

[1756] The device collects user input and emotions through sensors such as cameras and microphones. The collected emotion data is sent from the device to a server. This includes facial expression recognition, voice recognition, and input pattern analysis.

[1757] Input: User's facial expressions, voice, and input data

[1758] Output: Emotion data sent to the server

[1759] Step 8: Send and process emotion data

[1760] The server analyzes the received emotional data using an emotion engine to identify the user's emotional state. The analysis results are stored on the server and used for further processing.

[1761] Input: Emotion data

[1762] Output: Analysis results

[1763] Step 9: Emotional Adjustment

[1764] The server then adjusts the menu suggestions and notification content based on the emotion data generated by the emotion engine. For example, if the user is feeling stressed, it will suggest easy-to-prepare meals. It also adjusts the tone and content of notifications about food items that are nearing their expiration date.

[1765] Input: Analysis results, menu suggestion request, expiration date information

[1766] Output: Adjusted menu suggestions, notification content

[1767] In this way, each processing step is explained in detail, and the inputs and outputs, as well as the data processing and calculations based on them, are clearly indicated, making the function and operation of the entire system clear.

[1768] (Application example 2)

[1769] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1770] Currently, there are many systems that efficiently manage ingredients and suggest menus for households, but few systems offer personalized suggestions based on the user's emotional state. Furthermore, it is also challenging to realize efficient ingredient purchasing through collaboration with local retailers. Furthermore, there is a need to support these functions on new devices such as virtual stores and smart glasses.

[1771] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a means for inputting food ingredient information for the household, a means for acquiring inventory information for local retailers, a means for proposing an optimal menu based on the food ingredient information for the household and the inventory information for the local retailers, a means for ordering the ingredients to be used in the proposed menu from the local retailers, a means for recognizing the user's emotions, and a means for adjusting the contents of the proposed menu based on the recognized user emotions. This makes it possible to efficiently manage food ingredients in the household and propose menus, while providing personalized services according to the user's emotional state and enabling efficient food purchasing through collaboration with local retailers.

[1772] The "means for inputting household food information" is an interface for inputting the items, quantities, and expiration dates of food items in the refrigerator or pantry in the home into the system.

[1773] The "means for obtaining inventory information from local retail stores" is a method for obtaining the inventory status of ingredients and products held by local retail stores by linking with the system.

[1774] The "means of proposing the optimal menu" is a process that generates the optimal menu for the user, taking into consideration household composition, allergy information, and food preferences, based on information about ingredients in the household and inventory information from local retailers.

[1775] The "means for ordering ingredients to be used in the proposed menu from the retail store" is a mechanism for ordering the necessary ingredients from a local retail store based on the menu proposed by the system.

[1776] "Means for recognizing user emotions" refers to technology that analyzes the user's facial expressions, tone of voice, input patterns, etc. to identify the user's emotional state.

[1777] The "means for adjusting the content of the suggested menu based on the recognized user's emotions" is an algorithm that changes the difficulty and content of the suggested menu depending on the recognized user's emotions.

[1778] "Means for inputting household composition, allergy information, and food preferences" refers to an interface that allows users to input information about their household composition, allergy information, and food preferences into the system.

[1779] "Means for managing expiration dates" refers to a method for tracking and managing expiration dates of food ingredients in the home.

[1780] "Means for notifying about ingredients approaching their expiration date" refers to the process by which the system identifies ingredients that are approaching their expiration date and sends a notification to the user.

[1781] The system for implementing this invention includes a wearable device such as smart glasses, a smartphone, a local server, and a database of local retailers. The operation of the entire system is configured as follows.

[1782] 1. Setting up your user profile

[1783] Users open the smartphone app and enter personal information such as household composition, allergy information, food preferences, etc. This data is sent from the device to a server and stored in a database.

[1784] 2. Input and management of ingredients

[1785] Users use smart glasses to scan ingredients in their refrigerators or pantries. The smart glasses' camera recognizes the ingredients and can also read QR codes and barcodes. This ingredient information is sent to a server and stored in a database. The server also manages expiration dates and sends notifications to users when ingredients are nearing their expiration date.

[1786] 3. Collaboration with local retailers

[1787] The server uses API integrations and data feeds to periodically retrieve inventory information from local retailers, which is then stored and updated in a database.

[1788] 4. Menu suggestions

[1789] Users can request menu suggestions using smart glasses or smartphones. The server uses AI algorithms to generate optimal menus based on user profiles, ingredient information, and local retailer inventory information. The generated menus are then displayed on the user's device.

[1790] 5. Order ingredients online

[1791] Based on the proposed menu, users can use the application to order the necessary ingredients from local retailers online. The order information is transferred to the local retailer's system via a server, and the retailer receives the order and prepares the ingredients.

[1792] 6. Emotion recognition function

[1793] The smart glasses use sensors (camera, microphone) built into them to analyze the user's facial expressions and tone of voice to recognize their emotions, and then use emotion-recognition AI to adjust the suggested menu and notification tone based on the user's emotional state.

[1794] For example, if it senses that the user is tired, it will suggest easy-to-prepare meals. The following prompts are fed into the generative AI model:

[1795] "Please suggest some simple recipes based on the user's current emotional state. Emotion recognition data is as follows: {emotion data}"

[1796] Specific examples

[1797] The user uses smart glasses in front of the refrigerator to scan "tomato," "chicken," and "pasta." The camera in the smart glasses recognizes these ingredients and sends the information to the server. When the user requests a dinner menu, the server considers the user's household composition, allergy information, and food preferences, and suggests "tomato and chicken basil pasta" based on local retailer inventory information (such as fresh basil). If the server detects that the user is tired, it prioritizes recipes with shorter cooking times.

[1798] This invention contributes to reducing food waste and the burden of housework by streamlining food management within the home and providing more personalized services based on the user's emotions.

[1799] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1800] Step 1:

[1801] Users open the smartphone app and enter their user profile, including household composition, allergy information, food preferences, etc. The entered data is sent from the device to the server, which then stores it in a database.

[1802] Step 2:

[1803] The user uses smart glasses to scan ingredients in the refrigerator or pantry. The camera in the smart glasses recognizes the ingredients and reads the QR code or barcode. This recognized ingredient information is sent to the server, which stores it in a database. During this process, information such as the ingredient type, quantity, and expiration date is processed.

[1804] Step 3:

[1805] The server periodically retrieves inventory information from local retailers via API. The retrieved inventory information is stored and updated in a database. The data used here includes product availability and price information for retailers.

[1806] Step 4:

[1807] The user requests menu suggestions using smart glasses or a smartphone. The server uses an AI algorithm to generate an optimal menu based on the user profile, existing ingredient information, and inventory information from local retailers. This generated menu is then displayed on the user's device. Menu generation involves a recipe generation process using a generative AI model.

[1808] Step 5:

[1809] Based on the proposed menu, users can order the necessary ingredients online through the application. The order information is sent from the terminal to the server, which then forwards this data to the local retailer's system. The retailer receives the order and prepares the ingredients.

[1810] Step 6:

[1811] The server uses sensors (camera, microphone) built into the smart glasses to analyze the user's facial expressions and tone of voice to recognize their emotions. This emotional data is then analyzed by emotion recognition AI to determine the user's current emotional state.

[1812] Step 7:

[1813] Based on the recognized user's emotional data, the server generates a prompt for the AI ​​model, saying, "Please suggest a simple dish based on the user's current emotional state." This prompt is input into the AI ​​model, which then generates an optimal menu based on the user's emotional state.

[1814] Step 8:

[1815] The server sends the generated menu to the user's device. For example, if the server detects that the user is tired, it will suggest a menu that is easy to prepare. For example, it might suggest "easy basil pasta with tomatoes and chicken."

[1816] Step 9:

[1817] The server manages expiration dates for food items in the home and sends notifications to the user about food items that are nearing their expiration date. The tone and content of the notifications are adjusted according to the user's emotional state.

[1818] The above processing steps will improve the efficiency of home food management and menu suggestions, and provide personalized services according to the user's emotional state. It will also enable efficient food purchasing through collaboration with local retailers.

[1819] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1820] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1821] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1822] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1823] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1824] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1825] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1826] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, motorcycles, and other devices, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1827] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1828] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1829] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1830] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1831] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1832] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1833] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1834] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1835] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1836] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1837] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1838] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1839] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1840] The following is further disclosed regarding the above embodiment.

[1841] (Claim 1)

[1842] A means for inputting household food information;

[1843] a means of obtaining local retailer inventory information;

[1844] a means for proposing an optimal menu based on the food ingredient information of the household and inventory information of the retail store;

[1845] A means for ordering ingredients to be used in the proposed menu from the retail store;

[1846] A system including:

[1847] (Claim 2)

[1848] A means of managing the inventory and expiration dates of household supplies;

[1849] a means for notifying the user of food ingredients whose expiration date is approaching;

[1850] The system of claim 1 further comprising:

[1851] (Claim 3)

[1852] A means for users to input their household composition, allergy information, and food preferences;

[1853] A means for proposing the optimal menu based on the family structure, allergy information, and food preferences;

[1854] The system of claim 1 further comprising:

[1855] (Claim 4)

[1856] A way to regularly obtain inventory information from supermarkets and greengrocers,

[1857] A means for identifying ingredients to be sold with priority based on the inventory information;

[1858] A means for suggesting the use of the specified ingredients;

[1859] The system of claim 1 further comprising:

[1860] (Claim 5)

[1861] a means for inputting information about ingredients in the household using a mobile terminal such as a smartphone or a tablet, displaying inventory information of the retail store and the proposed menu, and ordering the ingredients;

[1862] 10. The system of claim 1, comprising:

[1863] "Example 1"

[1864] (Claim 1)

[1865] A means for inputting household food information;

[1866] A means to periodically obtain inventory information from local retailers via API,

[1867] A means for proposing an optimal menu using a generation AI model based on the food ingredient information of the household and inventory information of the local retail store;

[1868] A means for placing an online order for ingredients to be used in the proposed menu from the retail store;

[1869] A system including:

[1870] (Claim 2)

[1871] A way to manage the inventory and expiration dates of household supplies and periodically scan them to identify food items that are nearing their expiration date;

[1872] means for generating and transmitting a notification to a user regarding the foodstuffs that are nearing their expiration date;

[1873] The system of claim 1 further comprising:

[1874] (Claim 3)

[1875] A means for users to input their household composition, allergy information, and food preferences;

[1876] A means for proposing the optimal menu based on the family structure, allergy information, and food preferences;

[1877] The system of claim 1 further comprising:

[1878] "Application Example 1"

[1879] (Claim 1)

[1880] A means for inputting household food information;

[1881] a means of obtaining local retailer inventory information;

[1882] a means for proposing an optimal menu based on the food ingredient information of the household and inventory information of the retail store;

[1883] A means for ordering ingredients to be used in the proposed menu from the retail store;

[1884] a means for coordinating with a food delivery service to deliver the ordered ingredients to the user;

[1885] A system including:

[1886] (Claim 2)

[1887] A means of managing the inventory and expiration dates of household supplies;

[1888] a means for notifying the user of food ingredients whose expiration date is approaching;

[1889] a means for generating a purchase prompt based on the proposed menu content using a generative AI model;

[1890] The system of claim 1 further comprising:

[1891] (Claim 3)

[1892] A means for users to input their household composition, allergy information, and food preferences;

[1893] A means for proposing the optimal menu based on the family structure, allergy information, and food preferences;

[1894] The system of claim 1 further comprising:

[1895] "Example 2: Combining Emotion Engines"

[1896] (Claim 1)

[1897] A means for inputting household food information;

[1898] a means of obtaining local retailer inventory information;

[1899] a means for proposing an optimal menu based on the food ingredient information of the household and inventory information of the retail store;

[1900] A means for ordering ingredients to be used in the proposed menu from the retail store;

[1901] A means of collecting user sentiment;

[1902] means for adjusting the content of suggestions and notifications based on said emotions;

[1903] A system including:

[1904] (Claim 2)

[1905] A means of managing the inventory and expiration dates of household supplies;

[1906] a means for notifying the user of food ingredients whose expiration date is approaching;

[1907] 10. The system of claim 1.

[1908] (Claim 3)

[1909] A means for users to input their household composition, allergy information, and food preferences;

[1910] A means for proposing the optimal menu based on the family structure, allergy information, and food preferences;

[1911] 10. The system of claim 1.

[1912] "Application example 2 when combining emotion engines"

[1913] (Claim 1)

[1914] A means for inputting household food information;

[1915] a means of obtaining local retailer inventory information;

[1916] a means for proposing an optimal menu based on the food ingredient information of the household and inventory information of the retail store;

[1917] A means for ordering ingredients to be used in the proposed menu from the retail store;

[1918] a means of recognizing a user's emotions;

[1919] means for adjusting the content of the suggested menu based on the recognized user sentiment;

[1920] A system including:

[1921] (Claim 2)

[1922] A means of managing the inventory and expiration dates of household supplies;

[1923] a means for notifying the user of food ingredients whose expiration date is approaching;

[1924] The system of claim 1 further comprising:

[1925] (Claim 3)

[1926] A means for users to input their household composition, allergy information, and food preferences;

[1927] A means for proposing the optimal menu based on the family structure, allergy information, and food preferences;

[1928] The system of claim 1 further comprising: [Explanation of symbols]

[1929] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for inputting household food information; a means of obtaining local retailer inventory information; a means for proposing an optimal menu based on the food ingredient information of the household and inventory information of the retail store; A means for ordering ingredients to be used in the proposed menu from the retail store; A system including:

2. A means of managing the inventory and expiration dates of household supplies; a means for notifying the user of food ingredients whose expiration date is approaching; The system of claim 1 further comprising:

3. A means for users to input their household composition, allergy information, and food preferences; A means for proposing the optimal menu based on the family structure, allergy information, and food preferences; The system of claim 1 further comprising:

4. A way to regularly obtain inventory information from supermarkets and greengrocers, A means for identifying ingredients to be sold with priority based on the inventory information; A means for suggesting the use of the specified ingredients; The system of claim 1 further comprising:

5. a means for inputting information about ingredients in the household using a mobile terminal such as a smartphone or a tablet, displaying inventory information of the retail store and the proposed menu, and ordering the ingredients; The system of claim 1 , comprising:

Images