system

Abstract

We provide the system. [Solution] A means for collecting flyer information from online retail stores, A means of receiving inventory information inside the refrigerator, A means of obtaining food ingredient purchase information based on logistics lead time, A means of receiving weather forecast information, Means of obtaining information on family structure and event dates, A system that analyzes this information and provides optimal recipe and ingredient purchasing advice.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In modern households, preparing daily meals takes a great deal of time and effort. In addition, there is a lack of means to streamline inventory management in the refrigerator and food purchase plans, often resulting in food waste. Furthermore, it is difficult for oneself to select the optimal recipe or propose a dining-out plan while being chased by work and other household activities. In such a situation, there is a demand for a system that allows users to more easily enjoy meal preparation, food shopping, and dining out.

Means for Solving the Problems

[0005] The present invention solves the above problems by providing a system that provides optimal recipes and ingredient purchase advice by analyzing information such as means for collecting flyer information from online retail stores, means for receiving inventory information in refrigerators, means for obtaining ingredient purchase information based on logistics lead times, means for receiving weather forecast information, and means for obtaining family composition and event date information. Furthermore, by including means for providing recipe videos, means for collecting and analyzing takeout and dining-out information to suggest dining out, means for providing paid options such as shopping assistance services, housekeeping services, restaurant reservations, and related product introductions, and means for collecting and analyzing user feedback to improve the accuracy of the system's suggestions, the invention enables users to efficiently prepare meals, purchase ingredients, and dine out.

[0006] "Online retail store flyer information" refers to information about discounts, special offers, and product lists from retail stores that are provided via the internet.

[0007] "Refrigerator inventory information" refers to information about food and products currently stored in the refrigerator, collected using IoT (Internet of Things) technology.

[0008] "Logistics lead time" refers to the time it takes from ordering ingredients or products to delivery, as well as related schedule information.

[0009] "Ingredient purchase information" refers to information such as the source, price, availability, and delivery lead time of ingredients that users need to purchase for cooking or consumption.

[0010] "Weather forecast information" refers to predicted weather conditions for a specified region and time.

[0011] "Family structure" refers to the number of people living together in the same household and their attribute information (age, gender, health status, etc.).

[0012] "Event Date Information" refers to information about special dates set by the user (such as birthdays, anniversaries, holidays, etc.).

[0013] "Recipe" refers to information indicating the procedures and necessary ingredients for making a specific dish.

[0014] "Ingredient Purchase Advice" refers to suggestions and advice for the user to purchase the ingredients necessary for cooking efficiently and economically.

[0015] "Recipe Video" refers to information visually presenting the cooking procedures and techniques as videos.

[0016] "Takeout Information" refers to information about takeout dishes and products provided by restaurants, etc.

[0017] "Dining Out Information" refers to information about meals and menus provided by restaurants, cafes, and other dining establishments.

[0018] "Shopping Proxy Service" refers to a service that purchases ingredients and products on behalf of the user and delivers them to a designated location.

[0019] "Housework Proxy Service" refers to a service that performs housework such as cleaning, laundry, and cooking based on the user's request.

[0020] "Restaurant Reservation" refers to the procedure for securing a seat at a dining establishment at a specific date and time.

[0021] "Related Product Introduction" refers to information recommending related products and services based on the user's purchase history and preferences.

[0022] "Feedback" refers to information providing evaluations and opinions on the services and systems provided to the user.

Brief Description of Drawings

[0023] [Figure 1]It is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] It is a conceptual diagram showing an example of the main functions of a data processing device and a smart device according to the first embodiment. [Figure 3] It is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] It is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which a plurality of emotions are mapped. [Figure 10] It shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when the emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when the emotion engine is combined.

BEST MODE FOR CARRYING OUT THE INVENTION

[0024] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0025] First, let's explain the terminology used in the following explanation.

[0026] In the following embodiments, the signed processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Furthermore, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include CPU (Central Processing Unit), GPU (Graphics Processing Unit), GPGPU (General-Purpose computing on Graphics Processing Units), and APU (Accelerated Processing Unit).

[0027] In the following embodiments, signed RAM (Random Access Memory) is a memory that temporarily stores information and is used as work memory by the processor.

[0028] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0029] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0030] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0031] [First Embodiment]

[0032] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0033] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0034] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0035] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0036] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0037] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0038] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0039] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0040] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0041] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0042] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0043] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0044] To carry out this invention, the following system and its processing procedure will be described.

[0045] Overall system configuration

[0046] This system primarily consists of three main components: the server, the terminal, and the user. The server plays a central role in data collection, analysis, and proposal development, while the terminal provides information and interacts with the user. The user is the system's user, providing input and feedback.

[0047] Server Processing

[0048] Data collection

[0049] The server uses the internet to collect the following information:

[0050] 1. Obtain flyer information from online retail stores. This ensures that discount and special offer information is always up-to-date.

[0051] 2. The system periodically receives inventory information from the IoT refrigerator. This information is used to keep track of the user's food inventory.

[0052] 3. Obtain inventory information and logistics lead times from partner online supermarkets and grocery delivery sites. This will allow us to understand which groceries are available for purchase and their delivery schedules.

[0053] 4. Obtain the latest weather information from the weather forecast API.

[0054] 5. Collect takeout information and menu information from partner restaurants and food service providers.

[0055] 6. Obtain information about family structure and event dates (such as birthdays and anniversaries) that the user has entered in advance.

[0056] Data Analysis

[0057] The server performs multiple data analyses based on the collected information.

[0058] 1. Compare the inventory information obtained from the refrigerator with the information from online retail store flyers to create a list of seasonal ingredients.

[0059] 2. Select the optimal recipe by considering parameters such as weather forecasts, family composition, and event dates.

[0060] 3. Compare the ingredients required for the selected recipe with the ingredients the user has in their refrigerator and list any missing ingredients.

[0061] 4. Select the best source (online supermarket or e-commerce site) for any missing ingredients and calculate logistics lead times and pricing information.

[0062] 5. Analyze available takeout and dining-in options and suggest them to users.

[0063] Providing information

[0064] The server sends the analysis results to the terminal. Specifically, it provides the following information:

[0065] 1. Video link to the recommended recipe.

[0066] 2. Required ingredients and information on where to purchase them (price, availability, delivery lead time).

[0067] 3. Information suggesting takeout and dining-out options.

[0068] Terminal processing

[0069] User Interface

[0070] The device (smartphone or tablet) is primarily responsible for user interaction.

[0071] 1. Notify users of recipe video links and ingredient purchase information received from the server.

[0072] 2. Receive user input (selection of optional services and feedback) and send it to the server.

[0073] Optional Services

[0074] If a user selects a paid optional service (such as shopping assistance, housekeeping assistance, or restaurant reservations), the terminal forwards the request to the server. The server processes the request and sends it to the appropriate service provider.

[0075] User actions

[0076] Information entry

[0077] Users input information such as family composition and event dates into the system. They also update food inventory information via the IoT refrigerator.

[0078] Acceptance of proposals and feedback

[0079] Users review and implement recipe suggestions, ingredient purchase information, and dining options received from the system. They then provide feedback on the services they used.

[0080] Specific example: When preparing dinner for the family

[0081] 1. The user enters into the system that their child's birthday is approaching. The system also updates the inventory information in the refrigerator.

[0082] 2. The server suggests party recipes based on the inventory information in the refrigerator, a list of seasonal ingredients, and the weather forecast.

[0083] 3. The server lists the missing ingredients and provides information on the best place to purchase them.

[0084] 4. The device notifies the user of the suggested recipe video link and purchase information.

[0085] 5. The user accepts the system's suggestions and uses the shopping assistance service as needed.

[0086] 6. The server processes the request and instructs a partner grocery delivery service to purchase the ingredients.

[0087] This invention allows users to efficiently perform a series of tasks, from purchasing ingredients to cooking and eating out, and as a result, significantly reduce the effort and time required for daily meal preparation.

[0088] The following describes the processing flow.

[0089] Step 1: Enter user information

[0090] Users input their family structure, food preferences, and event dates into the system. They also connect to an IoT refrigerator to synchronize current food inventory information.

[0091] Step 2: Data Collection (Refrigerator)

[0092] The terminal retrieves current inventory information from the IoT refrigerator and sends it to the server. The information about the food items in the refrigerator includes the type of food, quantity, and expiration date.

[0093] Step 3: Data Collection (Weather Forecast)

[0094] The server uses a weather forecast API to collect the latest weather information. This weather information includes local temperature, probability of precipitation, and humidity.

[0095] Step 4: Data Collection (Market Information)

[0096] The server collects flyer information, inventory status, and price information from online retail stores and online supermarkets via the internet.

[0097] Step 5: Data Collection (Restaurant and Takeout Information)

[0098] The server collects information from partner restaurants and food service providers. This includes menu items, prices, takeout availability, and reservation status.

[0099] Step 6: Data Analysis (Recipe Selection)

[0100] The server analyzes the collected data and generates optimal recipe suggestions based on the user's preferences, event dates, and weather forecasts. For example, it might suggest a warm stew recipe on a rainy day and a barbecue recipe on a sunny day.

[0101] Step 7: Check inventory and list any missing ingredients.

[0102] The server compares the list of ingredients required for the recipe with the inventory information in the refrigerator and creates a list of missing ingredients.

[0103] Step 8: Selecting a supplier

[0104] The server searches for the best store to purchase the missing ingredients. It compares prices, inventory, and delivery lead times from online retail stores and online supermarkets to determine the most efficient source of supplies.

[0105] Step 9: Information Provision

[0106] The device will notify the user's smartphone or tablet of the following information:

[0107] Recommended recipe video link

[0108] A list of necessary ingredients and information on where to buy them (price, availability, delivery lead time)

[0109] Takeout and dining-out suggestions

[0110] Step 10: Selecting Optional Services

[0111] Users select the option they want to use from paid services such as shopping assistance, housekeeping assistance, and restaurant reservations, and send a request to the server via their device.

[0112] Step 11: Service Request

[0113] The server sends a request to a partner service provider based on the user's request. For grocery shopping, it provides a list of selected groceries from various suppliers. For housekeeping services, it sends the specific tasks to be done and the desired date and time. For restaurant reservations, it processes the reservation based on the user's desired date and time and the restaurant's information.

[0114] Step 12: Gathering Feedback

[0115] The device collects user feedback and sends it to the server. Users enter their ratings and opinions on the provided recipes and services.

[0116] Step 13: Analyzing Feedback and Learning

[0117] The server analyzes the collected feedback and uses it to improve the accuracy of the system's suggestions. The feedback data is then reflected in future recipe and service suggestions.

[0118] This specific processing flow will allow users to efficiently manage everything from meal preparation and grocery shopping to dining out.

[0119] (Example 1)

[0120] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0121] In today's busy lifestyle, efficiently managing and purchasing groceries, as well as preparing meals, is a significant burden for many. In particular, there is a need for a system that centrally manages and provides information such as grocery inventory, optimal recipes, weather forecasts, and takeout options, but such a comprehensive system does not yet exist. As a result, users spend a great deal of time and effort on this, making it difficult to live an efficient daily life.

[0122] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0123] In this invention, the server includes means for collecting flyer information from online retail stores, means for receiving inventory information in the refrigerator, means for obtaining food purchase information based on logistics lead times, means for receiving weather forecast information, means for obtaining family composition and event date information, means for analyzing this information and providing optimal recipes and food purchase advice, means for providing video links to recommended recipes, means for collecting takeout information and menu information from affiliated restaurants and food service providers, user interface means for receiving input from the user and transmitting it to the server, and means for creating a list of missing ingredients and calculating prices and inventory information for the best suppliers. This enables the user to efficiently manage and purchase ingredients, cook, and choose where to eat out.

[0124] An "online retail store" is a website or application established for the purpose of selling goods online.

[0125] "Flyer information" refers to digital promotional materials used in retail stores, including information on special offers, discounts, and campaigns.

[0126] "Refrigerator inventory information" refers to data showing the types, quantities, and expiration dates of food items currently stored in a household refrigerator.

[0127] "Logistics lead time" refers to the time it takes from the time an order is placed until the product is actually delivered.

[0128] "Weather forecast information" refers to data that predicts future weather conditions, including temperature, probability of precipitation, and wind speed.

[0129] "Family structure" refers to information indicating the number of people in the user's household, their age distribution, and their relationships.

[0130] "Event date information" refers to information that users have entered, such as birthdays, anniversaries, and dates of special events.

[0131] The "optimal recipe" is a list of cooking steps and ingredients that best suit the user's needs and circumstances, based on the collected data.

[0132] "Food purchase advice" refers to recommendations regarding how to purchase the ingredients a user needs, the best places to buy them, and their prices.

[0133] The "recommended recipe video link" is a link to access video content related to the selected recipe.

[0134] "Partner restaurants and food service providers" refer to restaurants and food service providers that collaborate with the system to share information and provide services.

[0135] "Takeout information" refers to the take-out menus offered by restaurants and related details.

[0136] "Menu information" refers to a list of dishes and drinks offered by a restaurant, along with detailed information about them.

[0137] "User interface means" refers to the means by which a user interacts with a system, and includes devices such as smartphones and tablets.

[0138] The "list of missing ingredients" is a list of ingredients needed for the selected recipe that are not currently in the refrigerator.

[0139] The "optimal supplier" refers to a place to purchase food ingredients that has been evaluated based on criteria such as price, stock availability, and logistics lead time.

[0140] Modes for carrying out the invention

[0141] To implement this invention, three main entities are primarily used: a server, a terminal, and a user. The server plays a central role in data collection, analysis, and proposal, while the terminal provides information and interacts with the user. The user is the system user and the entity that provides input and feedback.

[0142] Server Processing

[0143] Data collection

[0144] The server collects the following information via the internet:

[0145] 1. Obtaining flyer information from online retail stores.

[0146] The server uses scraping technology to periodically access the websites of online retail stores and automatically retrieve the latest flyer information. This ensures that discount and special offer information is always up-to-date.

[0147] 2. Receiving inventory information from the IoT refrigerator.

[0148] The server periodically receives inventory information from IoT refrigerators installed in the user's home. MQTT and HTTP are used as communication protocols. This allows the server to monitor the food inventory inside the refrigerator.

[0149] 3. Obtain inventory information and logistics lead times from online supermarkets and food delivery websites.

[0150] The server calls APIs from partner online supermarkets and grocery delivery sites to obtain the latest inventory information and logistics lead times.

[0151] 4. Obtaining weather information from a weather forecast API

[0152] The server periodically calls the weather forecast API to retrieve the latest weather information. Authentication is performed using the API key.

[0153] 5. Obtaining takeout information and menu information from partner restaurants and food service providers.

[0154] The server calls APIs from partner restaurants and food service providers to retrieve the latest takeout and menu information.

[0155] 6. Retrieving family structure and event date information entered by the user.

[0156] Users enter information such as family composition and event dates (birthdays, anniversaries, etc.) through the application's input form. This information is sent to the server and stored in the database.

[0157] Data Analysis

[0158] Based on the information collected above, the server performs the following data analysis:

[0159] 1. Create a list of seasonal ingredients.

[0160] The server compares the retrieved inventory information from the refrigerator with online retail store flyers to create a list of seasonal ingredients. This process uses the Python Pandas library.

[0161] 2. Selecting the optimal recipe

[0162] The server selects the optimal recipe by considering parameters such as weather forecasts, family composition, and event dates. This recipe selection is performed using AI models (e.g., TENSORFLOW® or PyTorch).

[0163] 3. List of missing ingredients

[0164] The system compares the ingredients required for the selected recipe with the user's refrigerator inventory and lists any missing ingredients. The Numpy library is used for the comparison process.

[0165] 4. Selecting the optimal supplier and calculating price and logistics lead time.

[0166] For any missing ingredients, the server selects the optimal supplier and calculates the total cost based on price information and logistics lead time. Linear programming may be used.

[0167] 5. Analysis of takeout and dining-in options

[0168] The system also analyzes available takeout and dining-in options and suggests them to the user. Suggestions are generated based on past user preference data.

[0169] Providing information

[0170] The server sends the analysis results to the terminal. Specifically, it provides the following information:

[0171] 1. Recommended recipe video link

[0172] The server generates a video link for the selected recipe and sends it to the device. The user can view the video by clicking the link.

[0173] 2. Required ingredients and where to buy them

[0174] The server compiles the necessary ingredients and their supplier information (price, availability, delivery lead time) and sends it to the terminal.

[0175] 3. Information on takeout and dining out options.

[0176] Based on the analysis results, the server sends suggested takeout and dining-in options to the terminal.

[0177] Terminal processing

[0178] User Interface

[0179] The device (smartphone or tablet) is responsible for user interaction:

[0180] 1. Sending notifications

[0181] The system notifies users of recipe video links and ingredient purchase information received from the server. Notification methods include push notifications and in-app notifications.

[0182] 2. Receiving and sending user input

[0183] The terminal receives input from the user (for example, selection of optional services or feedback) and sends it to the server.

[0184] Provision of optional services

[0185] If a user selects a paid optional service (such as shopping assistance, housekeeping assistance, or restaurant reservations), the terminal forwards the request to the server. The server processes the request and sends it to the appropriate service provider.

[0186] User actions

[0187] Information entry

[0188] Users input information about their family structure and event dates (e.g., birthdays and anniversaries) into the system. They can also update their food inventory information via the IoT refrigerator.

[0189] Acceptance of proposals and feedback

[0190] Users review and implement recipe suggestions, ingredient purchase information, and dining options received from the system. They then provide feedback on the services they used.

[0191] Specific example: When preparing dinner for the family

[0192] 1. The user enters into the system that their child's birthday is approaching. The system also updates the inventory information in the refrigerator.

[0193] 2. The server suggests party recipes based on the inventory information in the refrigerator, a list of seasonal ingredients, and the weather forecast.

[0194] 3. The server lists the missing ingredients and provides information on the best place to purchase them.

[0195] 4. The device notifies the user of the suggested recipe video link and purchase information.

[0196] 5. The user accepts the system's suggestions and uses the shopping assistance service as needed.

[0197] 6. The server processes the request and instructs a partner grocery delivery service to purchase the ingredients.

[0198] Using this specific example, users can efficiently manage and purchase ingredients, cook, and even choose restaurants, completing a series of tasks. As a result, the time and effort involved in daily meal preparation can be significantly reduced.

[0199] Examples of prompts to input into a generative AI model are as follows:

[0200] "My child's birthday is approaching. Please update the ingredients in the refrigerator and suggest party recipes based on online retail stores and weather forecasts. Based on the suggestions, please list any missing ingredients and provide information on the best places to buy them (price, availability, delivery lead time)."

[0201] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0202] Step 1: Data Collection

[0203] The server collects necessary information from multiple sources. Specifically, it accesses the websites of online retail stores and uses scraping techniques to obtain the latest flyer information. This flyer information includes data on discounts and special offers.

[0204] Input: URL of a retail store's website on the internet

[0205] Processing: Use scraping techniques to obtain flyer information and save it to a database.

[0206] Output: Update of flyer information database

[0207] Specifically, it uses Python libraries such as BeautifulSoup and Scrapy to analyze the HTML structure of a web page and extract the necessary information.

[0208] Step 2: Obtain inventory information from the refrigerator

[0209] The server periodically receives inventory information from IoT refrigerators installed in the user's home. MQTT and HTTP are used as communication protocols.

[0210] Input: Inventory information sent from IoT refrigerator

[0211] Processing: Analyze inventory information and save it to the database.

[0212] Output: Refrigerator inventory database update

[0213] Specifically, the IoT refrigerator sends inventory information to the cloud at regular intervals, and a server retrieves and analyzes this data.

[0214] Step 3: Obtain logistics lead time and inventory information.

[0215] The server calls APIs from partner online supermarkets and grocery delivery sites to obtain the latest inventory information and logistics lead times.

[0216] Input: API key and endpoint for online supermarkets and grocery delivery sites.

[0217] Processing: Make an API call and save the retrieved inventory information and logistics lead time to the database.

[0218] Output: Update of inventory and logistics lead time databases

[0219] Specifically, the process involves making HTTPS requests using a RESTful API, parsing the data returned in JSON format, and saving it.

[0220] Step 4: Gathering weather forecast information

[0221] The server periodically calls the weather forecast API to retrieve the latest weather information. Authentication is performed using the API key.

[0222] Input: API key and endpoint for the weather forecast API.

[0223] Process: Make an API call and save the retrieved weather information to the database.

[0224] Output: Weather forecast database update

[0225] Specifically, the process involves making an HTTPS request using an API client and then parsing the returned JSON data.

[0226] Step 5: Gathering information on partner restaurants

[0227] The server calls APIs from partner restaurants and food service providers to retrieve the latest takeout and menu information.

[0228] Input: API key and endpoint for partner restaurants and food service providers.

[0229] Process: Make an API call and save the retrieved takeout information and menu information to the database.

[0230] Output: Updates to the takeout and menu databases.

[0231] In terms of specific operations, it will similarly make HTTPS requests using a RESTful API, parse the data, and save it.

[0232] Step 6: Obtaining User Information

[0233] Users enter information such as family composition and event dates (birthdays, anniversaries, etc.) through the application's input form. This information is sent to the server and stored in the database.

[0234] Input: Family composition and event date information entered by the user.

[0235] Processing: Parse the input information and save it to the database.

[0236] Output: User information database update

[0237] In terms of specific operation, a POST request is sent from the frontend input form to the server, and the data is received and analyzed on the backend.

[0238] Step 7: Data Analysis and Recipe Selection

[0239] The server performs data analysis based on the collected information and selects the optimal recipe that matches the user's conditions. It uses AI models (e.g., TensorFlow, PyTorch).

[0240] Input: Refrigerator inventory information, flyer information, weather forecast, family composition, event date information

[0241] Processing: Use an AI model to select the optimal recipe.

[0242] Output: Selected recipes

[0243] In terms of specific operations, data is input into the AI ​​model, and the optimal recipe is obtained as a result of the model's prediction.

[0244] Step 8: List the missing ingredients

[0245] The server compares the ingredients required for the selected recipe with the ingredients currently in the refrigerator and lists any missing ingredients.

[0246] Input: Selected recipe, refrigerator inventory information

[0247] Processing: Use the Numpy library to list the missing ingredients.

[0248] Output: List of missing ingredients

[0249] Specifically, the process involves using a data frame to compare required ingredients with current inventory and extracting any missing ingredients.

[0250] Step 9: Selecting the best supplier and calculating the price.

[0251] The server selects the best supplier for any missing ingredients and calculates the total cost based on price information and logistics lead time.

[0252] Input: List of missing ingredients, online supermarket inventory information, price information, logistics lead time

[0253] Process: Use linear programming to calculate the optimal supplier and total cost.

[0254] Output: Optimal purchasing information and total cost

[0255] Specifically, the program performs linear programming using the Scipy library.

[0256] Step 10: Provide a link to a recommended recipe video.

[0257] The server generates links to video content related to the selected recipe and sends them to the device.

[0258] Input: Selected recipe

[0259] Processing: Generate video link and notify device.

[0260] Output: Recommended recipe video link notification

[0261] Specifically, the system retrieves video links related to the recipe from the database and notifies the user.

[0262] Step 11: Notifying and providing feedback to the user

[0263] The device notifies the user based on information received from the server. It also receives feedback from the user.

[0264] Input: Notification information from the server, user feedback

[0265] Processing: Sending notifications, receiving feedback and forwarding it to the server.

[0266] Output: User notification and feedback sent to the server

[0267] Specifically, the system sends push notifications via a mobile application and receives user feedback.

[0268] Step 12: Requesting and processing optional services

[0269] The server processes user requests for paid optional services and forwards the requests to the appropriate service providers.

[0270] Input: User's optional service request

[0271] Processing: Processing the request and forwarding the request to the service provider.

[0272] Output: Request to service provider

[0273] In terms of specific operation, the API is used to send a corresponding request to the service provider.

[0274] (Application Example 1)

[0275] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0276] In today's busy lifestyle, meal preparation has become a significant burden for users. In particular, keeping track of what's in the refrigerator, planning appropriate recipes, purchasing necessary ingredients, and finding restaurant or delivery options when time is limited are all cumbersome tasks. In this situation, there is a need for a system that centrally manages scattered information and provides optimal suggestions to help users efficiently plan everything from purchasing ingredients to cooking and dining out.

[0277] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0278] In this invention, the server includes means for collecting leaflet information of retail stores on the Internet, means for receiving inventory information in the refrigerator, means for obtaining food purchase information based on the logistics lead time, means for receiving weather forecast information, means for obtaining family composition and event date information, means for analyzing these information and providing an optimal recipe and food purchase advice, means for proposing an optimal recipe based on the food ingredients in the user's refrigerator and the weather forecast, means for listing the missing food ingredients and providing optimal purchase destination information, and means for providing menu information that can be ordered online from nearby restaurants. As a result, the user can efficiently manage and execute the process from food purchase to cooking and dining-out planning in a unified manner.

[0279] The "leaflet information of retail stores on the Internet" refers to special sale information and discount information provided by retail stores accessible through the Internet.

[0280] The "inventory information in the refrigerator" refers to information regarding the types and quantities of food ingredients and products stored in the refrigerator.

[0281] The "logistics lead time" refers to the time from order placement to delivery completion.

[0282] The "weather forecast information" refers to predicted information on weather conditions and includes data such as temperature, precipitation probability, wind speed, etc.

[0283] The "family composition" refers to information such as the number of members and relationships in the user's family.

[0284] The "event date information" refers to information regarding special days for the user, such as birthdays and anniversaries.

[0285] The "optimal recipe" refers to the method of cooking optimized for the user's food inventory and individual conditions.

[0286] The "food purchase advice" refers to advice for the user to efficiently purchase the necessary food ingredients.

[0287] "Food in the user's refrigerator" refers to food stored in the refrigerator owned by the user.

[0288] "Missing ingredients" refers to ingredients that are necessary to make the suggested recipe, but are not present in the refrigerator.

[0289] "Optimal purchasing information" refers to information about stores and online services where you can obtain the necessary ingredients in the most efficient and economical way.

[0290] "Menu information available for online ordering from nearby restaurants" refers to information about menus that can be ordered online from restaurants near the user's place of residence.

[0291] "Method of providing recipe videos" refers to a method of distributing videos showing how to prepare a dish to users.

[0292] "Food delivery information" refers to information about services that deliver meals to your home or a designated location.

[0293] "Takeout information" refers to information about menus and options that users can pick up at the store.

[0294] This document describes the system configuration and processing procedures necessary to implement this invention. This system mainly consists of three main components: a server, a terminal, and a user.

[0295] Overall system configuration

[0296] server

[0297] The server is responsible for collecting and analyzing the following data.

[0298] We collect flyer information from online retail stores via the internet.

[0299] Receive inventory information from an IoT refrigerator.

[0300] Obtain logistics lead time and inventory information from partnered online supermarkets and food ingredient e-commerce sites.

[0301] Obtain the latest weather information from the weather forecast API.

[0302] Collect takeout information and menu information from partnered restaurants and food delivery services.

[0303] Obtain information on family composition and event days (such as birthdays and anniversaries) previously entered by the user.

[0304] Terminal

[0305] The terminal (mainly smartphones and tablets) provides information to the user and interacts with the user.

[0306] Notify the user of the recipe video link and food ingredient purchase information received from the server.

[0307] Receive input from the user (selection of optional services and feedback) and send it to the server.

[0308] User

[0309] The user uses the system to streamline daily food ingredient purchases and cooking.

[0310] Enter information such as family composition and event days in advance.

[0311] Check the recipes, purchase advice, and dining options proposed by the system and execute them as necessary.

[0312] Provide feedback.

[0313] Explanation of the technologies and processes used

[0314] Data collection

[0315] The server uses Python to collect weather forecasts, online flyer information, and IoT refrigerator inventory information via APIs. The Requests library is one of the libraries used.

[0316] Data Analysis

[0317] The server uses Python scripts to analyze data stored in a MySQL® database. Specifically, it creates a list of seasonal ingredients by comparing inventory information in the refrigerator with flyers from online retail stores, and then selects the most suitable recipe considering the weather forecast, family size, and event dates.

[0318] Providing information

[0319] The results analyzed on the server are provided to the user through the smartphone application "Smart Cook." This application is developed using React Native and notifies users of recipe video links and ingredient purchase information through its user interface.

[0320] A concrete example of its use would be: if you have tomatoes and cheese in your refrigerator but are missing other ingredients, the system will check the weather forecast and, if sunny weather is expected to continue, suggest a salad recipe using tomatoes and cheese, and prompt you to order the missing ingredients from an online supermarket. If you're short on time, it will also suggest salad delivery from a nearby restaurant.

[0321] Example of a prompt

[0322] A user is planning a family BBQ this weekend. They have chicken and vegetables in the refrigerator, and the weather forecast is sunny. Based on seasonal ingredients and other data, please suggest the best recipe, where to buy any missing ingredients, and a suitable restaurant delivery menu for a BBQ.

[0323] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0324] Step 1:

[0325] The server uses a means of collecting flyer information from online retail stores to obtain special sale and discount information via an API. The input data is flyer information obtained from online retail stores, and the output data is the latest flyer information stored on the server. This flyer information is stored in a database.

[0326] Step 2:

[0327] The server receives inventory information from the IoT refrigerator through a connection with the refrigerator itself. The input data consists of the types and quantities of food items in the refrigerator, while the output data is the refrigerator inventory information stored on the server. The refrigerator inventory information is updated in the database in real time.

[0328] Step 3:

[0329] The server retrieves logistics lead time and inventory information from partner online supermarkets and grocery delivery sites. Input data consists of inventory status and delivery lead time information from these sites, while output data is this information stored on the server. This allows the server to understand which groceries are available for purchase and their delivery schedules.

[0330] Step 4:

[0331] The server uses a weather forecast API to obtain the latest weather information. The input data is weather information from the weather forecast API, and the output data is weather forecast information stored on the server. This weather information is also stored in a database and used for analysis.

[0332] Step 5:

[0333] The server collects takeout and menu information from partner restaurants and food service providers. Input data consists of menus and takeout information from restaurants, while output data is this information stored on the server. This information is also updated in the database in real time.

[0334] Step 6:

[0335] The server retrieves information about family structure and event dates (such as birthdays and anniversaries) that the user has entered in advance. The input data consists of family structure and event information from the user, and the output data is this information stored on the server. This information is stored in the database as a user profile.

[0336] Step 7:

[0337] The server performs data analysis based on the collected information. Input data includes flyer information, refrigerator inventory information, logistics lead times, weather forecasts, takeout information, family composition, and event information. Output data includes optimal recipes and ingredient purchasing advice. The server analyzes this data to generate, for example, a list of seasonal ingredients and recipes tailored to the weather.

[0338] Step 8:

[0339] The server sends the analysis results to the terminal and notifies the user. The input data consists of recipes and ingredient purchase information analyzed by the server, and the output data consists of notification messages sent to the terminal. The terminal has an interface that provides the user with recipe video links and ingredient purchase information, and displays this in real time.

[0340] Step 9:

[0341] The user reviews and inputs recipes, shopping advice, and dining options suggested by the system. The input data consists of the user's selected recipes and service options, while the output data is the execution request sent to the server. Based on this, services such as grocery shopping or delivery are executed.

[0342] Step 10:

[0343] The server receives user feedback and analyzes it to improve service quality. Input data consists of user feedback, while output data provides insights for service improvement and informs future suggestions. The feedback is stored in a database and used for future recipe suggestions and service improvements.

[0344] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0345] To implement this invention, the following system and its processing procedure are described. This system provides more accurate recipe and ingredient purchasing advice by combining it with an emotion engine that recognizes the user's emotions.

[0346] Overall system configuration

[0347] This system primarily consists of three main components: the server, the terminal, and the user. The server plays a central role in data collection, analysis, and proposals, while the terminal provides information and interacts with the user. The user is the system's user, providing input and feedback. Furthermore, this system incorporates an emotion engine.

[0348] Server Processing

[0349] Data collection

[0350] The server uses the internet to collect the following information:

[0351] 1. Obtain flyer information from online retail stores. This ensures that discount and special offer information is always up-to-date.

[0352] 2. The system periodically receives inventory information from the IoT refrigerator. This information is used to keep track of the user's food inventory.

[0353] 3. Obtain inventory information and logistics lead times from partner online supermarkets and grocery delivery sites. This will allow us to understand which groceries are available for purchase and their delivery schedules.

[0354] 4. Obtain the latest weather information from the weather forecast API.

[0355] 5. Collect takeout information and menu information from partner restaurants and food service providers.

[0356] 6. Obtain information about family structure and event dates (such as birthdays and anniversaries) that the user has entered in advance.

[0357] Data Analysis

[0358] The server performs multiple data analyses based on the collected information.

[0359] 1. Compare the inventory information obtained from the refrigerator with the information from online retail store flyers to create a list of seasonal ingredients.

[0360] 2. The system selects the optimal recipe by considering factors such as the weather forecast, family structure, event dates, and the user's emotional state as determined by the emotion engine. For example, if the user is feeling stressed, the system will suggest a recipe using ingredients that promote relaxation.

[0361] 3. Compare the ingredients required for the selected recipe with the ingredients the user has in their refrigerator and list any missing ingredients.

[0362] 4. Select the best source (online supermarket or e-commerce site) for any missing ingredients and calculate logistics lead times and pricing information.

[0363] 5. Analyze available takeout and dining-in options and suggest them to users.

[0364] Providing information

[0365] The server sends the analysis results to the terminal. Specifically, it provides the following information:

[0366] 1. Video link to the recommended recipe.

[0367] 2. Required ingredients and information on where to purchase them (price, availability, delivery lead time).

[0368] 3. Information suggesting takeout and dining-out options.

[0369] Emotional Engine Processing

[0370] emotion recognition

[0371] 1. The emotion engine analyzes the user's facial expressions, voice tone, and natural language text input through the device to recognize the user's emotional state (joy, sadness, stress, etc.).

[0372] 2. The recognized emotional state is sent to the server and used as part of the data analysis.

[0373] Emotion-based adjustment

[0374] 1. Based on data from the emotion engine, the server adjusts recipe and ingredient purchase advice. For example, if the user is feeling tired, it will suggest easy-to-make recipes and ingredients.

[0375] Terminal processing

[0376] User Interface

[0377] The device (smartphone or tablet) is primarily responsible for user interaction.

[0378] 1. Notify users of recipe video links and ingredient purchase information received from the server.

[0379] 2. Receive user input (selection of optional services and feedback) and send it to the server.

[0380] 3. Utilize an emotion engine to recognize and analyze the user's emotional state.

[0381] Optional Services

[0382] If a user selects a paid optional service (such as shopping assistance, housekeeping assistance, or restaurant reservations), the terminal forwards the request to the server. The server processes the request and sends it to the appropriate service provider.

[0383] User actions

[0384] Information entry

[0385] Users input information such as family composition and event dates into the system. They also update food inventory information via the IoT refrigerator.

[0386] Acceptance of proposals and feedback

[0387] Users review and implement recipe suggestions, ingredient purchase information, and dining options received from the system. They then provide feedback on the services they used.

[0388] Specific example: How to cope with stressful days

[0389] 1. The user inputs information into the device in natural language, indicating that they are stressed at work. The emotion engine analyzes this information and recognizes the user's emotional state.

[0390] 2. Based on data from the emotion engine, the server selects recipes using ingredients that have a relaxing effect. For example, it might suggest herbal teas or dishes that promote relaxation.

[0391] 3. The server lists the missing ingredients and provides information on the best place to purchase them.

[0392] 4. The device notifies the user of the suggested recipe video link and purchase information.

[0393] 5. The user accepts the system's suggestions and uses the shopping assistance service as needed.

[0394] 6. The server processes the request and instructs a partner grocery delivery service to purchase the ingredients.

[0395] This specific processing flow allows users to efficiently perform a series of tasks, from purchasing ingredients to cooking and dining out, resulting in a significant reduction in the time and effort required for daily meal preparation. Furthermore, the emotion engine takes into account the user's emotional state to provide personalized and highly satisfying service.

[0396] The following describes the processing flow.

[0397] Step 1: Enter user information

[0398] Users input their family structure, food preferences, and event dates into the system. The system also synchronizes with an IoT refrigerator, sending current food inventory information to the system.

[0399] Step 2: Data Collection (Refrigerator)

[0400] The terminal retrieves current inventory information from the IoT refrigerator and sends it to the server. The information about the food items in the refrigerator includes the type of food, quantity, and expiration date.

[0401] Step 3: Data Collection (Weather Forecast)

[0402] The server uses a weather forecast API to collect the latest weather information. This weather information includes local temperature, probability of precipitation, and humidity.

[0403] Step 4: Data Collection (Market Information)

[0404] The server collects flyer information, inventory status, and price information from online retail stores and online supermarkets via the internet.

[0405] Step 5: Data Collection (Restaurant and Takeout Information)

[0406] The server collects information from partner restaurants and food service providers. This includes menu items, prices, takeout availability, and reservation status.

[0407] Step 6: Data Collection (Sentimental Information)

[0408] The device's emotion engine analyzes the user's facial expressions, voice tone, and natural language text input to recognize the user's emotional state. The recognized emotional state is then sent to the server.

[0409] Step 7: Data Analysis (Recipe Selection)

[0410] The server performs multiple data analyses based on refrigerator inventory information, market information, weather forecasts, family composition, event dates, and the user's emotional state to generate optimal recipe suggestions. For example, if the user is feeling stressed, it will suggest recipes using ingredients that have a relaxing effect.

[0411] Step 8: Check inventory and list any missing ingredients.

[0412] The server compares the list of ingredients required for the recipe with the inventory information in the refrigerator and creates a list of missing ingredients.

[0413] Step 9: Selecting a supplier

[0414] The server searches for the best store to purchase the missing ingredients. It matches prices, inventory, and logistics lead times from online retail stores and online supermarkets to determine the most efficient source of purchase.

[0415] Step 10: Provide Information

[0416] The device will notify the user's smartphone or tablet of the following information:

[0417] Recommended recipe video link

[0418] A list of necessary ingredients and information on where to buy them (price, availability, delivery lead time)

[0419] Takeout and dining-out suggestions

[0420] Step 11: Select Optional Services

[0421] Users select the option they want to use from paid services such as shopping assistance, housekeeping assistance, and restaurant reservations, and send a request to the server via their device.

[0422] Step 12: Request a service

[0423] The server sends a request to a partner service provider based on the user's request. For grocery shopping, it provides a list of selected groceries from various suppliers. For housekeeping services, it sends the specific tasks to be done and the desired date and time. For restaurant reservations, it processes the reservation based on the user's desired date and time and the restaurant's information.

[0424] Step 13: Gathering Feedback

[0425] The device collects user feedback and sends it to the server. Users enter their ratings and opinions on the provided recipes and services.

[0426] Step 14: Analyzing Feedback and Learning

[0427] The server analyzes the collected feedback and uses it to improve the accuracy of the system's suggestions. The feedback data is then incorporated into future recipe and service suggestions.

[0428] As a concrete example, consider a day when the user is experiencing a lot of stress:

[0429] 1. The user enters into the terminal in natural language that they are experiencing work-related stress.

[0430] 2. The emotion engine analyzes that information and recognizes the user's emotional state.

[0431] 3. Based on data from the emotion engine, the server selects recipes that use ingredients with relaxing effects.

[0432] 4. The server lists the missing ingredients and provides information on the best place to purchase them.

[0433] 5. The device notifies the user of the suggested recipe video link and purchase information.

[0434] 6. The user accepts the system's suggestions and uses the shopping assistance service as needed.

[0435] 7. The server processes the request and instructs a partner grocery delivery service to purchase the ingredients.

[0436] This specific processing flow allows users to efficiently perform a series of tasks, from purchasing ingredients to cooking and dining out, resulting in a significant reduction in the time and effort required for daily meal preparation. The emotion engine takes the user's emotional state into account when making suggestions, enabling a more personalized and satisfying service.

[0437] (Example 2)

[0438] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0439] In today's busy lifestyle, users often spend a lot of time and effort preparing their daily meals. Furthermore, there is no system that suggests ingredients or recipes based on emotional states or schedules, nor does it offer meal suggestions tailored to individual user circumstances. Therefore, there is a need for a system that increases user satisfaction and allows for efficient meal preparation.

[0440] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0441] In this invention, the server includes means for collecting flyer information from online retail stores, means for receiving inventory information in the refrigerator, means for obtaining food purchase information based on logistics lead times, means for receiving weather forecast information, means for obtaining family composition and event date information, means for analyzing and recognizing the user's emotional state, and means for analyzing this information and emotional state to provide optimal recipes and food purchase advice. This makes it possible to select ingredients and suggest recipes that are tailored to the user's individual circumstances, enabling efficient and satisfying meal preparation.

[0442] "Online retail store flyer information" refers to special sale information and advertising data for products offered by retail stores, obtained via the internet.

[0443] "Refrigerator inventory information" refers to specific information about food items and products stored inside a refrigerator, obtained using IoT technology and other methods.

[0444] "Logistics lead time" is an indicator that shows the time it takes from ordering ingredients or products to delivery.

[0445] "Weather forecast information" refers to data that shows changes in weather and forecasts, and is obtained through APIs, etc.

[0446] "Family structure information" refers to information such as the members of the user's household, their number, and their ages.

[0447] "Event date information" refers to information about specific dates that are important to the user or their family (e.g., birthdays, anniversaries, etc.).

[0448] "Emotional state" refers to data that indicates the user's current emotional state (joy, sadness, stress, etc.).

[0449] "Recipe and ingredient purchasing advice" refers to suggestions and advice provided to users regarding how to prepare dishes, the necessary ingredients, and how to purchase them.

[0450] "Methods for providing recipe videos" refer to methods and technologies for providing users with cooking instructions in video format.

[0451] "Takeout information" refers to information about meals available for takeout from restaurants and other establishments.

[0452] "Restaurant information" refers to information about the menus and services offered by restaurants.

[0453] A "server" is a central computer system used for collecting, analyzing, storing, and distributing data.

[0454] A "terminal" refers to a device (e.g., a smartphone or tablet) that a user uses to input or receive information.

[0455] This invention relates to a system that provides optimal recipes and ingredient purchasing advice tailored to the individual circumstances of the user. This system mainly consists of three components: a server, a terminal, and the user.

[0456] Server Processing

[0457] The server plays a central role in data collection, analysis, and proposal development. First, the server collects the following information via the internet:

[0458] 1. Collecting flyer information from retail stores: The server uses web scraping tools such as Python's BeautifulSoup and Selenium to retrieve the latest flyer information from online retail stores.

[0459] 2. Obtaining inventory information from the IoT refrigerator: The server periodically receives inventory information from the refrigerator via the IoT refrigerator's API.

[0460] 3. Information acquisition from online supermarkets and grocery delivery sites: The server uses the APIs of each online supermarket and grocery delivery site to acquire inventory information and logistics lead times.

[0461] 4. Obtaining weather forecast information: The server uses weather forecast APIs such as the OpenWeatherMap API to obtain the latest weather information.

[0462] 5. Information gathering from partner restaurants and food service providers: The server uses the APIs of each service to collect takeout information and menu information from partner restaurants and food service providers.

[0463] Next, the server analyzes the collected data:

[0464] 1. Creating a list of seasonal ingredients: Using the Python Pandas library, we will create a list of seasonal ingredients by matching the inventory information in the refrigerator with information from retail store flyers.

[0465] 2. Recipe Selection: Using a generative AI model (e.g., GPT model), the optimal recipe is selected considering weather forecasts, family structure, event dates, and emotional states determined by an emotion engine.

[0466] 3. Listing ingredients: Perform a difference calculation on the list data and compare the required ingredients for the selected recipe with the ingredients in the refrigerator to list any missing ingredients.

[0467] 4. Provision of supplier information: An optimization algorithm that takes logistics lead time into consideration will be executed to select the best supplier for the missing ingredients, and the calculation will include price information.

[0468] 5. Suggesting dining options: Using an ML model, we analyze available takeout and dining options and suggest them to users in a ranked format.

[0469] Emotional Engine Processing

[0470] The emotion engine recognizes the user's emotional state and uses it as part of data analysis. Specifically, it performs the following processes:

[0471] 1. Analysis of emotion recognition data: Using an emotion recognition model that utilizes deep learning, we analyze the user's facial expressions, voice tone, and natural language text sent through the device.

[0472] 2. Transmission of emotional data: The recognized emotional state is sent to the server and used to provide advice on recipe selection and ingredient purchases.

[0473] Terminal processing

[0474] The device (smartphone or tablet) is primarily responsible for user interaction:

[0475] 1. Information Notification: Notify users of recipe video links and ingredient purchase information received from the server. Use push notification functionality on smartphones and tablets.

[0476] 2. Receiving user input: User input (selection of optional services and feedback) is received via the user interface and sent to the server via the REST API.

[0477] 3. Emotional State Analysis: An emotion recognition model is executed in real time to recognize and analyze the user's emotional state.

[0478] User actions

[0479] The user performs the following actions:

[0480] 1. Information Input: Information such as family composition and event dates is entered into the system using a dedicated application or terminal interface. Additionally, food inventory information is updated via the IoT refrigerator.

[0481] 2. Review and Feedback on Suggestions: Review and implement recipe suggestions, ingredient purchase information, and dining-out options received from the system. Afterwards, provide feedback on the services used.

[0482] Specific example

[0483] For example, if a user says, "Today was very stressful," the emotion engine analyzes this input and recognizes that the user is feeling stressed. Based on this emotion data, the server selects a relaxing herbal tea and an easy-to-make recipe, and provides information on any missing ingredients. It also sends a corresponding video link to the device and notifies the user. The user can then review the suggested recipe and ingredient list and, if necessary, utilize a shopping assistance service to efficiently purchase ingredients and prepare the meal.

[0484] Example of a prompt

[0485] "I don't know what to cook today."

[0486] "Please tell me some recommended recipes using ingredients I have in my refrigerator."

[0487] "I'm feeling stressed, so please suggest some dishes that will help me relax."

[0488] This allows users to receive ingredient selections and recipe suggestions tailored to their individual circumstances, enabling efficient and satisfying meal preparation.

[0489] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0490] Processing steps

[0491] Step 1:

[0492] The server collects flyer information from online retail stores. Specifically, it uses web scraping tools such as Python's BeautifulSoup and Selenium. Using these tools, it retrieves special offer information and advertising data from each retail store's website. The input is the URL of the retail store's website, and the output is the collected flyer information data.

[0493] Step 2:

[0494] The server receives inventory information from the IoT refrigerator. This is done using the IoT refrigerator's API. The input is an API request from the IoT refrigerator, and the output is inventory information for each food item detected inside the refrigerator. This information is stored in JSON format.

[0495] Step 3:

[0496] The server retrieves inventory information and logistics lead times from partner online supermarkets and grocery delivery sites. It uses the APIs of each service to collect this information. The input is the API endpoint of the online supermarket or grocery delivery site, and the output is inventory information and its corresponding logistics lead time.

[0497] Step 4:

[0498] The server obtains the latest weather information through a weather forecast API. Specifically, it uses services such as the OpenWeatherMap API. The input is a request to the weather forecast API, and the output is weather information data. This data includes detailed information such as temperature, probability of precipitation, and wind speed.

[0499] Step 5:

[0500] The server collects takeout and menu information from partner restaurants and food service providers. It retrieves the necessary information through each store's API. The input is the API endpoint of the partner restaurants and food service providers, and the output is takeout and menu information.

[0501] Step 6:

[0502] The server retrieves family structure and event date information previously entered by the user from the database. It uses databases such as SQLite or MySQL for searching. The input is a query string, and the output is the corresponding family structure and event date information.

[0503] Step 7:

[0504] The server matches the retrieved inventory information from the refrigerator with information from retail store flyers to create a list of seasonal ingredients. The Python Pandas library is used for data matching. The input is the refrigerator inventory information and the flyer information, and the output is a list of seasonal ingredients.

[0505] Step 8:

[0506] The server selects the optimal recipe by considering the weather forecast, family structure, event dates, and the user's emotional state as determined by the emotion engine. It suggests recipes using a generative AI model (e.g., GPT model). The input is this information and emotion data, and the output is the recommended recipe.

[0507] Step 9:

[0508] The server compares the required ingredients for the selected recipe with the inventory in the refrigerator and lists any missing ingredients. A Python data processing script is used to calculate the difference. The input is a list of required ingredients for the recipe and the inventory information in the refrigerator, and the output is a list of missing ingredients.

[0509] Step 10:

[0510] The server selects the optimal supplier (online supermarket or e-commerce site) for the missing ingredients and calculates logistics lead time and price information. It retrieves this data using the API of each supplier and runs an optimization algorithm. The input is a list of missing ingredients and inventory and price information for each supplier, and the output is information on the optimal supplier.

[0511] Step 11:

[0512] The server uses an ML model to analyze available takeout and dining-in options and suggests them to the user in a ranked format. The input is the user's location information and data on partner restaurants, and the output is a ranked list of takeout and dining-in options.

[0513] Step 12:

[0514] The emotion engine analyzes the user's facial expressions, voice tone, and text input through the device to recognize the user's emotional state. It uses an emotion recognition model based on deep learning. The input is the user's facial expression data and voice data, and the output is the recognized emotional state.

[0515] Step 13:

[0516] Emotional data from the emotion engine is sent to the server and used for recipe selection and ingredient purchasing advice. The input is recognized emotional data, and the output is the input data necessary for analysis performed by the server.

[0517] Step 14:

[0518] The device notifies the user of recipe video links and ingredient purchase information received from the server. Specifically, it uses the push notification function of smartphones and tablets. The input is analysis result data from the server, and the output is a notification to the user.

[0519] Step 15:

[0520] The terminal receives user input (selection of optional services and feedback) via the user interface and sends it to the server via a REST API. Input is user input data, and output is data sent to the server.

[0521] Step 16:

[0522] The device utilizes an emotion engine to run an emotion recognition model in real time, recognizing and analyzing the user's emotional state. The input is real-time facial expressions and voice data, and the output is the recognized emotional state.

[0523] Step 17:

[0524] Users input information such as family composition and event dates into the system using a dedicated application or terminal interface. They also update food inventory information via an IoT refrigerator. Input consists of manually entered data from the user and data from the IoT refrigerator, while output is user information stored on the server.

[0525] Step 18:

[0526] Users review recipe suggestions, ingredient purchase information, and dining-out options received from the system and choose whether to implement them. They then provide feedback on the services used. Input is the suggestion data provided by the server, and output is the user's selections and feedback data.

[0527] (Application Example 2)

[0528] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0529] In modern life, users spend a significant amount of time and effort purchasing ingredients and selecting appropriate recipes amidst their busy schedules. Furthermore, a lack of advice and product suggestions that take into account users' emotional states often leads to stress during the ingredient purchasing and cooking processes. Therefore, there is a need for a system that can adapt to users' emotional states and provide efficient and personalized service.

[0530] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0531] In this invention, the server includes means for collecting flyer information from online retail stores, means for receiving inventory information in the refrigerator, means for obtaining food purchase information based on logistics lead times, means for receiving weather forecast information, means for obtaining family composition and event date information, means for recognizing the user's emotional state, and means for analyzing this information to provide optimal recipes and food purchase advice tailored to the user's emotional state. This enables efficient and personalized recipe suggestions and food purchase advice tailored to the user's emotional state.

[0532] "Means of collecting flyer information from online retail stores" refers to a function that collects information on discounts and special offers provided by online retail stores via the internet.

[0533] "Means of receiving inventory information inside the refrigerator" refers to a function that obtains inventory information of food currently stored inside the refrigerator from a device such as an IoT refrigerator.

[0534] "Method for obtaining food ingredient purchase information based on logistics lead time" refers to a function that obtains information to enable users to make optimal purchases by considering the delivery schedule and lead time of food ingredients.

[0535] "Means of receiving weather forecast information" refers to the function of obtaining information about current and future weather from the internet.

[0536] "Means of obtaining family structure and event date information" refers to a function that retrieves information about the number and composition of family members, as well as information about special dates such as birthdays and anniversaries, which have been entered by the user in advance.

[0537] "Means of recognizing the user's emotional state" refers to a function that analyzes the user's facial expressions, tone of voice, text input, etc., to determine their emotional state at any given time.

[0538] "A means of providing optimal recipes and ingredient purchasing advice tailored to emotions" refers to a function that, based on collected information and emotional states, provides users with the most suitable recipes and ingredient purchasing advice that is sensitive to their emotions.

[0539] The "recipe video provision method" is a function that provides users with links and information to cooking videos related to the selected recipe.

[0540] A "shopping assistant based on emotion recognition" is a function that suggests the most suitable products to purchase in a physical store based on the user's emotional state.

[0541] "Means for collecting takeout and dining-out information" refers to the function of collecting takeout and menu information provided by restaurants and dining-out services.

[0542] "A means of suggesting the optimal product" refers to a function that suggests the most suitable product for the user based on the information acquired.

[0543] Modes for carrying out the invention

[0544] The system necessary to implement this invention consists of three entities: a server, a terminal, and a user. The server plays a central role in data collection, analysis, and proposal, while the terminal provides information and interacts with the user. The user is the user of the system and is the entity that provides input and feedback.

[0545] The server includes the following measures:

[0546] 1. Means of collecting online retail store flyer information: This system has the function of regularly collecting discount and special offer information provided by various retail stores via the internet. For example, it uses an API to obtain the latest flyer information from each store.

[0547] 2. Means of receiving inventory information from the refrigerator: Accurate inventory information is received in real time from the IoT refrigerator. This allows the user to understand the status of food items in their refrigerator.

[0548] 3. Method for obtaining food ingredient purchase information based on logistics lead time: Food ingredient purchase information, including delivery schedules and lead times, is obtained based on logistics data. This makes it possible to suggest appropriate purchase timings.

[0549] 4. Means of receiving weather forecast information: Obtain the latest weather forecast via API and reflect it in the user's ingredient selection and recipe suggestions.

[0550] 5. Means for obtaining family structure and event date information: The system has a function to store family structure, birthdays, anniversaries, and other event information registered by the user in advance in a database and retrieve that information.

[0551] 6. Means for recognizing the user's emotional state: The user's facial expressions and voice tone are analyzed via the camera and microphone to determine their emotional state. An emotion recognition algorithm is used for this analysis.

[0552] 7. Means of providing optimal recipes and emotionally appropriate ingredient purchasing advice: Based on the collected information, we will provide users with optimal recipes and emotionally appropriate ingredient purchasing advice.

[0553] Specific examples and your hardware and software

[0554] The server is built using the Flask framework with Python, and the emotion recognition function utilizes the EmotionRecognition library. The server collects necessary information through APIs on the internet and performs analysis based on that data. Specifically, when a user inputs emotional information such as "I'm tired" using their smartphone's camera and microphone, EmotionRecognition analyzes that information and selects recipes that include ingredients with relaxing effects.

[0555] The device (smartphone or tablet) functions as an application that notifies the user of information received from the server. This application provides the user with selected recipe videos and optimal ingredient purchasing advice, and also functions as an emotion-based shopping assistant. For example, it might suggest relaxing scented candles or relaxation foods to a user who is feeling stressed.

[0556] Users periodically input and update their emotional information, family structure, and refrigerator inventory information into the system. This allows the system to always provide advice based on the most up-to-date information.

[0557] Example of a prompt

[0558] "We will provide image and audio data. Please analyze this user's emotions. If the emotion is 'stressed,' suggest products that promote relaxation; if it's 'sad,' suggest products that offer comfort; and for other emotions, suggest products that generally support health."

[0559] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0560] Step 1:

[0561] Data collection

[0562] The server uses online APIs to collect information such as retail store flyers, weather forecasts, inventory information from partner online supermarkets and grocery delivery sites, and menu information from partner restaurants and food service providers. Inputs include data from various APIs, and output is a dataset integrating this data. The specific operation involves sending requests to each API and saving the returned information to the database.

[0563] Step 2:

[0564] Receiving inventory information from the refrigerator

[0565] The server receives inventory information periodically transmitted from the IoT refrigerator. The input is inventory data transmitted from the IoT refrigerator, and the output is the latest inventory status data in the user's refrigerator. Specifically, the server receives data from the IoT refrigerator in real time and updates the database.

[0566] Step 3:

[0567] Recognizing the user's emotional state

[0568] The device (smartphone) uses its camera and microphone to collect the user's facial expressions and voice tone. The collected data is sent to a server and analyzed by an emotion recognition algorithm (EmotionRecognition). The input is the user's image and audio data, and the output is the user's emotional state as a result of the analysis (e.g., stress, joy, sadness). The specific operation is to pass the data captured by the camera and microphone to EmotionRecognition and estimate the emotional state.

[0569] Step 4:

[0570] Data analysis and proposal generation

[0571] The server analyzes the collected data and generates optimal recipes and ingredient purchase advice, taking into account the user's refrigerator inventory, weather forecast, family structure, event dates, and recognized emotional state. The input consists of various collected data and recognized emotional states, while the output is the optimal recipe and ingredient purchase advice based on the analysis. Specifically, it uses an algorithm to combine various data and determine the most suitable recipe and purchase advice.

[0572] Step 5:

[0573] Providing information

[0574] The server sends the analysis results to the terminal and notifies the user. Specifically, it provides links to recipe videos, information on necessary ingredients and where to purchase them, and suggestions for takeout and dining out. The input is the analysis result data, and the output is the information displayed on the user's terminal. The specific operation is to send the analysis results to the terminal in an appropriate format and display them to the user.

[0575] Step 6:

[0576] User interaction

[0577] Users view suggested recipes and ingredient purchase information through their terminals and select paid optional services, such as grocery shopping assistance, as needed. Input is the user's selection data, and output is request data related to the selected service. The specific operation involves the user making selections via touch operations and sending that data to the server.

[0578] Step 7:

[0579] Arranging a shopping assistance service

[0580] The server processes user requests and sends grocery purchase requests to affiliated shopping agents. The input is the user's request data, and the output is the request data sent to the shopping agent. Specifically, it parses the request data and sends the request details to the specified shopping agent's API.

[0581] Example of a prompt

[0582] "We will provide image and audio data. Please analyze this user's emotions. If the emotion is 'stressed,' suggest products that promote relaxation; if it's 'sad,' suggest products that offer comfort; and for other emotions, suggest products that generally support health."

[0583] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0584] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0585] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0586] [Second Embodiment]

[0587] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0588] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0589] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0590] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0591] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0592] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0593] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0594] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0595] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0596] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0597] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0598] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0599] To carry out this invention, the following system and its processing procedure will be described.

[0600] Overall system configuration

[0601] This system primarily consists of three main components: the server, the terminal, and the user. The server plays a central role in data collection, analysis, and proposal development, while the terminal provides information and interacts with the user. The user is the system's user, providing input and feedback.

[0602] Server Processing

[0603] Data collection

[0604] The server uses the internet to collect the following information:

[0605] 1. Obtain flyer information from online retail stores. This ensures that discount and special offer information is always up-to-date.

[0606] 2. The system periodically receives inventory information from the IoT refrigerator. This information is used to keep track of the user's food inventory.

[0607] 3. Obtain inventory information and logistics lead times from partner online supermarkets and grocery delivery sites. This will allow us to understand which groceries are available for purchase and their delivery schedules.

[0608] 4. Obtain the latest weather information from the weather forecast API.

[0609] 5. Collect takeout information and menu information from partner restaurants and food service providers.

[0610] 6. Obtain information about family structure and event dates (such as birthdays and anniversaries) that the user has entered in advance.

[0611] Data Analysis

[0612] The server performs multiple data analyses based on the collected information.

[0613] 1. Compare the inventory information obtained from the refrigerator with the information from online retail store flyers to create a list of seasonal ingredients.

[0614] 2. Select the optimal recipe by considering parameters such as weather forecasts, family composition, and event dates.

[0615] 3. Compare the ingredients required for the selected recipe with the ingredients the user has in their refrigerator and list any missing ingredients.

[0616] 4. Select the best source (online supermarket or e-commerce site) for any missing ingredients and calculate logistics lead times and pricing information.

[0617] 5. Analyze available takeout and dining-in options and suggest them to users.

[0618] Information provision

[0619] The server sends the analysis results to the terminal. Specifically, it provides the following information:

[0620] 1. Video link to the recommended recipe.

[0621] 2. Required ingredients and information on where to purchase them (price, availability, delivery lead time).

[0622] 3. Information suggesting takeout and dining-out options.

[0623] Terminal processing

[0624] User Interface

[0625] The device (smartphone or tablet) is primarily responsible for user interaction.

[0626] 1. Notify users of recipe video links and ingredient purchase information received from the server.

[0627] 2. Receive user input (selection of optional services and feedback) and send it to the server.

[0628] Optional Services

[0629] If a user selects a paid optional service (such as shopping assistance, housekeeping assistance, or restaurant reservations), the terminal forwards the request to the server. The server processes the request and sends it to the appropriate service provider.

[0630] User actions

[0631] Information entry

[0632] Users input information such as family composition and event dates into the system. They also update food inventory information via the IoT refrigerator.

[0633] Acceptance of proposals and feedback

[0634] Users review and implement recipe suggestions, ingredient purchase information, and dining-out options received from the system. They then provide feedback on the services they used.

[0635] Specific example: When preparing dinner for the family

[0636] 1. The user enters into the system that their child's birthday is approaching. The system also updates the inventory information in the refrigerator.

[0637] 2. The server suggests party recipes based on the inventory information in the refrigerator, a list of seasonal ingredients, and the weather forecast.

[0638] 3. The server lists the missing ingredients and provides information on the best place to purchase them.

[0639] 4. The device notifies the user of the suggested recipe video link and purchase information.

[0640] 5. The user accepts the system's suggestions and uses the shopping assistance service as needed.

[0641] 6. The server processes the request and instructs a partner grocery delivery service to purchase the ingredients.

[0642] This invention allows users to efficiently perform a series of tasks, from purchasing ingredients to cooking and eating out, and as a result, significantly reduce the effort and time required for daily meal preparation.

[0643] The following describes the processing flow.

[0644] Step 1: Enter user information

[0645] Users input their family structure, food preferences, and event dates into the system. They also connect to an IoT refrigerator to synchronize current food inventory information.

[0646] Step 2: Data Collection (Refrigerator)

[0647] The terminal retrieves current inventory information from the IoT refrigerator and sends it to the server. The information about the food items in the refrigerator includes the type of food, quantity, and expiration date.

[0648] Step 3: Data Collection (Weather Forecast)

[0649] The server uses a weather forecast API to collect the latest weather information. This weather information includes local temperature, probability of precipitation, and humidity.

[0650] Step 4: Data Collection (Market Information)

[0651] The server collects flyer information, inventory status, and price information from online retail stores and online supermarkets via the internet.

[0652] Step 5: Data Collection (Restaurant and Takeout Information)

[0653] The server collects information from partner restaurants and food service providers. This includes menu items, prices, takeout availability, and reservation status.

[0654] Step 6: Data Analysis (Recipe Selection)

[0655] The server analyzes the collected data and generates optimal recipe suggestions based on the user's preferences, event dates, and weather forecasts. For example, it might suggest a warm stew recipe on a rainy day and a barbecue recipe on a sunny day.

[0656] Step 7: Check inventory and list any missing ingredients.

[0657] The server compares the list of ingredients required for the recipe with the inventory information in the refrigerator and creates a list of missing ingredients.

[0658] Step 8: Selecting a supplier

[0659] The server searches for the best store to purchase the missing ingredients. It compares prices, inventory, and delivery lead times from online retail stores and online supermarkets to determine the most efficient source of supplies.

[0660] Step 9: Information Provision

[0661] The device will notify the user's smartphone or tablet of the following information:

[0662] Recommended recipe video link

[0663] A list of necessary ingredients and information on where to buy them (price, availability, delivery lead time)

[0664] Takeout and dining-out suggestions

[0665] Step 10: Selecting Optional Services

[0666] Users select the option they want to use from paid services such as shopping assistance, housekeeping assistance, and restaurant reservations, and send a request to the server via their device.

[0667] Step 11: Service Request

[0668] The server sends a request to a partner service provider based on the user's request. For grocery shopping, it provides a list of selected groceries from various suppliers. For housekeeping services, it sends the specific tasks to be done and the desired date and time. For restaurant reservations, it processes the reservation based on the user's desired date and time and the restaurant's information.

[0669] Step 12: Gathering Feedback

[0670] The device collects user feedback and sends it to the server. Users enter their ratings and opinions on the provided recipes and services.

[0671] Step 13: Analyzing Feedback and Learning

[0672] The server analyzes the collected feedback and uses it to improve the accuracy of the system's suggestions. The feedback data is then reflected in future recipe and service suggestions.

[0673] This specific processing flow will allow users to efficiently manage everything from meal preparation and grocery shopping to dining out.

[0674] (Example 1)

[0675] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0676] In today's busy lifestyle, efficiently managing and purchasing groceries, as well as preparing meals, is a significant burden for many. In particular, there is a need for a system that centrally manages and provides information such as grocery inventory, optimal recipes, weather forecasts, and takeout options, but such a comprehensive system does not yet exist. As a result, users spend a great deal of time and effort on this, making it difficult to live an efficient daily life.

[0677] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0678] In this invention, the server includes means for collecting flyer information from online retail stores, means for receiving inventory information in the refrigerator, means for obtaining food purchase information based on logistics lead times, means for receiving weather forecast information, means for obtaining family composition and event date information, means for analyzing this information and providing optimal recipes and food purchase advice, means for providing video links to recommended recipes, means for collecting takeout information and menu information from affiliated restaurants and food service providers, user interface means for receiving input from the user and transmitting it to the server, and means for creating a list of missing ingredients and calculating prices and inventory information for the best suppliers. This enables the user to efficiently manage and purchase ingredients, cook, and choose where to eat out.

[0679] An "online retail store" is a website or application established for the purpose of selling goods online.

[0680] "Flyer information" refers to digital promotional materials used in retail stores, including information on special offers, discounts, and campaigns.

[0681] "Refrigerator inventory information" refers to data showing the types, quantities, and expiration dates of food items currently stored in a household refrigerator.

[0682] "Logistics lead time" refers to the time it takes from the time an order is placed until the product is actually delivered.

[0683] "Weather forecast information" refers to data that predicts future weather conditions, including temperature, probability of precipitation, and wind speed.

[0684] "Family structure" refers to information indicating the number of people in the user's household, their age distribution, and their relationships.

[0685] "Event date information" refers to information that users have entered, such as birthdays, anniversaries, and dates of special events.

[0686] The "optimal recipe" is a list of cooking steps and ingredients that best suit the user's needs and circumstances, based on the collected data.

[0687] "Food purchase advice" refers to recommendations regarding how to purchase the ingredients a user needs, the best places to buy them, and their prices.

[0688] The "recommended recipe video link" is a link to access video content related to the selected recipe.

[0689] "Partner restaurants and food service providers" refer to restaurants and food service providers that collaborate with the system to share information and provide services.

[0690] "Takeout information" refers to the take-out menus offered by restaurants and related details.

[0691] "Menu information" refers to a list of dishes and drinks offered by a restaurant, along with detailed information about them.

[0692] "User interface means" refers to the means by which a user interacts with a system, and includes devices such as smartphones and tablets.

[0693] The "list of missing ingredients" is a list of ingredients needed for the selected recipe that are not currently in the refrigerator.

[0694] The "optimal supplier" refers to a place to purchase food ingredients that has been evaluated based on criteria such as price, stock availability, and logistics lead time.

[0695] Modes for carrying out the invention

[0696] To implement this invention, three main entities are primarily used: a server, a terminal, and a user. The server plays a central role in data collection, analysis, and proposal, while the terminal provides information and interacts with the user. The user is the system user and the entity that provides input and feedback.

[0697] Server Processing

[0698] Data collection

[0699] The server collects the following information via the internet:

[0700] 1. Obtaining flyer information from online retail stores.

[0701] The server uses scraping technology to periodically access the websites of online retail stores and automatically retrieve the latest flyer information. This ensures that discount and special offer information is always up-to-date.

[0702] 2. Receiving inventory information from the IoT refrigerator.

[0703] The server periodically receives inventory information from IoT refrigerators installed in the user's home. MQTT and HTTP are used as communication protocols. This allows the server to monitor the food inventory inside the refrigerator.

[0704] 3. Obtain inventory information and logistics lead times from online supermarkets and food delivery websites.

[0705] The server calls APIs from partner online supermarkets and grocery delivery sites to obtain the latest inventory information and logistics lead times.

[0706] 4. Obtaining weather information from a weather forecast API

[0707] The server periodically calls the weather forecast API to retrieve the latest weather information. Authentication is performed using the API key.

[0708] 5. Obtaining takeout information and menu information from partner restaurants and food service providers.

[0709] The server calls APIs from partner restaurants and food service providers to retrieve the latest takeout and menu information.

[0710] 6. Retrieving family structure and event date information entered by the user.

[0711] Users enter information such as family composition and event dates (birthdays, anniversaries, etc.) through the application's input form. This information is sent to the server and stored in the database.

[0712] Data Analysis

[0713] Based on the information collected above, the server performs the following data analysis:

[0714] 1. Create a list of seasonal ingredients.

[0715] The server compares the retrieved inventory information from the refrigerator with online retail store flyers to create a list of seasonal ingredients. This process uses the Python Pandas library.

[0716] 2. Selecting the optimal recipe

[0717] The server selects the optimal recipe by considering parameters such as weather forecasts, family composition, and event dates. This recipe selection is performed using an AI model (e.g., TensorFlow or PyTorch).

[0718] 3. List of missing ingredients

[0719] The system compares the required ingredients for the selected recipe with the user's refrigerator inventory and lists any missing ingredients. The Numpy library is used for the comparison process.

[0720] 4. Selecting the optimal supplier and calculating price and logistics lead time.

[0721] For any missing ingredients, the server selects the optimal supplier and calculates the total cost based on price information and logistics lead time. Linear programming may be used.

[0722] 5. Analysis of takeout and dining-in options

[0723] The system also analyzes available takeout and dining-in options and suggests them to the user. Suggestions are generated based on past user preference data.

[0724] Information provision

[0725] The server sends the analysis results to the terminal. Specifically, it provides the following information:

[0726] 1. Recommended recipe video link

[0727] The server generates a video link for the selected recipe and sends it to the device. The user can view the video by clicking the link.

[0728] 2. Required ingredients and where to buy them

[0729] The server compiles the necessary ingredients and their supplier information (price, availability, delivery lead time) and sends it to the terminal.

[0730] 3. Information on takeout and dining out options.

[0731] Based on the analysis results, the server sends suggested takeout and dining-in options to the terminal.

[0732] Terminal processing

[0733] User Interface

[0734] The device (smartphone or tablet) is responsible for user interaction:

[0735] 1. Sending notifications

[0736] The system notifies users of recipe video links and ingredient purchase information received from the server. Notification methods include push notifications and in-app notifications.

[0737] 2. Receiving and sending user input

[0738] The terminal receives input from the user (for example, selection of optional services or feedback) and sends it to the server.

[0739] Provision of optional services

[0740] If a user selects a paid optional service (such as shopping assistance, housekeeping assistance, or restaurant reservations), the terminal forwards the request to the server. The server processes the request and sends it to the appropriate service provider.

[0741] User actions

[0742] Information entry

[0743] Users input information about their family structure and event dates (e.g., birthdays and anniversaries) into the system. They can also update their food inventory information via the IoT refrigerator.

[0744] Acceptance of proposals and feedback

[0745] Users review and implement recipe suggestions, ingredient purchase information, and dining options received from the system. They then provide feedback on the services they used.

[0746] Specific example: When preparing dinner for the family

[0747] 1. The user enters into the system that their child's birthday is approaching. The system also updates the inventory information in the refrigerator.

[0748] 2. The server suggests party recipes based on the inventory information in the refrigerator, a list of seasonal ingredients, and the weather forecast.

[0749] 3. The server lists the missing ingredients and provides information on the best place to purchase them.

[0750] 4. The device notifies the user of the suggested recipe video link and purchase information.

[0751] 5. The user accepts the system's suggestions and uses the shopping assistance service as needed.

[0752] 6. The server processes the request and instructs a partner grocery delivery service to purchase the ingredients.

[0753] Using this specific example, users can efficiently manage and purchase ingredients, cook, and even choose restaurants, completing a series of tasks. As a result, the time and effort involved in daily meal preparation can be significantly reduced.

[0754] Examples of prompts to input into a generative AI model are as follows:

[0755] "My child's birthday is approaching. Please update the ingredients in the refrigerator and suggest party recipes based on online retail stores and weather forecasts. Based on the suggestions, please list any missing ingredients and provide information on the best places to buy them (price, availability, delivery lead time)."

[0756] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0757] Step 1: Data Collection

[0758] The server collects necessary information from multiple sources. Specifically, it accesses the websites of online retail stores and uses scraping techniques to obtain the latest flyer information. This flyer information includes data on discounts and special offers.

[0759] Input: URL of a retail store's website on the internet

[0760] Processing: Use scraping techniques to obtain flyer information and save it to a database.

[0761] Output: Update of flyer information database

[0762] Specifically, it uses Python libraries such as BeautifulSoup and Scrapy to analyze the HTML structure of a web page and extract the necessary information.

[0763] Step 2: Obtain inventory information from the refrigerator

[0764] The server periodically receives inventory information from IoT refrigerators installed in the user's home. MQTT and HTTP are used as communication protocols.

[0765] Input: Inventory information sent from IoT refrigerator

[0766] Processing: Analyze inventory information and save it to the database.

[0767] Output: Refrigerator inventory database update

[0768] Specifically, the IoT refrigerator sends inventory information to the cloud at regular intervals, and a server retrieves and analyzes this data.

[0769] Step 3: Obtain logistics lead time and inventory information.

[0770] The server calls APIs from partner online supermarkets and grocery delivery sites to obtain the latest inventory information and logistics lead times.

[0771] Input: API key and endpoint for online supermarkets and grocery delivery sites.

[0772] Processing: Make an API call and save the retrieved inventory information and logistics lead time to the database.

[0773] Output: Update of inventory and logistics lead time databases

[0774] Specifically, the process involves making HTTPS requests using a RESTful API, parsing the data returned in JSON format, and saving it.

[0775] Step 4: Gathering weather forecast information

[0776] The server periodically calls the weather forecast API to retrieve the latest weather information. Authentication is performed using the API key.

[0777] Input: API key and endpoint for the weather forecast API.

[0778] Process: Make an API call and save the retrieved weather information to the database.

[0779] Output: Weather forecast database update

[0780] Specifically, the process involves making an HTTPS request using an API client and then parsing the returned JSON data.

[0781] Step 5: Gathering information on partner restaurants

[0782] The server calls APIs from partner restaurants and food service providers to retrieve the latest takeout and menu information.

[0783] Input: API key and endpoint for partner restaurants and food service providers

[0784] Processing: Make an API call and save the retrieved takeout information and menu information to the database.

[0785] Output: Updates to the takeout and menu databases.

[0786] In terms of specific operations, it will similarly make HTTPS requests using a RESTful API, parse the data, and save it.

[0787] Step 6: Obtaining User Information

[0788] Users enter information such as family composition and event dates (birthdays, anniversaries, etc.) through the application's input form. This information is sent to the server and stored in the database.

[0789] Input: Family composition and event date information entered by the user.

[0790] Processing: Parse the input information and save it to the database.

[0791] Output: User information database update

[0792] In terms of specific operation, a POST request is sent from the frontend input form to the server, and the data is received and analyzed on the backend.

[0793] Step 7: Data Analysis and Recipe Selection

[0794] The server performs data analysis based on the collected information and selects the optimal recipe that matches the user's conditions. It uses AI models (e.g., TensorFlow, PyTorch).

[0795] Input: Refrigerator inventory information, flyer information, weather forecast, family composition, event date information

[0796] Processing: Use an AI model to select the optimal recipe.

[0797] Output: Selected recipes

[0798] In terms of specific operations, data is input into the AI ​​model, and the optimal recipe is obtained as a result of the model's prediction.

[0799] Step 8: List the missing ingredients

[0800] The server compares the ingredients required for the selected recipe with the ingredients currently in the refrigerator and lists any missing ingredients.

[0801] Input: Selected recipe, refrigerator inventory information

[0802] Processing: Use the Numpy library to list the missing ingredients.

[0803] Output: List of missing ingredients

[0804] Specifically, the process involves using a data frame to compare required ingredients with current inventory and extracting any missing ingredients.

[0805] Step 9: Selecting the best supplier and calculating the price.

[0806] The server selects the best supplier for any missing ingredients and calculates the total cost based on price information and logistics lead time.

[0807] Input: List of missing ingredients, online supermarket inventory information, price information, logistics lead time

[0808] Process: Use linear programming to calculate the optimal supplier and total cost.

[0809] Output: Optimal purchasing information and total cost

[0810] Specifically, the program performs linear programming using the Scipy library.

[0811] Step 10: Provide a link to a recommended recipe video.

[0812] The server generates links to video content related to the selected recipe and sends them to the device.

[0813] Input: Selected recipe

[0814] Processing: Generate video link and notify device.

[0815] Output: Recommended recipe video link notification

[0816] Specifically, the system retrieves video links related to the recipe from the database and notifies the user.

[0817] Step 11: Notifying and providing feedback to the user

[0818] The device notifies the user based on information received from the server. It also receives feedback from the user.

[0819] Input: Notification information from the server, user feedback

[0820] Processing: Sending notifications, receiving feedback and forwarding it to the server.

[0821] Output: User notification and feedback sent to the server

[0822] Specifically, the system sends push notifications via a mobile application and receives user feedback.

[0823] Step 12: Requesting and processing optional services

[0824] The server processes user requests for paid optional services and forwards the requests to the appropriate service providers.

[0825] Input: User's optional service request

[0826] Processing: Processing the request and forwarding the request to the service provider.

[0827] Output: Request to service provider

[0828] In terms of specific operation, the API is used to send a corresponding request to the service provider.

[0829] (Application Example 1)

[0830] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0831] In today's busy lifestyle, meal preparation has become a significant burden for users. In particular, keeping track of what's in the refrigerator, planning appropriate recipes, purchasing necessary ingredients, and finding restaurant or delivery options when time is limited are all cumbersome tasks. In this situation, there is a need for a system that centrally manages scattered information and provides optimal suggestions to help users efficiently plan everything from purchasing ingredients to cooking and dining out.

[0832] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0833] In this invention, the server includes means for collecting flyer information from online retail stores, means for receiving inventory information in the refrigerator, means for obtaining food purchase information based on logistics lead times, means for receiving weather forecast information, means for obtaining family composition and event date information, means for analyzing this information and providing optimal recipes and food purchase advice, means for suggesting optimal recipes based on the ingredients in the user's refrigerator and the weather forecast, means for listing missing ingredients and providing information on the best place to buy them, and means for providing menu information that can be ordered online from nearby restaurants. This enables the user to efficiently manage and execute everything from purchasing ingredients to cooking and planning meals out in a unified manner.

[0834] "Online retail store flyer information" refers to special sale and discount information provided by retail stores that can be accessed via the internet.

[0835] "Refrigerator inventory information" refers to information about the types and quantities of food and products stored inside the refrigerator.

[0836] "Logistics lead time" refers to the time from the time an order is placed until delivery is completed.

[0837] "Weather forecast information" refers to predicted weather conditions and includes data such as temperature, probability of precipitation, and wind speed.

[0838] "Family structure" refers to information such as the number of members in the user's household and their relationships.

[0839] "Event date information" refers to information about special days for the user, such as birthdays and anniversaries.

[0840] An "optimal recipe" refers to a cooking method that is optimized for the user's ingredient inventory and individual circumstances.

[0841] "Food purchase advice" refers to guidance on how users can efficiently purchase the ingredients they need.

[0842] "Food in the user's refrigerator" refers to food stored in the refrigerator owned by the user.

[0843] "Missing ingredients" refers to ingredients that are necessary to make the suggested recipe, but are not present in the refrigerator.

[0844] "Optimal purchasing information" refers to information about stores and online services where you can obtain the necessary ingredients in the most efficient and economical way.

[0845] "Menu information available for online ordering from nearby restaurants" refers to information about menus that can be ordered online from restaurants near the user's place of residence.

[0846] "Method of providing recipe videos" refers to a method of distributing videos showing how to prepare a dish to users.

[0847] "Food delivery information" refers to information about services that deliver meals to your home or a designated location.

[0848] "Takeout information" refers to information about menus and options that users can pick up at the store.

[0849] This document describes the system configuration and processing procedures necessary to implement this invention. This system mainly consists of three main components: a server, a terminal, and a user.

[0850] Overall system configuration

[0851] server

[0852] The server is responsible for collecting and analyzing the following data.

[0853] We collect flyer information from online retail stores via the internet.

[0854] Receive inventory information from an IoT refrigerator.

[0855] We obtain logistics lead time and inventory information from partner online supermarkets and grocery delivery sites.

[0856] Obtain the latest weather information from the weather forecast API.

[0857] We collect takeout information and menu information from partner restaurants and food service providers.

[0858] The system retrieves information about the user's family structure and event dates (such as birthdays and anniversaries) that they have entered in advance.

[0859] terminal

[0860] Devices (primarily smartphones and tablets) provide information to users and interact with them.

[0861] The server notifies users of recipe video links and ingredient purchase information received from the server.

[0862] It receives input from the user (selection of optional services and feedback) and sends it to the server.

[0863] User

[0864] Users will use the system to streamline their daily grocery shopping and cooking.

[0865] Enter information such as family composition and event dates in advance.

[0866] Review the system's suggested recipes, shopping advice, and dining options, and implement them as needed.

[0867] Provide feedback.

[0868] Description of the technology and process used

[0869] Data collection

[0870] The server uses Python to collect weather forecasts, online flyer information, and IoT refrigerator inventory information via APIs. The Requests library is one of the libraries used.

[0871] Data Analysis

[0872] The server uses Python scripts to analyze data stored in a MySQL database. Specifically, it creates a list of seasonal ingredients by comparing inventory information in the refrigerator with online retail store flyers, and then selects the most suitable recipe considering weather forecasts, family size, and event dates.

[0873] Information provision

[0874] The results analyzed on the server are provided to the user through the smartphone application "Smart Cook." This application is developed using React Native and notifies users of recipe video links and ingredient purchase information through its user interface.

[0875] A concrete example of its use would be: if you have tomatoes and cheese in your refrigerator but are missing other ingredients, the system will check the weather forecast and, if sunny weather is expected to continue, suggest a salad recipe using tomatoes and cheese, and prompt you to order the missing ingredients from an online supermarket. If you're short on time, it will also suggest salad delivery from a nearby restaurant.

[0876] Example of a prompt

[0877] A user is planning a family BBQ this weekend. They have chicken and vegetables in the refrigerator, and the weather forecast is sunny. Based on seasonal ingredients and other data, please suggest the best recipe, where to buy any missing ingredients, and a suitable restaurant delivery menu for a BBQ.

[0878] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0879] Step 1:

[0880] The server uses a means of collecting flyer information from online retail stores to obtain special sale and discount information via API. The input data is flyer information obtained from online retail stores, and the output data is the latest flyer information stored on the server. This flyer information is stored in a database.

[0881] Step 2:

[0882] The server receives inventory information from the IoT refrigerator through a connection with the refrigerator itself. The input data consists of the types and quantities of food items in the refrigerator, while the output data is the refrigerator inventory information stored on the server. The refrigerator inventory information is updated in the database in real time.

[0883] Step 3:

[0884] The server retrieves logistics lead time and inventory information from partner online supermarkets and grocery delivery sites. Input data consists of inventory status and delivery lead time information from these sites, while output data is this information stored on the server. This allows the server to understand which groceries are available for purchase and their delivery schedules.

[0885] Step 4:

[0886] The server obtains the latest weather information using a weather forecast API. The input data is weather information from the weather forecast API, and the output data is weather forecast information stored on the server. This weather information is also stored in a database and used for analysis.

[0887] Step 5:

[0888] The server collects takeout and menu information from partner restaurants and food service providers. Input data consists of menus and takeout information from restaurants, while output data is this information stored on the server. This information is also updated in the database in real time.

[0889] Step 6:

[0890] The server retrieves information about family structure and event dates (such as birthdays and anniversaries) that the user has entered in advance. The input data consists of family structure and event information from the user, and the output data is this information stored on the server. This information is stored in the database as a user profile.

[0891] Step 7:

[0892] The server performs data analysis based on the collected information. Input data includes flyer information, refrigerator inventory information, logistics lead times, weather forecasts, takeout information, family composition, and event information. Output data includes optimal recipes and ingredient purchasing advice. The server analyzes this data to generate, for example, a list of seasonal ingredients and recipes tailored to the weather.

[0893] Step 8:

[0894] The server sends the analysis results to the terminal and notifies the user. The input data consists of recipes and ingredient purchase information analyzed by the server, and the output data consists of notification messages sent to the terminal. The terminal has an interface that provides the user with recipe video links and ingredient purchase information, and displays this in real time.

[0895] Step 9:

[0896] The user reviews and inputs recipes, shopping advice, and dining options suggested by the system. The input data consists of the user's selected recipes and service options, while the output data is the execution request sent to the server. Based on this, services such as grocery shopping or delivery are executed.

[0897] Step 10:

[0898] The server receives user feedback and analyzes it to improve service quality. Input data consists of user feedback, while output data provides insights for service improvement and informs future suggestions. The feedback is stored in a database and used for future recipe suggestions and service improvements.

[0899] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0900] To implement this invention, the following system and its processing procedure are described. This system provides more accurate recipe and ingredient purchasing advice by combining it with an emotion engine that recognizes the user's emotions.

[0901] Overall system configuration

[0902] This system primarily consists of three main components: the server, the terminal, and the user. The server plays a central role in data collection, analysis, and proposals, while the terminal provides information and interacts with the user. The user is the system's user, providing input and feedback. Furthermore, this system incorporates an emotion engine.

[0903] Server Processing

[0904] Data collection

[0905] The server uses the internet to collect the following information:

[0906] 1. Obtain flyer information from online retail stores. This ensures that discount and special offer information is always up-to-date.

[0907] 2. The system periodically receives inventory information from the IoT refrigerator. This information is used to keep track of the user's food inventory.

[0908] 3. Obtain inventory information and logistics lead times from partner online supermarkets and grocery delivery sites. This will allow us to understand which groceries are available for purchase and their delivery schedules.

[0909] 4. Obtain the latest weather information from the weather forecast API.

[0910] 5. Collect takeout information and menu information from partner restaurants and food service providers.

[0911] 6. Obtain information about family structure and event dates (such as birthdays and anniversaries) that the user has entered in advance.

[0912] Data Analysis

[0913] The server performs multiple data analyses based on the collected information.

[0914] 1. Compare the inventory information obtained from the refrigerator with the information from online retail store flyers to create a list of seasonal ingredients.

[0915] 2. The system selects the optimal recipe by considering factors such as the weather forecast, family structure, event dates, and the user's emotional state as determined by the emotion engine. For example, if the user is feeling stressed, the system will suggest a recipe using ingredients that promote relaxation.

[0916] 3. Compare the ingredients required for the selected recipe with the ingredients the user has in their refrigerator and list any missing ingredients.

[0917] 4. Select the best source (online supermarket or e-commerce site) for any missing ingredients and calculate logistics lead times and pricing information.

[0918] 5. Analyze available takeout and dining-in options and suggest them to users.

[0919] Information provision

[0920] The server sends the analysis results to the terminal. Specifically, it provides the following information:

[0921] 1. Video link to the recommended recipe.

[0922] 2. Required ingredients and information on where to purchase them (price, availability, delivery lead time).

[0923] 3. Information suggesting takeout and dining-out options.

[0924] Emotional Engine Processing

[0925] emotion recognition

[0926] 1. The emotion engine analyzes the user's facial expressions, voice tone, and natural language text input through the device to recognize the user's emotional state (joy, sadness, stress, etc.).

[0927] 2. The recognized emotional state is sent to the server and used as part of the data analysis.

[0928] Emotion-based adjustment

[0929] 1. Based on data from the emotion engine, the server adjusts recipe and ingredient purchase advice. For example, if the user is feeling tired, it will suggest easy-to-make recipes and ingredients.

[0930] Terminal processing

[0931] User Interface

[0932] The device (smartphone or tablet) is primarily responsible for user interaction.

[0933] 1. Notify users of recipe video links and ingredient purchase information received from the server.

[0934] 2. Receive user input (selection of optional services and feedback) and send it to the server.

[0935] 3. Utilize an emotion engine to recognize and analyze the user's emotional state.

[0936] Optional Services

[0937] If a user selects a paid optional service (such as shopping assistance, housekeeping assistance, or restaurant reservations), the terminal forwards the request to the server. The server processes the request and sends it to the appropriate service provider.

[0938] User actions

[0939] Information entry

[0940] Users input information such as family composition and event dates into the system. They also update food inventory information via the IoT refrigerator.

[0941] Acceptance of proposals and feedback

[0942] Users review and implement recipe suggestions, ingredient purchase information, and dining-out options received from the system. They then provide feedback on the services they used.

[0943] Specific example: How to cope with stressful days

[0944] 1. The user inputs information into the device in natural language, indicating that they are stressed at work. The emotion engine analyzes this information and recognizes the user's emotional state.

[0945] 2. Based on data from the emotion engine, the server selects recipes using ingredients that have a relaxing effect. For example, it might suggest herbal teas or dishes that promote relaxation.

[0946] 3. The server lists the missing ingredients and provides information on the best place to purchase them.

[0947] 4. The device notifies the user of the suggested recipe video link and purchase information.

[0948] 5. The user accepts the system's suggestions and uses the shopping assistance service as needed.

[0949] 6. The server processes the request and instructs a partner grocery delivery service to purchase the ingredients.

[0950] This specific processing flow allows users to efficiently perform a series of tasks, from purchasing ingredients to cooking and dining out, resulting in a significant reduction in the time and effort required for daily meal preparation. Furthermore, the emotion engine takes into account the user's emotional state to provide personalized and highly satisfying service.

[0951] The following describes the processing flow.

[0952] Step 1: Enter user information

[0953] Users input their family structure, food preferences, and event dates into the system. The system also synchronizes with an IoT refrigerator, sending current food inventory information to the system.

[0954] Step 2: Data Collection (Refrigerator)

[0955] The terminal retrieves current inventory information from the IoT refrigerator and sends it to the server. The information about the food items in the refrigerator includes the type of food, quantity, and expiration date.

[0956] Step 3: Data Collection (Weather Forecast)

[0957] The server uses a weather forecast API to collect the latest weather information. This weather information includes local temperature, probability of precipitation, and humidity.

[0958] Step 4: Data Collection (Market Information)

[0959] The server collects flyer information, inventory status, and price information from online retail stores and online supermarkets via the internet.

[0960] Step 5: Data Collection (Restaurant and Takeout Information)

[0961] The server collects information from partner restaurants and food service providers. This includes menu items, prices, takeout availability, and reservation status.

[0962] Step 6: Data Collection (Sentimental Information)

[0963] The device's emotion engine analyzes the user's facial expressions, voice tone, and natural language text input to recognize the user's emotional state. The recognized emotional state is then sent to the server.

[0964] Step 7: Data Analysis (Recipe Selection)

[0965] The server performs multiple data analyses based on refrigerator inventory information, market information, weather forecasts, family composition, event dates, and the user's emotional state to generate optimal recipe suggestions. For example, if the user is feeling stressed, it will suggest recipes using ingredients that have a relaxing effect.

[0966] Step 8: Check inventory and list any missing ingredients.

[0967] The server compares the list of ingredients required for the recipe with the inventory information in the refrigerator and creates a list of missing ingredients.

[0968] Step 9: Selecting a supplier

[0969] The server searches for the best store to purchase the missing ingredients. It matches prices, inventory, and logistics lead times from online retail stores and online supermarkets to determine the most efficient source of purchase.

[0970] Step 10: Provide Information

[0971] The device will notify the user's smartphone or tablet of the following information:

[0972] Recommended recipe video link

[0973] A list of necessary ingredients and information on where to buy them (price, availability, delivery lead time)

[0974] Takeout and dining-out suggestions

[0975] Step 11: Select Optional Services

[0976] Users select the option they want to use from paid services such as shopping assistance, housekeeping assistance, and restaurant reservations, and send a request to the server via their device.

[0977] Step 12: Request a service

[0978] The server sends a request to a partner service provider based on the user's request. For grocery shopping, it provides a list of selected groceries from various suppliers. For housekeeping services, it sends the specific tasks to be done and the desired date and time. For restaurant reservations, it processes the reservation based on the user's desired date and time and the restaurant's information.

[0979] Step 13: Gathering Feedback

[0980] The device collects user feedback and sends it to the server. Users enter their ratings and opinions on the provided recipes and services.

[0981] Step 14: Analyzing Feedback and Learning

[0982] The server analyzes the collected feedback and uses it to improve the accuracy of the system's suggestions. The feedback data is then incorporated into future recipe and service suggestions.

[0983] As a concrete example, consider a day when the user is experiencing a lot of stress:

[0984] 1. The user enters into the terminal in natural language that they are experiencing work-related stress.

[0985] 2. The emotion engine analyzes that information and recognizes the user's emotional state.

[0986] 3. Based on data from the emotion engine, the server selects recipes that use ingredients with relaxing effects.

[0987] 4. The server lists the missing ingredients and provides information on the best place to purchase them.

[0988] 5. The device notifies the user of the suggested recipe video link and purchase information.

[0989] 6. The user accepts the system's suggestions and uses the shopping assistance service as needed.

[0990] 7. The server processes the request and instructs a partner grocery delivery service to purchase the ingredients.

[0991] This specific processing flow allows users to efficiently perform a series of tasks, from purchasing ingredients to cooking and dining out, resulting in a significant reduction in the time and effort required for daily meal preparation. The emotion engine takes the user's emotional state into account when making suggestions, enabling a more personalized and satisfying service.

[0992] (Example 2)

[0993] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0994] In today's busy lifestyle, users often spend a lot of time and effort preparing their daily meals. Furthermore, there is no system that suggests ingredients or recipes based on emotional states or schedules, nor does it offer meal suggestions tailored to individual user circumstances. Therefore, there is a need for a system that increases user satisfaction and allows for efficient meal preparation.

[0995] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0996] In this invention, the server includes means for collecting flyer information from online retail stores, means for receiving inventory information in the refrigerator, means for obtaining food purchase information based on logistics lead times, means for receiving weather forecast information, means for obtaining family composition and event date information, means for analyzing and recognizing the user's emotional state, and means for analyzing this information and emotional state to provide optimal recipes and food purchase advice. This makes it possible to select ingredients and suggest recipes that are tailored to the user's individual circumstances, enabling efficient and satisfying meal preparation.

[0997] "Online retail store flyer information" refers to special sale information and advertising data for products offered by retail stores, obtained via the internet.

[0998] "Refrigerator inventory information" refers to specific information about food items and products stored inside a refrigerator, obtained using IoT technology and other methods.

[0999] "Logistics lead time" is an indicator that shows the time it takes from ordering ingredients or products to delivery.

[1000] "Weather forecast information" refers to data that shows changes in weather and forecasts, and is obtained through APIs, etc.

[1001] "Family structure information" refers to information such as the members of the user's household, their number, and their ages.

[1002] "Event date information" refers to information about specific dates that are important to the user or their family (e.g., birthdays, anniversaries, etc.).

[1003] "Emotional state" refers to data that indicates the user's current emotional state (joy, sadness, stress, etc.).

[1004] "Recipe and ingredient purchasing advice" refers to suggestions and advice provided to users regarding how to prepare dishes, the necessary ingredients, and how to purchase them.

[1005] "Methods for providing recipe videos" refer to methods and technologies for providing users with cooking instructions in video format.

[1006] "Takeout information" refers to information about meals available for takeout from restaurants and other establishments.

[1007] "Restaurant information" refers to information about the menus and services offered by restaurants.

[1008] A "server" is a central computer system used for collecting, analyzing, storing, and distributing data.

[1009] A "terminal" refers to a device (e.g., a smartphone or tablet) that a user uses to input or receive information.

[1010] This invention relates to a system that provides optimal recipes and ingredient purchasing advice tailored to the individual circumstances of the user. This system mainly consists of three components: a server, a terminal, and the user.

[1011] Server Processing

[1012] The server plays a central role in data collection, analysis, and proposal development. First, the server collects the following information via the internet:

[1013] 1. Collecting flyer information from retail stores: The server uses web scraping tools such as Python's BeautifulSoup and Selenium to retrieve the latest flyer information from online retail stores.

[1014] 2. Obtaining inventory information from the IoT refrigerator: The server periodically receives inventory information from the refrigerator via the IoT refrigerator's API.

[1015] 3. Information acquisition from online supermarkets and grocery delivery sites: The server uses the APIs of each online supermarket and grocery delivery site to acquire inventory information and logistics lead times.

[1016] 4. Obtaining weather forecast information: The server uses weather forecast APIs such as the OpenWeatherMap API to obtain the latest weather information.

[1017] 5. Information gathering from partner restaurants and food service providers: The server uses the APIs of each service to collect takeout information and menu information from partner restaurants and food service providers.

[1018] Next, the server analyzes the collected data:

[1019] 1. Creating a list of seasonal ingredients: Using the Python Pandas library, we will create a list of seasonal ingredients by matching inventory information in the refrigerator with information from retail store flyers.

[1020] 2. Recipe Selection: Using a generative AI model (e.g., GPT model), the optimal recipe is selected considering weather forecasts, family structure, event dates, and emotional states determined by an emotion engine.

[1021] 3. Listing ingredients: Perform a difference calculation on the list data and compare the required ingredients for the selected recipe with the ingredients in the refrigerator to list any missing ingredients.

[1022] 4. Provision of supplier information: An optimization algorithm that takes logistics lead time into consideration will be executed to select the best supplier for the missing ingredients, and the calculation will include price information.

[1023] 5. Suggesting dining options: Using an ML model, we analyze available takeout and dining options and suggest them to users in a ranked format.

[1024] Emotional Engine Processing

[1025] The emotion engine recognizes the user's emotional state and uses it as part of data analysis. Specifically, it performs the following processes:

[1026] 1. Analysis of emotion recognition data: Using an emotion recognition model that utilizes deep learning, we analyze the user's facial expressions, voice tone, and natural language text sent through the device.

[1027] 2. Transmission of emotional data: The recognized emotional state is sent to the server and used to provide advice on recipe selection and ingredient purchases.

[1028] Terminal processing

[1029] The device (smartphone or tablet) is primarily responsible for user interaction:

[1030] 1. Information Notification: Notify users of recipe video links and ingredient purchase information received from the server. Use push notification functionality on smartphones and tablets.

[1031] 2. Receiving user input: User input (selection of optional services and feedback) is received via the user interface and sent to the server via the REST API.

[1032] 3. Emotional State Analysis: An emotion recognition model is executed in real time to recognize and analyze the user's emotional state.

[1033] User actions

[1034] The user performs the following actions:

[1035] 1. Information Input: Information such as family composition and event dates is entered into the system using a dedicated application or terminal interface. Additionally, food inventory information is updated via the IoT refrigerator.

[1036] 2. Review and Feedback on Suggestions: Review and implement recipe suggestions, ingredient purchase information, and dining-out options received from the system. Afterwards, provide feedback on the services used.

[1037] Specific example

[1038] For example, if a user says, "Today was very stressful," the emotion engine analyzes this input and recognizes that the user is feeling stressed. Based on this emotion data, the server selects a relaxing herbal tea and an easy-to-make recipe, and provides information on any missing ingredients. It also sends a corresponding video link to the device and notifies the user. The user can then review the suggested recipe and ingredient list and, if necessary, utilize a shopping assistance service to efficiently purchase ingredients and prepare the meal.

[1039] Example of a prompt

[1040] "I don't know what to cook today."

[1041] "Please tell me some recommended recipes using ingredients I have in my refrigerator."

[1042] "I'm feeling stressed, so please suggest some dishes that will help me relax."

[1043] This allows users to receive ingredient selections and recipe suggestions tailored to their individual circumstances, enabling efficient and satisfying meal preparation.

[1044] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1045] Processing steps

[1046] Step 1:

[1047] The server collects flyer information from online retail stores. Specifically, it uses web scraping tools such as Python's BeautifulSoup and Selenium. Using these tools, it retrieves special offer information and advertising data from each retail store's website. The input is the URL of the retail store's website, and the output is the collected flyer information data.

[1048] Step 2:

[1049] The server receives inventory information from the IoT refrigerator. This is done using the IoT refrigerator's API. The input is an API request from the IoT refrigerator, and the output is inventory information for each food item detected inside the refrigerator. This information is stored in JSON format.

[1050] Step 3:

[1051] The server retrieves inventory information and logistics lead times from partner online supermarkets and grocery delivery sites. It uses the APIs of each service to collect this information. The input is the API endpoint of the online supermarket or grocery delivery site, and the output is inventory information and its corresponding logistics lead time.

[1052] Step 4:

[1053] The server obtains the latest weather information through a weather forecast API. Specifically, it uses services such as the OpenWeatherMap API. The input is a request to the weather forecast API, and the output is weather information data. This data includes detailed information such as temperature, probability of precipitation, and wind speed.

[1054] Step 5:

[1055] The server collects takeout and menu information from partner restaurants and food service providers. It retrieves the necessary information through each store's API. The input is the API endpoint of the partner restaurants and food service providers, and the output is takeout and menu information.

[1056] Step 6:

[1057] The server retrieves family structure and event date information previously entered by the user from the database. It uses databases such as SQLite or MySQL for searching. The input is a query string, and the output is the corresponding family structure and event date information.

[1058] Step 7:

[1059] The server matches the retrieved inventory information from the refrigerator with information from retail store flyers to create a list of seasonal ingredients. The Python Pandas library is used for data matching. The input is the refrigerator inventory information and the flyer information, and the output is a list of seasonal ingredients.

[1060] Step 8:

[1061] The server selects the optimal recipe by considering the weather forecast, family structure, event dates, and the user's emotional state as determined by the emotion engine. It suggests recipes using a generative AI model (e.g., GPT model). The input is this information and emotion data, and the output is the recommended recipe.

[1062] Step 9:

[1063] The server compares the required ingredients for the selected recipe with the inventory in the refrigerator and lists any missing ingredients. A Python data processing script is used to calculate the difference. The input is a list of required ingredients for the recipe and the inventory information in the refrigerator, and the output is a list of missing ingredients.

[1064] Step 10:

[1065] The server selects the optimal supplier (online supermarket or e-commerce site) for the missing ingredients and calculates logistics lead time and price information. It retrieves this data using the API of each supplier and runs an optimization algorithm. The input is a list of missing ingredients and inventory and price information for each supplier, and the output is information on the optimal supplier.

[1066] Step 11:

[1067] The server uses an ML model to analyze available takeout and dining-in options and suggests them to the user in a ranked format. The input is the user's location information and data on partner restaurants, and the output is a ranked list of takeout and dining-in options.

[1068] Step 12:

[1069] The emotion engine analyzes the user's facial expressions, voice tone, and text input through the device to recognize the user's emotional state. It uses an emotion recognition model based on deep learning. The input is the user's facial expression data and voice data, and the output is the recognized emotional state.

[1070] Step 13:

[1071] Emotional data from the emotion engine is sent to the server and used for recipe selection and ingredient purchasing advice. The input is recognized emotional data, and the output is the input data necessary for analysis performed by the server.

[1072] Step 14:

[1073] The device notifies the user of recipe video links and ingredient purchase information received from the server. Specifically, it uses the push notification function of smartphones and tablets. The input is analysis result data from the server, and the output is notifications to the user.

[1074] Step 15:

[1075] The terminal receives user input (selection of optional services and feedback) via the user interface and sends it to the server via a REST API. Input is user input data, and output is data sent to the server.

[1076] Step 16:

[1077] The device utilizes an emotion engine to run an emotion recognition model in real time, recognizing and analyzing the user's emotional state. The input is real-time facial and voice data, and the output is the recognized emotional state.

[1078] Step 17:

[1079] Users input information such as family composition and event dates into the system using a dedicated application or terminal interface. They also update food inventory information via an IoT refrigerator. Input consists of manually entered data from the user and data from the IoT refrigerator, while output is user information stored on the server.

[1080] Step 18:

[1081] Users review recipe suggestions, ingredient purchase information, and dining-out options received from the system and choose whether to implement them. They then provide feedback on the services used. Input is the suggestion data provided by the server, and output is the user's selections and feedback data.

[1082] (Application Example 2)

[1083] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[1084] In modern life, users spend a significant amount of time and effort purchasing ingredients and selecting appropriate recipes amidst their busy schedules. Furthermore, a lack of advice and product suggestions that take into account users' emotional states often leads to stress during the ingredient purchasing and cooking processes. Therefore, there is a need for a system that can adapt to users' emotional states and provide efficient and personalized service.

[1085] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[1086] In this invention, the server includes means for collecting flyer information from online retail stores, means for receiving inventory information in the refrigerator, means for obtaining food purchase information based on logistics lead times, means for receiving weather forecast information, means for obtaining family composition and event date information, means for recognizing the user's emotional state, and means for analyzing this information to provide optimal recipes and food purchase advice tailored to the user's emotional state. This enables efficient and personalized recipe suggestions and food purchase advice tailored to the user's emotional state.

[1087] "Means of collecting flyer information from online retail stores" refers to a function that collects information on discounts and special offers provided by online retail stores via the internet.

[1088] "Means of receiving inventory information inside the refrigerator" refers to a function that obtains inventory information of food currently stored inside the refrigerator from a device such as an IoT refrigerator.

[1089] "Method for obtaining food purchase information based on logistics lead time" refers to a function that obtains information to enable users to make optimal purchases by considering the delivery schedule and lead time of food ingredients.

[1090] "Means of receiving weather forecast information" refers to the function of obtaining information about current and future weather from the internet.

[1091] "Means of obtaining family structure and event date information" refers to a function that retrieves information about the number and composition of family members, as well as information about special dates such as birthdays and anniversaries, which have been entered by the user in advance.

[1092] "Means of recognizing the user's emotional state" refers to a function that analyzes the user's facial expressions, tone of voice, text input, etc., to determine their emotional state at any given time.

[1093] "A means of providing optimal recipes and ingredient purchasing advice tailored to emotions" refers to a function that, based on collected information and emotional states, provides users with the most suitable recipes and ingredient purchasing advice that is sensitive to their emotions.

[1094] The "recipe video provision method" is a function that provides users with links and information to cooking videos related to the selected recipe.

[1095] A "shopping assistant based on emotion recognition" is a function that suggests the most suitable products to purchase in a physical store based on the user's emotional state.

[1096] "Means for collecting takeout and dining-out information" refers to the function of collecting takeout and menu information provided by restaurants and dining-out services.

[1097] "A means of suggesting the optimal product" refers to a function that suggests the most suitable product for the user based on the information acquired.

[1098] Modes for carrying out the invention

[1099] The system necessary to implement this invention consists of three entities: a server, a terminal, and a user. The server plays a central role in data collection, analysis, and proposal, while the terminal provides information and interacts with the user. The user is the user of the system and is the entity that provides input and feedback.

[1100] The server includes the following measures:

[1101] 1. Means of collecting online retail store flyer information: This system has the function of regularly collecting discount and special offer information provided by various retail stores via the internet. For example, it uses an API to obtain the latest flyer information from each store.

[1102] 2. Means of receiving inventory information from the refrigerator: Accurate inventory information is received in real time from the IoT refrigerator. This allows the user to understand the status of food items in their refrigerator.

[1103] 3. Method for obtaining food ingredient purchase information based on logistics lead time: Food ingredient purchase information, including delivery schedules and lead times, is obtained based on logistics data. This makes it possible to suggest appropriate purchase timings.

[1104] 4. Means of receiving weather forecast information: Obtain the latest weather forecast via API and reflect it in the user's ingredient selection and recipe suggestions.

[1105] 5. Means for obtaining family structure and event date information: The system has a function to store family structure, birthdays, anniversaries, and other event information registered by the user in advance in a database and retrieve that information.

[1106] 6. Means for recognizing the user's emotional state: The user's facial expressions and voice tone are analyzed via the camera and microphone to determine their emotional state. An emotion recognition algorithm is used for this analysis.

[1107] 7. Means of providing optimal recipes and emotionally appropriate ingredient purchasing advice: Based on the collected information, we will provide users with optimal recipes and emotionally appropriate ingredient purchasing advice.

[1108] Specific examples and your hardware and software

[1109] The server is built using the Flask framework with Python, and the emotion recognition function utilizes the EmotionRecognition library. The server collects necessary information through APIs on the internet and performs analysis based on that data. Specifically, when a user inputs emotional information such as "I'm tired" using their smartphone's camera and microphone, EmotionRecognition analyzes that information and selects recipes that include ingredients with relaxing effects.

[1110] The device (smartphone or tablet) functions as an application that notifies the user of information received from the server. This application provides the user with selected recipe videos and optimal ingredient purchasing advice, and also functions as an emotion-based shopping assistant. For example, it might suggest relaxing scented candles or relaxation foods to a user who is feeling stressed.

[1111] Users periodically input and update their emotional information, family structure, and refrigerator inventory information into the system. This allows the system to always provide advice based on the most up-to-date information.

[1112] Example of a prompt

[1113] "We will provide image and audio data. Please analyze this user's emotions. If the emotion is 'stressed,' suggest products that promote relaxation; if it's 'sad,' suggest products that offer comfort; and for other emotions, suggest products that generally support health."

[1114] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1115] Step 1:

[1116] Data collection

[1117] The server uses online APIs to collect information such as retail store flyers, weather forecasts, inventory information from partner online supermarkets and grocery delivery sites, and menu information from partner restaurants and food service providers. Inputs include data from various APIs, and output is a dataset integrating this data. The specific operation involves sending requests to each API and saving the returned information to the database.

[1118] Step 2:

[1119] Receiving inventory information from the refrigerator

[1120] The server receives inventory information periodically transmitted from the IoT refrigerator. The input is inventory data transmitted from the IoT refrigerator, and the output is the latest inventory status data in the user's refrigerator. Specifically, the server receives data from the IoT refrigerator in real time and updates the database.

[1121] Step 3:

[1122] Recognizing the user's emotional state

[1123] The device (smartphone) uses its camera and microphone to collect the user's facial expressions and voice tone. The collected data is sent to a server and analyzed by an emotion recognition algorithm (EmotionRecognition). The input is the user's image and audio data, and the output is the user's emotional state as a result of the analysis (e.g., stress, joy, sadness). The specific operation is to pass the data captured by the camera and microphone to EmotionRecognition and estimate the emotional state.

[1124] Step 4:

[1125] Data analysis and proposal generation

[1126] The server analyzes the collected data and generates optimal recipes and ingredient purchase advice, taking into account the user's refrigerator inventory, weather forecast, family structure, event dates, and recognized emotional state. The input consists of various collected data and recognized emotional states, while the output is the optimal recipe and ingredient purchase advice based on the analysis. Specifically, it uses an algorithm to combine various data and determine the most suitable recipe and purchase advice.

[1127] Step 5:

[1128] Providing information

[1129] The server sends the analysis results to the terminal and notifies the user. Specifically, it provides links to recipe videos, information on necessary ingredients and where to purchase them, and suggestions for takeout and dining out. The input is the analysis result data, and the output is the information displayed on the user's terminal. The specific operation is to send the analysis results to the terminal in an appropriate format and display them to the user.

[1130] Step 6:

[1131] User interaction

[1132] Users view suggested recipes and ingredient purchase information through their terminals and select paid optional services, such as grocery shopping assistance, as needed. Input is the user's selection data, and output is request data related to the selected service. The specific operation involves the user making selections via touch operations and sending that data to the server.

[1133] Step 7:

[1134] Arranging a shopping assistance service

[1135] The server processes user requests and sends grocery purchase requests to affiliated shopping agents. The input is the user's request data, and the output is the request data sent to the shopping agent. Specifically, it parses the request data and sends the request details to the specified shopping agent's API.

[1136] Example of a prompt

[1137] "We will provide image and audio data. Please analyze this user's emotions. If the emotion is 'stressed,' suggest products that promote relaxation; if it's 'sad,' suggest products that offer comfort; and for other emotions, suggest products that generally support health."

[1138] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1139] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1140] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[1141] [Third Embodiment]

[1142] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[1143] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1144] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1145] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[1146] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1147] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1148] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1149] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1150] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1151] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1152] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1153] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[1154] To carry out this invention, the following system and its processing procedure will be described.

[1155] Overall system configuration

[1156] This system primarily consists of three main components: the server, the terminal, and the user. The server plays a central role in data collection, analysis, and proposal development, while the terminal provides information and interacts with the user. The user is the system's user, providing input and feedback.

[1157] Server Processing

[1158] Data collection

[1159] The server uses the internet to collect the following information:

[1160] 1. Obtain flyer information from online retail stores. This ensures that discount and special offer information is always up-to-date.

[1161] 2. The system periodically receives inventory information from the IoT refrigerator. This information is used to keep track of the user's food inventory.

[1162] 3. Obtain inventory information and logistics lead times from partner online supermarkets and grocery delivery sites. This will allow us to understand which groceries are available for purchase and their delivery schedules.

[1163] 4. Obtain the latest weather information from the weather forecast API.

[1164] 5. Collect takeout information and menu information from partner restaurants and food service providers.

[1165] 6. Obtain information about family structure and event dates (such as birthdays and anniversaries) that the user has entered in advance.

[1166] Data Analysis

[1167] The server performs multiple data analyses based on the collected information.

[1168] 1. Compare the inventory information obtained from the refrigerator with the information from online retail store flyers to create a list of seasonal ingredients.

[1169] 2. Select the optimal recipe by considering parameters such as weather forecasts, family composition, and event dates.

[1170] 3. Compare the ingredients required for the selected recipe with the ingredients the user has in their refrigerator and list any missing ingredients.

[1171] 4. Select the best source (online supermarket or e-commerce site) for any missing ingredients and calculate logistics lead times and pricing information.

[1172] 5. Analyze available takeout and dining-in options and suggest them to users.

[1173] Information provision

[1174] The server sends the analysis results to the terminal. Specifically, it provides the following information:

[1175] 1. Video link to the recommended recipe.

[1176] 2. Required ingredients and information on where to purchase them (price, availability, delivery lead time).

[1177] 3. Information suggesting takeout and dining-out options.

[1178] Terminal processing

[1179] User Interface

[1180] The device (smartphone or tablet) is primarily responsible for user interaction.

[1181] 1. Notify users of recipe video links and ingredient purchase information received from the server.

[1182] 2. Receive user input (selection of optional services and feedback) and send it to the server.

[1183] Optional Services

[1184] If a user selects a paid optional service (such as shopping assistance, housekeeping assistance, or restaurant reservations), the terminal forwards the request to the server. The server processes the request and sends it to the appropriate service provider.

[1185] User actions

[1186] Information entry

[1187] Users input information such as family composition and event dates into the system. They also update food inventory information via the IoT refrigerator.

[1188] Acceptance of proposals and feedback

[1189] Users review and implement recipe suggestions, ingredient purchase information, and dining-out options received from the system. They then provide feedback on the services they used.

[1190] Specific example: When preparing dinner for the family

[1191] 1. The user enters into the system that their child's birthday is approaching. The system also updates the inventory information in the refrigerator.

[1192] 2. The server suggests party recipes based on the inventory information in the refrigerator, a list of seasonal ingredients, and the weather forecast.

[1193] 3. The server lists the missing ingredients and provides information on the best place to purchase them.

[1194] 4. The device notifies the user of the suggested recipe video link and purchase information.

[1195] 5. The user accepts the system's suggestions and uses the shopping assistance service as needed.

[1196] 6. The server processes the request and instructs a partner grocery delivery service to purchase the ingredients.

[1197] This invention allows users to efficiently perform a series of tasks, from purchasing ingredients to cooking and eating out, and as a result, significantly reduce the effort and time required for daily meal preparation.

[1198] The following describes the processing flow.

[1199] Step 1: Enter user information

[1200] Users input their family structure, food preferences, and event dates into the system. They also connect to an IoT refrigerator to synchronize current food inventory information.

[1201] Step 2: Data Collection (Refrigerator)

[1202] The terminal retrieves current inventory information from the IoT refrigerator and sends it to the server. The information about the food items in the refrigerator includes the type of food, quantity, and expiration date.

[1203] Step 3: Data Collection (Weather Forecast)

[1204] The server uses a weather forecast API to collect the latest weather information. This weather information includes local temperature, probability of precipitation, and humidity.

[1205] Step 4: Data Collection (Market Information)

[1206] The server collects flyer information, inventory status, and price information from online retail stores and online supermarkets via the internet.

[1207] Step 5: Data Collection (Restaurant and Takeout Information)

[1208] The server collects information from partner restaurants and food service providers. This includes menu items, prices, takeout availability, and reservation status.

[1209] Step 6: Data Analysis (Recipe Selection)

[1210] The server analyzes the collected data and generates optimal recipe suggestions based on the user's preferences, event dates, and weather forecasts. For example, it might suggest a warm stew recipe on a rainy day and a barbecue recipe on a sunny day.

[1211] Step 7: Check inventory and list any missing ingredients.

[1212] The server compares the list of ingredients required for the recipe with the inventory information in the refrigerator and creates a list of missing ingredients.

[1213] Step 8: Selecting a supplier

[1214] The server searches for the best store to purchase the missing ingredients. It compares prices, inventory, and delivery lead times from online retail stores and online supermarkets to determine the most efficient source of supplies.

[1215] Step 9: Information Provision

[1216] The device will notify the user's smartphone or tablet of the following information:

[1217] Recommended recipe video link

[1218] A list of necessary ingredients and information on where to buy them (price, availability, delivery lead time)

[1219] Takeout and dining-out suggestions

[1220] Step 10: Selecting Optional Services

[1221] Users select the option they want to use from paid services such as shopping assistance, housekeeping assistance, and restaurant reservations, and send a request to the server via their device.

[1222] Step 11: Service Request

[1223] The server sends a request to a partner service provider based on the user's request. For grocery shopping, it provides a list of selected groceries from various suppliers. For housekeeping services, it sends the specific tasks to be done and the desired date and time. For restaurant reservations, it processes the reservation based on the user's desired date and time and the restaurant's information.

[1224] Step 12: Gathering Feedback

[1225] The device collects user feedback and sends it to the server. Users enter their ratings and opinions on the provided recipes and services.

[1226] Step 13: Analyzing Feedback and Learning

[1227] The server analyzes the collected feedback and uses it to improve the accuracy of the system's suggestions. The feedback data is then reflected in future recipe and service suggestions.

[1228] This specific processing flow will allow users to efficiently manage everything from meal preparation and grocery shopping to dining out.

[1229] (Example 1)

[1230] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1231] In today's busy lifestyle, efficiently managing and purchasing groceries, as well as preparing meals, is a significant burden for many. In particular, there is a need for a system that centrally manages and provides information such as grocery inventory, optimal recipes, weather forecasts, and takeout options, but such a comprehensive system does not yet exist. As a result, users spend a great deal of time and effort on this, making it difficult to live an efficient daily life.

[1232] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1233] In this invention, the server includes means for collecting flyer information from online retail stores, means for receiving inventory information in the refrigerator, means for obtaining food purchase information based on logistics lead times, means for receiving weather forecast information, means for obtaining family composition and event date information, means for analyzing this information and providing optimal recipes and food purchase advice, means for providing video links to recommended recipes, means for collecting takeout information and menu information from affiliated restaurants and food service providers, user interface means for receiving input from the user and transmitting it to the server, and means for creating a list of missing ingredients and calculating prices and inventory information for the best suppliers. This enables the user to efficiently manage and purchase ingredients, cook, and choose where to eat out.

[1234] An "online retail store" is a website or application established for the purpose of selling goods online.

[1235] "Flyer information" refers to digital promotional materials used in retail stores, including information on special offers, discounts, and campaigns.

[1236] "Refrigerator inventory information" refers to data showing the types, quantities, and expiration dates of food items currently stored in a household refrigerator.

[1237] "Logistics lead time" refers to the time it takes from the time an order is placed until the product is actually delivered.

[1238] "Weather forecast information" refers to data that predicts future weather conditions, including temperature, probability of precipitation, and wind speed.

[1239] "Family structure" refers to information indicating the number of people in the user's household, their age distribution, and their relationships.

[1240] "Event date information" refers to information that users have entered, such as birthdays, anniversaries, and dates of special events.

[1241] The "optimal recipe" is a list of cooking steps and ingredients that best suit the user's needs and circumstances, based on the collected data.

[1242] "Food purchase advice" refers to recommendations regarding how to purchase the ingredients a user needs, the best places to buy them, and their prices.

[1243] The "recommended recipe video link" is a link to access video content related to the selected recipe.

[1244] "Partner restaurants and food service providers" refer to restaurants and food service providers that collaborate with the system to share information and provide services.

[1245] "Takeout information" refers to the take-out menus offered by restaurants and related details.

[1246] "Menu information" refers to a list of dishes and drinks offered by a restaurant, along with detailed information about them.

[1247] "User interface means" refers to the means by which a user interacts with a system, and includes devices such as smartphones and tablets.

[1248] The "list of missing ingredients" is a list of ingredients needed for the selected recipe that are not currently in the refrigerator.

[1249] The "optimal supplier" refers to a place to purchase food ingredients that has been evaluated based on criteria such as price, stock availability, and logistics lead time.

[1250] Modes for carrying out the invention

[1251] To implement this invention, three main entities are primarily used: a server, a terminal, and a user. The server plays a central role in data collection, analysis, and proposal, while the terminal provides information and interacts with the user. The user is the system user and the entity that provides input and feedback.

[1252] Server Processing

[1253] Data collection

[1254] The server collects the following information via the internet:

[1255] 1. Obtaining flyer information from online retail stores.

[1256] The server uses scraping technology to periodically access the websites of online retail stores and automatically retrieve the latest flyer information. This ensures that discount and special offer information is always up-to-date.

[1257] 2. Receiving inventory information from the IoT refrigerator.

[1258] The server periodically receives inventory information from IoT refrigerators installed in the user's home. MQTT and HTTP are used as communication protocols. This allows the server to monitor the food inventory inside the refrigerator.

[1259] 3. Obtain inventory information and logistics lead times from online supermarkets and food delivery websites.

[1260] The server calls APIs from partner online supermarkets and grocery delivery sites to obtain the latest inventory information and logistics lead times.

[1261] 4. Obtaining weather information from a weather forecast API

[1262] The server periodically calls the weather forecast API to retrieve the latest weather information. Authentication is performed using the API key.

[1263] 5. Obtaining takeout information and menu information from partner restaurants and food service providers.

[1264] The server calls APIs from partner restaurants and food service providers to retrieve the latest takeout and menu information.

[1265] 6. Retrieving family structure and event date information entered by the user.

[1266] Users enter information such as family composition and event dates (birthdays, anniversaries, etc.) through the application's input form. This information is sent to the server and stored in the database.

[1267] Data Analysis

[1268] Based on the information collected above, the server performs the following data analysis:

[1269] 1. Create a list of seasonal ingredients.

[1270] The server compares the retrieved inventory information from the refrigerator with online retail store flyers to create a list of seasonal ingredients. This process uses the Python Pandas library.

[1271] 2. Selecting the optimal recipe

[1272] The server selects the optimal recipe by considering parameters such as weather forecasts, family composition, and event dates. This recipe selection is performed using an AI model (e.g., TensorFlow or PyTorch).

[1273] 3. List of missing ingredients

[1274] The system compares the required ingredients for the selected recipe with the user's refrigerator inventory and lists any missing ingredients. The Numpy library is used for the comparison process.

[1275] 4. Selecting the optimal supplier and calculating price and logistics lead time.

[1276] For any missing ingredients, the server selects the optimal supplier and calculates the total cost based on price information and logistics lead time. Linear programming may be used.

[1277] 5. Analysis of takeout and dining-in options

[1278] The system also analyzes available takeout and dining-in options and suggests them to the user. Suggestions are generated based on past user preference data.

[1279] Information provision

[1280] The server sends the analysis results to the terminal. Specifically, it provides the following information:

[1281] 1. Recommended recipe video link

[1282] The server generates a video link for the selected recipe and sends it to the device. The user can view the video by clicking the link.

[1283] 2. Required ingredients and where to buy them

[1284] The server compiles the necessary ingredients and their supplier information (price, availability, delivery lead time) and sends it to the terminal.

[1285] 3. Information on takeout and dining out options.

[1286] Based on the analysis results, the server sends suggested takeout and dining-in options to the terminal.

[1287] Terminal processing

[1288] User Interface

[1289] The device (smartphone or tablet) is responsible for user interaction:

[1290] 1. Sending notifications

[1291] The system notifies users of recipe video links and ingredient purchase information received from the server. Notification methods include push notifications and in-app notifications.

[1292] 2. Receiving and sending user input

[1293] The terminal receives input from the user (for example, selection of optional services or feedback) and sends it to the server.

[1294] Provision of optional services

[1295] If a user selects a paid optional service (such as shopping assistance, housekeeping assistance, or restaurant reservations), the terminal forwards the request to the server. The server processes the request and sends it to the appropriate service provider.

[1296] User actions

[1297] Information entry

[1298] Users input information about their family structure and event dates (e.g., birthdays and anniversaries) into the system. They can also update their food inventory information via the IoT refrigerator.

[1299] Acceptance of proposals and feedback

[1300] Users review and implement recipe suggestions, ingredient purchase information, and dining options received from the system. They then provide feedback on the services they used.

[1301] Specific example: When preparing dinner for the family

[1302] 1. The user enters into the system that their child's birthday is approaching. The system also updates the inventory information in the refrigerator.

[1303] 2. The server suggests party recipes based on the inventory information in the refrigerator, a list of seasonal ingredients, and the weather forecast.

[1304] 3. The server lists the missing ingredients and provides information on the best place to purchase them.

[1305] 4. The device notifies the user of the suggested recipe video link and purchase information.

[1306] 5. The user accepts the system's suggestions and uses the shopping assistance service as needed.

[1307] 6. The server processes the request and instructs a partner grocery delivery service to purchase the ingredients.

[1308] Using this specific example, users can efficiently manage and purchase ingredients, cook, and even choose restaurants, completing a series of tasks. As a result, the time and effort involved in daily meal preparation can be significantly reduced.

[1309] Examples of prompts to input into a generative AI model are as follows:

[1310] "My child's birthday is approaching. Please update the ingredients in the refrigerator and suggest party recipes based on online retail stores and weather forecasts. Based on the suggestions, please list any missing ingredients and provide information on the best places to buy them (price, availability, delivery lead time)."

[1311] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1312] Step 1: Data Collection

[1313] The server collects necessary information from multiple sources. Specifically, it accesses the websites of online retail stores and uses scraping techniques to obtain the latest flyer information. This flyer information includes data on discounts and special offers.

[1314] Input: URL of a retail store's website on the internet

[1315] Processing: Use scraping techniques to obtain flyer information and save it to a database.

[1316] Output: Update of flyer information database

[1317] Specifically, it uses Python libraries such as BeautifulSoup and Scrapy to analyze the HTML structure of a web page and extract the necessary information.

[1318] Step 2: Obtain inventory information from the refrigerator

[1319] The server periodically receives inventory information from IoT refrigerators installed in the user's home. MQTT and HTTP are used as communication protocols.

[1320] Input: Inventory information sent from IoT refrigerator

[1321] Processing: Analyze inventory information and save it to the database.

[1322] Output: Refrigerator inventory database update

[1323] Specifically, the IoT refrigerator sends inventory information to the cloud at regular intervals, and a server retrieves and analyzes this data.

[1324] Step 3: Obtain logistics lead time and inventory information.

[1325] The server calls APIs from partner online supermarkets and grocery delivery sites to obtain the latest inventory information and logistics lead times.

[1326] Input: API key and endpoint for online supermarkets and grocery delivery sites.

[1327] Processing: Make an API call and save the retrieved inventory information and logistics lead time to the database.

[1328] Output: Update of inventory and logistics lead time databases

[1329] Specifically, the process involves making HTTPS requests using a RESTful API, parsing the data returned in JSON format, and saving it.

[1330] Step 4: Gathering weather forecast information

[1331] The server periodically calls the weather forecast API to retrieve the latest weather information. Authentication is performed using the API key.

[1332] Input: API key and endpoint for the weather forecast API.

[1333] Process: Make an API call and save the retrieved weather information to the database.

[1334] Output: Weather forecast database update

[1335] Specifically, the process involves making an HTTPS request using an API client and then parsing the returned JSON data.

[1336] Step 5: Gathering information on partner restaurants

[1337] The server calls APIs from partner restaurants and food service providers to retrieve the latest takeout and menu information.

[1338] Input: API key and endpoint for partner restaurants and food service providers

[1339] Processing: Make an API call and save the retrieved takeout information and menu information to the database.

[1340] Output: Updates to the takeout and menu databases.

[1341] In terms of specific operations, it will similarly make HTTPS requests using a RESTful API, parse the data, and save it.

[1342] Step 6: Obtaining User Information

[1343] Users enter information such as family composition and event dates (birthdays, anniversaries, etc.) through the application's input form. This information is sent to the server and stored in the database.

[1344] Input: Family composition and event date information entered by the user.

[1345] Processing: Parse the input information and save it to the database.

[1346] Output: User information database update

[1347] In terms of specific operation, a POST request is sent from the frontend input form to the server, and the data is received and analyzed on the backend.

[1348] Step 7: Data Analysis and Recipe Selection

[1349] The server performs data analysis based on the collected information and selects the optimal recipe that matches the user's conditions. It uses AI models (e.g., TensorFlow, PyTorch).

[1350] Input: Refrigerator inventory information, flyer information, weather forecast, family composition, event date information

[1351] Processing: Use an AI model to select the optimal recipe.

[1352] Output: Selected recipes

[1353] In terms of specific operations, data is input into the AI ​​model, and the optimal recipe is obtained as a result of the model's prediction.

[1354] Step 8: List the missing ingredients

[1355] The server compares the ingredients required for the selected recipe with the ingredients currently in the refrigerator and lists any missing ingredients.

[1356] Input: Selected recipe, refrigerator inventory information

[1357] Processing: Use the Numpy library to list the missing ingredients.

[1358] Output: List of missing ingredients

[1359] Specifically, the process involves using a data frame to compare required ingredients with current inventory and extracting any missing ingredients.

[1360] Step 9: Selecting the best supplier and calculating the price.

[1361] The server selects the best supplier for any missing ingredients and calculates the total cost based on price information and logistics lead time.

[1362] Input: List of missing ingredients, online supermarket inventory information, price information, logistics lead time

[1363] Process: Use linear programming to calculate the optimal supplier and total cost.

[1364] Output: Optimal purchasing information and total cost

[1365] Specifically, the program performs linear programming using the Scipy library.

[1366] Step 10: Provide a link to a recommended recipe video.

[1367] The server generates links to video content related to the selected recipe and sends them to the device.

[1368] Input: Selected recipe

[1369] Processing: Generate video link and notify device.

[1370] Output: Recommended recipe video link notification

[1371] Specifically, the system retrieves video links related to the recipe from the database and notifies the user.

[1372] Step 11: Notifying and providing feedback to the user

[1373] The device notifies the user based on information received from the server. It also receives feedback from the user.

[1374] Input: Notification information from the server, user feedback

[1375] Processing: Sending notifications, receiving feedback and forwarding it to the server.

[1376] Output: User notification and feedback sent to the server

[1377] Specifically, the system sends push notifications via a mobile application and receives user feedback.

[1378] Step 12: Requesting and processing optional services

[1379] The server processes user requests for paid optional services and forwards the requests to the appropriate service providers.

[1380] Input: User's optional service request

[1381] Processing: Processing the request and forwarding the request to the service provider.

[1382] Output: Request to service provider

[1383] In terms of specific operation, the API is used to send a corresponding request to the service provider.

[1384] (Application Example 1)

[1385] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1386] In today's busy lifestyle, meal preparation has become a significant burden for users. In particular, keeping track of what's in the refrigerator, planning appropriate recipes, purchasing necessary ingredients, and finding restaurant or delivery options when time is limited are all cumbersome tasks. In this situation, there is a need for a system that centrally manages scattered information and provides optimal suggestions to help users efficiently plan everything from purchasing ingredients to cooking and dining out.

[1387] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1388] In this invention, the server includes means for collecting flyer information from online retail stores, means for receiving inventory information in the refrigerator, means for obtaining food purchase information based on logistics lead times, means for receiving weather forecast information, means for obtaining family composition and event date information, means for analyzing this information and providing optimal recipes and food purchase advice, means for suggesting optimal recipes based on the ingredients in the user's refrigerator and the weather forecast, means for listing missing ingredients and providing information on the best place to buy them, and means for providing menu information that can be ordered online from nearby restaurants. This enables the user to efficiently manage and execute everything from purchasing ingredients to cooking and planning meals out in a unified manner.

[1389] "Online retail store flyer information" refers to special sale and discount information provided by retail stores that can be accessed via the internet.

[1390] "Refrigerator inventory information" refers to information about the types and quantities of food and products stored inside the refrigerator.

[1391] "Logistics lead time" refers to the time from the time an order is placed until delivery is completed.

[1392] "Weather forecast information" refers to predicted weather conditions and includes data such as temperature, probability of precipitation, and wind speed.

[1393] "Family structure" refers to information such as the number of members in the user's household and their relationships.

[1394] "Event date information" refers to information about special days for the user, such as birthdays and anniversaries.

[1395] An "optimal recipe" refers to a cooking method that is optimized for the user's ingredient inventory and individual circumstances.

[1396] "Food purchase advice" refers to guidance on how users can efficiently purchase the ingredients they need.

[1397] "Food in the user's refrigerator" refers to food stored in the refrigerator owned by the user.

[1398] "Missing ingredients" refers to ingredients that are necessary to make the suggested recipe, but are not present in the refrigerator.

[1399] "Optimal purchasing information" refers to information about stores and online services where you can obtain the necessary ingredients in the most efficient and economical way.

[1400] "Menu information available for online ordering from nearby restaurants" refers to information about menus that can be ordered online from restaurants near the user's place of residence.

[1401] "Method of providing recipe videos" refers to a method of distributing videos showing how to prepare a dish to users.

[1402] "Food delivery information" refers to information about services that deliver meals to your home or a designated location.

[1403] "Takeout information" refers to information about menus and options that users can pick up at the store.

[1404] This document describes the system configuration and processing procedures necessary to implement this invention. This system mainly consists of three main components: a server, a terminal, and a user.

[1405] Overall system configuration

[1406] server

[1407] The server is responsible for collecting and analyzing the following data.

[1408] We collect flyer information from online retail stores via the internet.

[1409] Receive inventory information from an IoT refrigerator.

[1410] We obtain logistics lead time and inventory information from partner online supermarkets and grocery delivery sites.

[1411] Obtain the latest weather information from the weather forecast API.

[1412] We collect takeout information and menu information from partner restaurants and food service providers.

[1413] The system retrieves information about the user's family structure and event dates (such as birthdays and anniversaries) that they have entered in advance.

[1414] terminal

[1415] Devices (primarily smartphones and tablets) provide information to users and interact with them.

[1416] The server notifies users of recipe video links and ingredient purchase information received from the server.

[1417] It receives input from the user (selection of optional services and feedback) and sends it to the server.

[1418] User

[1419] Users will use the system to streamline their daily grocery shopping and cooking.

[1420] Enter information such as family composition and event dates in advance.

[1421] Review the system's suggested recipes, shopping advice, and dining options, and implement them as needed.

[1422] Provide feedback.

[1423] Description of the technology and process used

[1424] Data collection

[1425] The server uses Python to collect weather forecasts, online flyer information, and IoT refrigerator inventory information via APIs. The Requests library is one of the libraries used.

[1426] Data Analysis

[1427] The server uses Python scripts to analyze data stored in a MySQL database. Specifically, it creates a list of seasonal ingredients by comparing inventory information in the refrigerator with online retail store flyers, and then selects the most suitable recipe considering weather forecasts, family size, and event dates.

[1428] Information provision

[1429] The results analyzed on the server are provided to the user through the smartphone application "Smart Cook." This application is developed using React Native and notifies users of recipe video links and ingredient purchase information through its user interface.

[1430] A concrete example of its use would be: if you have tomatoes and cheese in your refrigerator but are missing other ingredients, the system will check the weather forecast and, if sunny weather is expected to continue, suggest a salad recipe using tomatoes and cheese, and prompt you to order the missing ingredients from an online supermarket. If you're short on time, it will also suggest salad delivery from a nearby restaurant.

[1431] Example of a prompt

[1432] A user is planning a family BBQ this weekend. They have chicken and vegetables in the refrigerator, and the weather forecast is sunny. Based on seasonal ingredients and other data, please suggest the best recipe, where to buy any missing ingredients, and a suitable restaurant delivery menu for a BBQ.

[1433] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1434] Step 1:

[1435] The server uses a means of collecting flyer information from online retail stores to obtain special sale and discount information via API. The input data is flyer information obtained from online retail stores, and the output data is the latest flyer information stored on the server. This flyer information is stored in a database.

[1436] Step 2:

[1437] The server receives inventory information from the IoT refrigerator through a connection with the refrigerator itself. The input data consists of the types and quantities of food items in the refrigerator, while the output data is the refrigerator inventory information stored on the server. The refrigerator inventory information is updated in the database in real time.

[1438] Step 3:

[1439] The server retrieves logistics lead time and inventory information from partner online supermarkets and grocery delivery sites. Input data consists of inventory status and delivery lead time information from these sites, while output data is this information stored on the server. This allows the server to understand which groceries are available for purchase and their delivery schedules.

[1440] Step 4:

[1441] The server obtains the latest weather information using a weather forecast API. The input data is weather information from the weather forecast API, and the output data is weather forecast information stored on the server. This weather information is also stored in a database and used for analysis.

[1442] Step 5:

[1443] The server collects takeout and menu information from partner restaurants and food service providers. Input data consists of menus and takeout information from restaurants, while output data is this information stored on the server. This information is also updated in the database in real time.

[1444] Step 6:

[1445] The server retrieves information about family structure and event dates (such as birthdays and anniversaries) that the user has entered in advance. The input data consists of family structure and event information from the user, and the output data is this information stored on the server. This information is stored in the database as a user profile.

[1446] Step 7:

[1447] The server performs data analysis based on the collected information. Input data includes flyer information, refrigerator inventory information, logistics lead times, weather forecasts, takeout information, family composition, and event information. Output data includes optimal recipes and ingredient purchasing advice. The server analyzes this data to generate, for example, a list of seasonal ingredients and recipes tailored to the weather.

[1448] Step 8:

[1449] The server sends the analysis results to the terminal and notifies the user. The input data consists of recipes and ingredient purchase information analyzed by the server, and the output data consists of notification messages sent to the terminal. The terminal has an interface that provides the user with recipe video links and ingredient purchase information, and displays this in real time.

[1450] Step 9:

[1451] The user reviews and inputs recipes, shopping advice, and dining options suggested by the system. The input data consists of the user's selected recipes and service options, while the output data is the execution request sent to the server. Based on this, services such as grocery shopping or delivery are executed.

[1452] Step 10:

[1453] The server receives user feedback and analyzes it to improve service quality. Input data consists of user feedback, while output data provides insights for service improvement and informs future suggestions. The feedback is stored in a database and used for future recipe suggestions and service improvements.

[1454] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1455] To implement this invention, the following system and its processing procedure are described. This system provides more accurate recipe and ingredient purchasing advice by combining it with an emotion engine that recognizes the user's emotions.

[1456] Overall system configuration

[1457] This system primarily consists of three main components: the server, the terminal, and the user. The server plays a central role in data collection, analysis, and proposals, while the terminal provides information and interacts with the user. The user is the system's user, providing input and feedback. Furthermore, this system incorporates an emotion engine.

[1458] Server Processing

[1459] Data collection

[1460] The server uses the internet to collect the following information:

[1461] 1. Obtain flyer information from online retail stores. This ensures that discount and special offer information is always up-to-date.

[1462] 2. The system periodically receives inventory information from the IoT refrigerator. This information is used to keep track of the user's food inventory.

[1463] 3. Obtain inventory information and logistics lead times from partner online supermarkets and grocery delivery sites. This will allow us to understand which groceries are available for purchase and their delivery schedules.

[1464] 4. Obtain the latest weather information from the weather forecast API.

[1465] 5. Collect takeout information and menu information from partner restaurants and food service providers.

[1466] 6. Obtain information about family structure and event dates (such as birthdays and anniversaries) that the user has entered in advance.

[1467] Data Analysis

[1468] The server performs multiple data analyses based on the collected information.

[1469] 1. Compare the inventory information obtained from the refrigerator with the information from online retail store flyers to create a list of seasonal ingredients.

[1470] 2. The system selects the optimal recipe by considering factors such as the weather forecast, family structure, event dates, and the user's emotional state as determined by the emotion engine. For example, if the user is feeling stressed, the system will suggest a recipe using ingredients that promote relaxation.

[1471] 3. Compare the ingredients required for the selected recipe with the ingredients the user has in their refrigerator and list any missing ingredients.

[1472] 4. Select the best source (online supermarket or e-commerce site) for any missing ingredients and calculate logistics lead times and pricing information.

[1473] 5. Analyze available takeout and dining-in options and suggest them to users.

[1474] Information provision

[1475] The server sends the analysis results to the terminal. Specifically, it provides the following information:

[1476] 1. Video link to the recommended recipe.

[1477] 2. Required ingredients and information on where to purchase them (price, availability, delivery lead time).

[1478] 3. Information suggesting takeout and dining-out options.

[1479] Emotional Engine Processing

[1480] emotion recognition

[1481] 1. The emotion engine analyzes the user's facial expressions, voice tone, and natural language text input through the device to recognize the user's emotional state (joy, sadness, stress, etc.).

[1482] 2. The recognized emotional state is sent to the server and used as part of the data analysis.

[1483] Emotion-based adjustment

[1484] 1. Based on data from the emotion engine, the server adjusts recipe and ingredient purchase advice. For example, if the user is feeling tired, it will suggest easy-to-make recipes and ingredients.

[1485] Terminal processing

[1486] User Interface

[1487] The device (smartphone or tablet) is primarily responsible for user interaction.

[1488] 1. Notify users of recipe video links and ingredient purchase information received from the server.

[1489] 2. Receive user input (selection of optional services and feedback) and send it to the server.

[1490] 3. Utilize an emotion engine to recognize and analyze the user's emotional state.

[1491] Optional Services

[1492] If a user selects a paid optional service (such as shopping assistance, housekeeping assistance, or restaurant reservations), the terminal forwards the request to the server. The server processes the request and sends it to the appropriate service provider.

[1493] User actions

[1494] Information entry

[1495] Users input information such as family composition and event dates into the system. They also update food inventory information via the IoT refrigerator.

[1496] Acceptance of proposals and feedback

[1497] Users review and implement recipe suggestions, ingredient purchase information, and dining-out options received from the system. They then provide feedback on the services they used.

[1498] Specific example: How to cope with stressful days

[1499] 1. The user inputs information into the device in natural language, indicating that they are stressed at work. The emotion engine analyzes this information and recognizes the user's emotional state.

[1500] 2. Based on data from the emotion engine, the server selects recipes using ingredients that have a relaxing effect. For example, it might suggest herbal teas or dishes that promote relaxation.

[1501] 3. The server lists the missing ingredients and provides information on the best place to purchase them.

[1502] 4. The device notifies the user of the suggested recipe video link and purchase information.

[1503] 5. The user accepts the system's suggestions and uses the shopping assistance service as needed.

[1504] 6. The server processes the request and instructs a partner grocery delivery service to purchase the ingredients.

[1505] This specific processing flow allows users to efficiently perform a series of tasks, from purchasing ingredients to cooking and dining out, resulting in a significant reduction in the time and effort required for daily meal preparation. Furthermore, the emotion engine takes into account the user's emotional state to provide personalized and highly satisfying service.

[1506] The following describes the processing flow.

[1507] Step 1: Enter user information

[1508] Users input their family structure, food preferences, and event dates into the system. The system also synchronizes with an IoT refrigerator, sending current food inventory information to the system.

[1509] Step 2: Data Collection (Refrigerator)

[1510] The terminal retrieves current inventory information from the IoT refrigerator and sends it to the server. The information about the food items in the refrigerator includes the type of food, quantity, and expiration date.

[1511] Step 3: Data Collection (Weather Forecast)

[1512] The server uses a weather forecast API to collect the latest weather information. This weather information includes local temperature, probability of precipitation, and humidity.

[1513] Step 4: Data Collection (Market Information)

[1514] The server collects flyer information, inventory status, and price information from online retail stores and online supermarkets via the internet.

[1515] Step 5: Data Collection (Restaurant and Takeout Information)

[1516] The server collects information from partner restaurants and food service providers. This includes menu items, prices, takeout availability, and reservation status.

[1517] Step 6: Data Collection (Sentimental Information)

[1518] The device's emotion engine analyzes the user's facial expressions, voice tone, and natural language text input to recognize the user's emotional state. The recognized emotional state is then sent to the server.

[1519] Step 7: Data Analysis (Recipe Selection)

[1520] The server performs multiple data analyses based on refrigerator inventory information, market information, weather forecasts, family composition, event dates, and the user's emotional state to generate optimal recipe suggestions. For example, if the user is feeling stressed, it will suggest recipes using ingredients that have a relaxing effect.

[1521] Step 8: Check inventory and list any missing ingredients.

[1522] The server compares the list of ingredients required for the recipe with the inventory information in the refrigerator and creates a list of missing ingredients.

[1523] Step 9: Selecting a supplier

[1524] The server searches for the best store to purchase the missing ingredients. It matches prices, inventory, and logistics lead times from online retail stores and online supermarkets to determine the most efficient source of purchase.

[1525] Step 10: Provide Information

[1526] The device will notify the user's smartphone or tablet of the following information:

[1527] Recommended recipe video link

[1528] A list of necessary ingredients and information on where to buy them (price, availability, delivery lead time)

[1529] Takeout and dining-out suggestions

[1530] Step 11: Select Optional Services

[1531] Users select the option they want to use from paid services such as shopping assistance, housekeeping assistance, and restaurant reservations, and send a request to the server via their device.

[1532] Step 12: Request a service

[1533] The server sends a request to a partner service provider based on the user's request. For grocery shopping, it provides a list of selected groceries from various suppliers. For housekeeping services, it sends the specific tasks to be done and the desired date and time. For restaurant reservations, it processes the reservation based on the user's desired date and time and the restaurant's information.

[1534] Step 13: Gathering Feedback

[1535] The device collects user feedback and sends it to the server. Users enter their ratings and opinions on the provided recipes and services.

[1536] Step 14: Analyzing Feedback and Learning

[1537] The server analyzes the collected feedback and uses it to improve the accuracy of the system's suggestions. The feedback data is then incorporated into future recipe and service suggestions.

[1538] As a concrete example, consider a day when the user is experiencing a lot of stress:

[1539] 1. The user enters into the terminal in natural language that they are experiencing work-related stress.

[1540] 2. The emotion engine analyzes that information and recognizes the user's emotional state.

[1541] 3. Based on data from the emotion engine, the server selects recipes that use ingredients with relaxing effects.

[1542] 4. The server lists the missing ingredients and provides information on the best place to purchase them.

[1543] 5. The device notifies the user of the suggested recipe video link and purchase information.

[1544] 6. The user accepts the system's suggestions and uses the shopping assistance service as needed.

[1545] 7. The server processes the request and instructs a partner grocery delivery service to purchase the ingredients.

[1546] This specific processing flow allows users to efficiently perform a series of tasks, from purchasing ingredients to cooking and dining out, resulting in a significant reduction in the time and effort required for daily meal preparation. The emotion engine takes the user's emotional state into account when making suggestions, enabling a more personalized and satisfying service.

[1547] (Example 2)

[1548] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1549] In today's busy lifestyle, users often spend a lot of time and effort preparing their daily meals. Furthermore, there is no system that suggests ingredients or recipes based on emotional states or schedules, nor does it offer meal suggestions tailored to individual user circumstances. Therefore, there is a need for a system that increases user satisfaction and allows for efficient meal preparation.

[1550] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1551] In this invention, the server includes means for collecting flyer information from online retail stores, means for receiving inventory information in the refrigerator, means for obtaining food purchase information based on logistics lead times, means for receiving weather forecast information, means for obtaining family composition and event date information, means for analyzing and recognizing the user's emotional state, and means for analyzing this information and emotional state to provide optimal recipes and food purchase advice. This makes it possible to select ingredients and suggest recipes that are tailored to the user's individual circumstances, enabling efficient and satisfying meal preparation.

[1552] "Online retail store flyer information" refers to special sale information and advertising data for products offered by retail stores, obtained via the internet.

[1553] "Refrigerator inventory information" refers to specific information about food items and products stored inside a refrigerator, obtained using IoT technology and other methods.

[1554] "Logistics lead time" is an indicator that shows the time it takes from ordering ingredients or products to delivery.

[1555] "Weather forecast information" refers to data that shows changes in weather and forecasts, and is obtained through APIs, etc.

[1556] "Family structure information" refers to information such as the members of the user's household, their number, and their ages.

[1557] "Event date information" refers to information about specific dates that are important to the user or their family (e.g., birthdays, anniversaries, etc.).

[1558] "Emotional state" refers to data that indicates the user's current emotional state (joy, sadness, stress, etc.).

[1559] "Recipe and ingredient purchasing advice" refers to suggestions and advice provided to users regarding how to prepare dishes, the necessary ingredients, and how to purchase them.

[1560] "Methods for providing recipe videos" refer to methods and technologies for providing users with cooking instructions in video format.

[1561] "Takeout information" refers to information about meals available for takeout from restaurants and other establishments.

[1562] "Restaurant information" refers to information about the menus and services offered by restaurants.

[1563] A "server" is a central computer system used for collecting, analyzing, storing, and distributing data.

[1564] A "terminal" refers to a device (e.g., a smartphone or tablet) that a user uses to input or receive information.

[1565] This invention relates to a system that provides optimal recipes and ingredient purchasing advice tailored to the individual circumstances of the user. This system mainly consists of three components: a server, a terminal, and the user.

[1566] Server Processing

[1567] The server plays a central role in data collection, analysis, and proposal development. First, the server collects the following information via the internet:

[1568] 1. Collecting flyer information from retail stores: The server uses web scraping tools such as Python's BeautifulSoup and Selenium to retrieve the latest flyer information from online retail stores.

[1569] 2. Obtaining inventory information from the IoT refrigerator: The server periodically receives inventory information from the refrigerator via the IoT refrigerator's API.

[1570] 3. Information acquisition from online supermarkets and grocery delivery sites: The server uses the APIs of each online supermarket and grocery delivery site to acquire inventory information and logistics lead times.

[1571] 4. Obtaining weather forecast information: The server uses weather forecast APIs such as the OpenWeatherMap API to obtain the latest weather information.

[1572] 5. Information gathering from partner restaurants and food service providers: The server uses the APIs of each service to collect takeout information and menu information from partner restaurants and food service providers.

[1573] Next, the server analyzes the collected data:

[1574] 1. Creating a list of seasonal ingredients: Using the Python Pandas library, we will create a list of seasonal ingredients by matching inventory information in the refrigerator with information from retail store flyers.

[1575] 2. Recipe Selection: Using a generative AI model (e.g., GPT model), the optimal recipe is selected considering weather forecasts, family structure, event dates, and emotional states determined by an emotion engine.

[1576] 3. Listing ingredients: Perform a difference calculation on the list data and compare the required ingredients for the selected recipe with the ingredients in the refrigerator to list any missing ingredients.

[1577] 4. Provision of supplier information: An optimization algorithm that takes logistics lead time into consideration will be executed to select the best supplier for the missing ingredients, and the calculation will include price information.

[1578] 5. Suggesting dining options: Using an ML model, we analyze available takeout and dining options and suggest them to users in a ranked format.

[1579] Emotional Engine Processing

[1580] The emotion engine recognizes the user's emotional state and uses it as part of data analysis. Specifically, it performs the following processes:

[1581] 1. Analysis of emotion recognition data: Using an emotion recognition model that utilizes deep learning, we analyze the user's facial expressions, voice tone, and natural language text sent through the device.

[1582] 2. Transmission of emotional data: The recognized emotional state is sent to the server and used to provide advice on recipe selection and ingredient purchases.

[1583] Terminal processing

[1584] The device (smartphone or tablet) is primarily responsible for user interaction:

[1585] 1. Information Notification: Notify users of recipe video links and ingredient purchase information received from the server. Use push notification functionality on smartphones and tablets.

[1586] 2. Receiving user input: User input (selection of optional services and feedback) is received via the user interface and sent to the server via the REST API.

[1587] 3. Emotional State Analysis: An emotion recognition model is executed in real time to recognize and analyze the user's emotional state.

[1588] User actions

[1589] The user performs the following actions:

[1590] 1. Information Input: Information such as family composition and event dates is entered into the system using a dedicated application or terminal interface. Additionally, food inventory information is updated via the IoT refrigerator.

[1591] 2. Review and Feedback on Suggestions: Review and implement recipe suggestions, ingredient purchase information, and dining-out options received from the system. Afterwards, provide feedback on the services used.

[1592] Specific example

[1593] For example, if a user says, "Today was very stressful," the emotion engine analyzes this input and recognizes that the user is feeling stressed. Based on this emotion data, the server selects a relaxing herbal tea and an easy-to-make recipe, and provides information on any missing ingredients. It also sends a corresponding video link to the device and notifies the user. The user can then review the suggested recipe and ingredient list and, if necessary, utilize a shopping assistance service to efficiently purchase ingredients and prepare the meal.

[1594] Example of a prompt

[1595] "I don't know what to cook today."

[1596] "Please tell me some recommended recipes using ingredients I have in my refrigerator."

[1597] "I'm feeling stressed, so please suggest some dishes that will help me relax."

[1598] This allows users to receive ingredient selections and recipe suggestions tailored to their individual circumstances, enabling efficient and satisfying meal preparation.

[1599] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1600] Processing steps

[1601] Step 1:

[1602] The server collects flyer information from online retail stores. Specifically, it uses web scraping tools such as Python's BeautifulSoup and Selenium. Using these tools, it retrieves special offer information and advertising data from each retail store's website. The input is the URL of the retail store's website, and the output is the collected flyer information data.

[1603] Step 2:

[1604] The server receives inventory information from the IoT refrigerator. This is done using the IoT refrigerator's API. The input is an API request from the IoT refrigerator, and the output is inventory information for each food item detected inside the refrigerator. This information is stored in JSON format.

[1605] Step 3:

[1606] The server retrieves inventory information and logistics lead times from partner online supermarkets and grocery delivery sites. It uses the APIs of each service to collect this information. The input is the API endpoint of the online supermarket or grocery delivery site, and the output is inventory information and its corresponding logistics lead time.

[1607] Step 4:

[1608] The server obtains the latest weather information through a weather forecast API. Specifically, it uses services such as the OpenWeatherMap API. The input is a request to the weather forecast API, and the output is weather information data. This data includes detailed information such as temperature, probability of precipitation, and wind speed.

[1609] Step 5:

[1610] The server collects takeout and menu information from partner restaurants and food service providers. It retrieves the necessary information through each store's API. The input is the API endpoint of the partner restaurants and food service providers, and the output is takeout and menu information.

[1611] Step 6:

[1612] The server retrieves family structure and event date information previously entered by the user from the database. It uses databases such as SQLite or MySQL for searching. The input is a query string, and the output is the corresponding family structure and event date information.

[1613] Step 7:

[1614] The server matches the retrieved inventory information from the refrigerator with information from retail store flyers to create a list of seasonal ingredients. The Python Pandas library is used for data matching. The input is the refrigerator inventory information and the flyer information, and the output is a list of seasonal ingredients.

[1615] Step 8:

[1616] The server selects the optimal recipe by considering the weather forecast, family structure, event dates, and the user's emotional state as determined by the emotion engine. It suggests recipes using a generative AI model (e.g., GPT model). The input is this information and emotion data, and the output is the recommended recipe.

[1617] Step 9:

[1618] The server compares the required ingredients for the selected recipe with the inventory in the refrigerator and lists any missing ingredients. A Python data processing script is used to calculate the difference. The input is a list of required ingredients for the recipe and the inventory information in the refrigerator, and the output is a list of missing ingredients.

[1619] Step 10:

[1620] The server selects the optimal supplier (online supermarket or e-commerce site) for the missing ingredients and calculates logistics lead time and price information. It retrieves this data using the API of each supplier and runs an optimization algorithm. The input is a list of missing ingredients and inventory and price information for each supplier, and the output is information on the optimal supplier.

[1621] Step 11:

[1622] The server uses an ML model to analyze available takeout and dining-in options and suggests them to the user in a ranked format. The input is the user's location information and data on partner restaurants, and the output is a ranked list of takeout and dining-in options.

[1623] Step 12:

[1624] The emotion engine analyzes the user's facial expressions, voice tone, and text input through the device to recognize the user's emotional state. It uses an emotion recognition model based on deep learning. The input is the user's facial expression data and voice data, and the output is the recognized emotional state.

[1625] Step 13:

[1626] Emotional data from the emotion engine is sent to the server and used for recipe selection and ingredient purchasing advice. The input is recognized emotional data, and the output is the input data necessary for analysis performed by the server.

[1627] Step 14:

[1628] The device notifies the user of recipe video links and ingredient purchase information received from the server. Specifically, it uses the push notification function of smartphones and tablets. The input is analysis result data from the server, and the output is notifications to the user.

[1629] Step 15:

[1630] The terminal receives user input (selection of optional services and feedback) via the user interface and sends it to the server via a REST API. Input is user input data, and output is data sent to the server.

[1631] Step 16:

[1632] The device utilizes an emotion engine to run an emotion recognition model in real time, recognizing and analyzing the user's emotional state. The input is real-time facial and voice data, and the output is the recognized emotional state.

[1633] Step 17:

[1634] Users input information such as family composition and event dates into the system using a dedicated application or terminal interface. They also update food inventory information via an IoT refrigerator. Input consists of manually entered data from the user and data from the IoT refrigerator, while output is user information stored on the server.

[1635] Step 18:

[1636] Users review recipe suggestions, ingredient purchase information, and dining-out options received from the system and choose whether to implement them. They then provide feedback on the services used. Input is the suggestion data provided by the server, and output is the user's selections and feedback data.

[1637] (Application Example 2)

[1638] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1639] In modern life, users spend a significant amount of time and effort purchasing ingredients and selecting appropriate recipes amidst their busy schedules. Furthermore, a lack of advice and product suggestions that take into account users' emotional states often leads to stress during the ingredient purchasing and cooking processes. Therefore, there is a need for a system that can adapt to users' emotional states and provide efficient and personalized service.

[1640] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[1641] In this invention, the server includes means for collecting flyer information from online retail stores, means for receiving inventory information in the refrigerator, means for obtaining food purchase information based on logistics lead times, means for receiving weather forecast information, means for obtaining family composition and event date information, means for recognizing the user's emotional state, and means for analyzing this information to provide optimal recipes and food purchase advice tailored to the user's emotional state. This enables efficient and personalized recipe suggestions and food purchase advice tailored to the user's emotional state.

[1642] "Means of collecting flyer information from online retail stores" refers to a function that collects information on discounts and special offers provided by online retail stores via the internet.

[1643] "Means of receiving inventory information inside the refrigerator" refers to a function that obtains inventory information of food currently stored inside the refrigerator from a device such as an IoT refrigerator.

[1644] "Method for obtaining food purchase information based on logistics lead time" refers to a function that obtains information to enable users to make optimal purchases by considering the delivery schedule and lead time of food ingredients.

[1645] "Means of receiving weather forecast information" refers to the function of obtaining information about current and future weather from the internet.

[1646] "Means of obtaining family structure and event date information" refers to a function that retrieves information about the number and composition of family members, as well as information about special dates such as birthdays and anniversaries, which have been entered by the user in advance.

[1647] "Means of recognizing the user's emotional state" refers to a function that analyzes the user's facial expressions, tone of voice, text input, etc., to determine their emotional state at any given time.

[1648] "A means of providing optimal recipes and ingredient purchasing advice tailored to emotions" refers to a function that, based on collected information and emotional states, provides users with the most suitable recipes and ingredient purchasing advice that is sensitive to their emotions.

[1649] The "recipe video provision method" is a function that provides users with links and information to cooking videos related to the selected recipe.

[1650] A "shopping assistant based on emotion recognition" is a function that suggests the most suitable products to purchase in a physical store based on the user's emotional state.

[1651] "Means for collecting takeout and dining-out information" refers to the function of collecting takeout and menu information provided by restaurants and dining-out services.

[1652] "A means of suggesting the optimal product" refers to a function that suggests the most suitable product for the user based on the information acquired.

[1653] Modes for carrying out the invention

[1654] The system necessary to implement this invention consists of three entities: a server, a terminal, and a user. The server plays a central role in data collection, analysis, and proposal, while the terminal provides information and interacts with the user. The user is the user of the system and is the entity that provides input and feedback.

[1655] The server includes the following measures:

[1656] 1. Means of collecting online retail store flyer information: This system has the function of regularly collecting discount and special offer information provided by various retail stores via the internet. For example, it uses an API to obtain the latest flyer information from each store.

[1657] 2. Means of receiving inventory information from the refrigerator: Accurate inventory information is received in real time from the IoT refrigerator. This allows the user to understand the status of food items in their refrigerator.

[1658] 3. Method for obtaining food ingredient purchase information based on logistics lead time: Food ingredient purchase information, including delivery schedules and lead times, is obtained based on logistics data. This makes it possible to suggest appropriate purchase timings.

[1659] 4. Means of receiving weather forecast information: Obtain the latest weather forecast via API and reflect it in the user's ingredient selection and recipe suggestions.

[1660] 5. Means for obtaining family structure and event date information: The system has a function to store family structure, birthdays, anniversaries, and other event information registered by the user in advance in a database and retrieve that information.

[1661] 6. Means for recognizing the user's emotional state: The user's facial expressions and voice tone are analyzed via the camera and microphone to determine their emotional state. An emotion recognition algorithm is used for this analysis.

[1662] 7. Means of providing optimal recipes and emotionally appropriate ingredient purchasing advice: Based on the collected information, we will provide users with optimal recipes and emotionally appropriate ingredient purchasing advice.

[1663] Specific examples and your hardware and software

[1664] The server is built using the Flask framework with Python, and the emotion recognition function utilizes the EmotionRecognition library. The server collects necessary information through APIs on the internet and performs analysis based on that data. Specifically, when a user inputs emotional information such as "I'm tired" using their smartphone's camera and microphone, EmotionRecognition analyzes that information and selects recipes that include ingredients with relaxing effects.

[1665] The device (smartphone or tablet) functions as an application that notifies the user of information received from the server. This application provides the user with selected recipe videos and optimal ingredient purchasing advice, and also functions as an emotion-based shopping assistant. For example, it might suggest relaxing scented candles or relaxation foods to a user who is feeling stressed.

[1666] Users periodically input and update their emotional information, family structure, and refrigerator inventory information into the system. This allows the system to always provide advice based on the most up-to-date information.

[1667] Example of a prompt

[1668] "We will provide image and audio data. Please analyze this user's emotions. If the emotion is 'stressed,' suggest products that promote relaxation; if it's 'sad,' suggest products that offer comfort; and for other emotions, suggest products that generally support health."

[1669] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1670] Step 1:

[1671] Data collection

[1672] The server uses online APIs to collect information such as retail store flyers, weather forecasts, inventory information from partner online supermarkets and grocery delivery sites, and menu information from partner restaurants and food service providers. Inputs include data from various APIs, and output is a dataset integrating this data. The specific operation involves sending requests to each API and saving the returned information to the database.

[1673] Step 2:

[1674] Receiving inventory information from the refrigerator

[1675] The server receives inventory information periodically transmitted from the IoT refrigerator. The input is inventory data transmitted from the IoT refrigerator, and the output is the latest inventory status data in the user's refrigerator. Specifically, the server receives data from the IoT refrigerator in real time and updates the database.

[1676] Step 3:

[1677] Recognizing the user's emotional state

[1678] The device (smartphone) uses its camera and microphone to collect the user's facial expressions and voice tone. The collected data is sent to a server and analyzed by an emotion recognition algorithm (EmotionRecognition). The input is the user's image and audio data, and the output is the user's emotional state as a result of the analysis (e.g., stress, joy, sadness). The specific operation is to pass the data captured by the camera and microphone to EmotionRecognition and estimate the emotional state.

[1679] Step 4:

[1680] Data analysis and proposal generation

[1681] The server analyzes the collected data and generates optimal recipes and ingredient purchase advice, taking into account the user's refrigerator inventory, weather forecast, family structure, event dates, and recognized emotional state. The input consists of various collected data and recognized emotional states, while the output is the optimal recipe and ingredient purchase advice based on the analysis. Specifically, it uses an algorithm to combine various data and determine the most suitable recipe and purchase advice.

[1682] Step 5:

[1683] Information provision

[1684] The server sends the analysis results to the terminal and notifies the user. Specifically, it provides links to recipe videos, information on necessary ingredients and where to purchase them, and suggestions for takeout and dining out. The input is the analysis result data, and the output is the information displayed on the user's terminal. The specific operation is to send the analysis results to the terminal in an appropriate format and display them to the user.

[1685] Step 6:

[1686] User interaction

[1687] Users view suggested recipes and ingredient purchase information through their terminals and select paid optional services, such as grocery shopping assistance, as needed. Input is the user's selection data, and output is request data related to the selected service. The specific operation involves the user making selections via touch operations and sending that data to the server.

[1688] Step 7:

[1689] Arranging a shopping assistance service

[1690] The server processes user requests and sends grocery purchase requests to affiliated shopping agents. The input is the user's request data, and the output is the request data sent to the shopping agent. Specifically, it parses the request data and sends the request details to the specified shopping agent's API.

[1691] Example of a prompt

[1692] "We will provide image and audio data. Please analyze this user's emotions. If the emotion is 'stressed,' suggest products that promote relaxation; if it's 'sad,' suggest products that offer comfort; and for other emotions, suggest products that generally support health."

[1693] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1694] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1695] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[1696] [Fourth Embodiment]

[1697] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[1698] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1699] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1700] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[1701] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1702] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1703] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1704] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[1705] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1706] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1707] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1708] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1709] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1710] To carry out this invention, the following system and its processing procedure will be described.

[1711] Overall system configuration

[1712] This system primarily consists of three main components: the server, the terminal, and the user. The server plays a central role in data collection, analysis, and proposal development, while the terminal provides information and interacts with the user. The user is the system's user, providing input and feedback.

[1713] Server Processing

[1714] Data collection

[1715] The server uses the internet to collect the following information:

[1716] 1. Obtain flyer information from online retail stores. This ensures that discount and special offer information is always up-to-date.

[1717] 2. The system periodically receives inventory information from the IoT refrigerator. This information is used to keep track of the user's food inventory.

[1718] 3. Obtain inventory information and logistics lead times from partner online supermarkets and grocery delivery sites. This will allow us to understand which groceries are available for purchase and their delivery schedules.

[1719] 4. Obtain the latest weather information from the weather forecast API.

[1720] 5. Collect takeout information and menu information from partner restaurants and food service providers.

[1721] 6. Obtain information about family structure and event dates (such as birthdays and anniversaries) that the user has entered in advance.

[1722] Data Analysis

[1723] The server performs multiple data analyses based on the collected information.

[1724] 1. Compare the inventory information obtained from the refrigerator with the information from online retail store flyers to create a list of seasonal ingredients.

[1725] 2. Select the optimal recipe by considering parameters such as weather forecasts, family composition, and event dates.

[1726] 3. Compare the ingredients required for the selected recipe with the ingredients the user has in their refrigerator and list any missing ingredients.

[1727] 4. Select the best source (online supermarket or e-commerce site) for any missing ingredients and calculate logistics lead times and pricing information.

[1728] 5. Analyze available takeout and dining-in options and suggest them to users.

[1729] Information provision

[1730] The server sends the analysis results to the terminal. Specifically, it provides the following information:

[1731] 1. Video link to the recommended recipe.

[1732] 2. Required ingredients and information on where to purchase them (price, availability, delivery lead time).

[1733] 3. Information suggesting takeout and dining-out options.

[1734] Terminal processing

[1735] User Interface

[1736] The device (smartphone or tablet) is primarily responsible for user interaction.

[1737] 1. Notify users of recipe video links and ingredient purchase information received from the server.

[1738] 2. Receive user input (selection of optional services and feedback) and send it to the server.

[1739] Optional Services

[1740] If a user selects a paid optional service (such as shopping assistance, housekeeping assistance, or restaurant reservations), the terminal forwards the request to the server. The server processes the request and sends it to the appropriate service provider.

[1741] User actions

[1742] Information entry

[1743] Users input information such as family composition and event dates into the system. They also update food inventory information via the IoT refrigerator.

[1744] Acceptance of proposals and feedback

[1745] Users review and implement recipe suggestions, ingredient purchase information, and dining-out options received from the system. They then provide feedback on the services they used.

[1746] Specific example: When preparing dinner for the family

[1747] 1. The user enters into the system that their child's birthday is approaching. The system also updates the inventory information in the refrigerator.

[1748] 2. The server suggests party recipes based on the inventory information in the refrigerator, a list of seasonal ingredients, and the weather forecast.

[1749] 3. The server lists the missing ingredients and provides information on the best place to purchase them.

[1750] 4. The device notifies the user of the suggested recipe video link and purchase information.

[1751] 5. The user accepts the system's suggestions and uses the shopping assistance service as needed.

[1752] 6. The server processes the request and instructs a partner grocery delivery service to purchase the ingredients.

[1753] This invention allows users to efficiently perform a series of tasks, from purchasing ingredients to cooking and eating out, and as a result, significantly reduce the effort and time required for daily meal preparation.

[1754] The following describes the processing flow.

[1755] Step 1: Enter user information

[1756] Users input their family structure, food preferences, and event dates into the system. They also connect to an IoT refrigerator to synchronize current food inventory information.

[1757] Step 2: Data Collection (Refrigerator)

[1758] The terminal retrieves current inventory information from the IoT refrigerator and sends it to the server. The information about the food items in the refrigerator includes the type of food, quantity, and expiration date.

[1759] Step 3: Data Collection (Weather Forecast)

[1760] The server uses a weather forecast API to collect the latest weather information. This weather information includes local temperature, probability of precipitation, and humidity.

[1761] Step 4: Data Collection (Market Information)

[1762] The server collects flyer information, inventory status, and price information from online retail stores and online supermarkets via the internet.

[1763] Step 5: Data Collection (Restaurant and Takeout Information)

[1764] The server collects information from partner restaurants and food service providers. This includes menu items, prices, takeout availability, and reservation status.

[1765] Step 6: Data Analysis (Recipe Selection)

[1766] The server analyzes the collected data and generates optimal recipe suggestions based on the user's preferences, event dates, and weather forecasts. For example, it might suggest a warm stew recipe on a rainy day and a barbecue recipe on a sunny day.

[1767] Step 7: Check inventory and list any missing ingredients.

[1768] The server compares the list of ingredients required for the recipe with the inventory information in the refrigerator and creates a list of missing ingredients.

[1769] Step 8: Selecting a supplier

[1770] The server searches for the best store to purchase the missing ingredients. It compares prices, inventory, and delivery lead times from online retail stores and online supermarkets to determine the most efficient source of supplies.

[1771] Step 9: Information Provision

[1772] The device will notify the user's smartphone or tablet of the following information:

[1773] Recommended recipe video link

[1774] A list of necessary ingredients and information on where to buy them (price, availability, delivery lead time)

[1775] Takeout and dining-out suggestions

[1776] Step 10: Selecting Optional Services

[1777] Users select the option they want to use from paid services such as shopping assistance, housekeeping assistance, and restaurant reservations, and send a request to the server via their device.

[1778] Step 11: Service Request

[1779] The server sends a request to a partner service provider based on the user's request. For grocery shopping, it provides a list of selected groceries from various suppliers. For housekeeping services, it sends the specific tasks to be done and the desired date and time. For restaurant reservations, it processes the reservation based on the user's desired date and time and the restaurant's information.

[1780] Step 12: Gathering Feedback

[1781] The device collects user feedback and sends it to the server. Users enter their ratings and opinions on the provided recipes and services.

[1782] Step 13: Analyzing Feedback and Learning

[1783] The server analyzes the collected feedback and uses it to improve the accuracy of the system's suggestions. The feedback data is then reflected in future recipe and service suggestions.

[1784] This specific processing flow will allow users to efficiently manage everything from meal preparation and grocery shopping to dining out.

[1785] (Example 1)

[1786] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1787] In today's busy lifestyle, efficiently managing and purchasing groceries, as well as preparing meals, is a significant burden for many. In particular, there is a need for a system that centrally manages and provides information such as grocery inventory, optimal recipes, weather forecasts, and takeout options, but such a comprehensive system does not yet exist. As a result, users spend a great deal of time and effort on this, making it difficult to live an efficient daily life.

[1788] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1789] In this invention, the server includes means for collecting flyer information from online retail stores, means for receiving inventory information in the refrigerator, means for obtaining food purchase information based on logistics lead times, means for receiving weather forecast information, means for obtaining family composition and event date information, means for analyzing this information and providing optimal recipes and food purchase advice, means for providing video links to recommended recipes, means for collecting takeout information and menu information from affiliated restaurants and food service providers, user interface means for receiving input from the user and transmitting it to the server, and means for creating a list of missing ingredients and calculating prices and inventory information for the best suppliers. This enables the user to efficiently manage and purchase ingredients, cook, and choose where to eat out.

[1790] An "online retail store" is a website or application established for the purpose of selling goods online.

[1791] "Flyer information" refers to digital promotional materials used in retail stores, including information on special offers, discounts, and campaigns.

[1792] "Refrigerator inventory information" refers to data showing the types, quantities, and expiration dates of food items currently stored in a household refrigerator.

[1793] "Logistics lead time" refers to the time it takes from the time an order is placed until the product is actually delivered.

[1794] "Weather forecast information" refers to data that predicts future weather conditions, including temperature, probability of precipitation, and wind speed.

[1795] "Family structure" refers to information indicating the number of people in the user's household, their age distribution, and their relationships.

[1796] "Event date information" refers to information that users have entered, such as birthdays, anniversaries, and dates of special events.

[1797] The "optimal recipe" is a list of cooking steps and ingredients that best suit the user's needs and circumstances, based on the collected data.

[1798] "Food purchase advice" refers to recommendations regarding how to purchase the ingredients a user needs, the best places to buy them, and their prices.

[1799] The "recommended recipe video link" is a link to access video content related to the selected recipe.

[1800] "Partner restaurants and food service providers" refer to restaurants and food service providers that collaborate with the system to share information and provide services.

[1801] "Takeout information" refers to the take-out menus offered by restaurants and related details.

[1802] "Menu information" refers to a list of dishes and drinks offered by a restaurant, along with detailed information about them.

[1803] "User interface means" refers to the means by which a user interacts with a system, and includes devices such as smartphones and tablets.

[1804] The "list of missing ingredients" is a list of ingredients needed for the selected recipe that are not currently in the refrigerator.

[1805] The "optimal supplier" refers to a place to purchase food ingredients that has been evaluated based on criteria such as price, stock availability, and logistics lead time.

[1806] Modes for carrying out the invention

[1807] To implement this invention, three main entities are primarily used: a server, a terminal, and a user. The server plays a central role in data collection, analysis, and proposal, while the terminal provides information and interacts with the user. The user is the system user and the entity that provides input and feedback.

[1808] Server Processing

[1809] Data collection

[1810] The server collects the following information via the internet:

[1811] 1. Obtaining flyer information from online retail stores.

[1812] The server uses scraping technology to periodically access the websites of online retail stores and automatically retrieve the latest flyer information. This ensures that discount and special offer information is always up-to-date.

[1813] 2. Receiving inventory information from the IoT refrigerator.

[1814] The server periodically receives inventory information from IoT refrigerators installed in the user's home. MQTT and HTTP are used as communication protocols. This allows the server to monitor the food inventory inside the refrigerator.

[1815] 3. Obtain inventory information and logistics lead times from online supermarkets and food delivery websites.

[1816] The server calls APIs from partner online supermarkets and grocery delivery sites to obtain the latest inventory information and logistics lead times.

[1817] 4. Obtaining weather information from a weather forecast API

[1818] The server periodically calls the weather forecast API to retrieve the latest weather information. Authentication is performed using the API key.

[1819] 5. Obtaining takeout information and menu information from partner restaurants and food service providers.

[1820] The server calls APIs from partner restaurants and food service providers to retrieve the latest takeout and menu information.

[1821] 6. Retrieving family structure and event date information entered by the user.

[1822] Users enter information such as family composition and event dates (birthdays, anniversaries, etc.) through the application's input form. This information is sent to the server and stored in the database.

[1823] Data Analysis

[1824] Based on the information collected above, the server performs the following data analysis:

[1825] 1. Create a list of seasonal ingredients.

[1826] The server compares the retrieved inventory information from the refrigerator with online retail store flyers to create a list of seasonal ingredients. This process uses the Python Pandas library.

[1827] 2. Selecting the optimal recipe

[1828] The server selects the optimal recipe by considering parameters such as weather forecasts, family composition, and event dates. This recipe selection is performed using an AI model (e.g., TensorFlow or PyTorch).

[1829] 3. List of missing ingredients

[1830] The system compares the required ingredients for the selected recipe with the user's refrigerator inventory and lists any missing ingredients. The Numpy library is used for the comparison process.

[1831] 4. Selecting the optimal supplier and calculating price and logistics lead time.

[1832] For any missing ingredients, the server selects the optimal supplier and calculates the total cost based on price information and logistics lead time. Linear programming may be used.

[1833] 5. Analysis of takeout and dining-in options

[1834] The system also analyzes available takeout and dining-in options and suggests them to the user. Suggestions are generated based on past user preference data.

[1835] Information provision

[1836] The server sends the analysis results to the terminal. Specifically, it provides the following information:

[1837] 1. Recommended recipe video link

[1838] The server generates a video link for the selected recipe and sends it to the device. The user can view the video by clicking the link.

[1839] 2. Required ingredients and where to buy them

[1840] The server compiles the necessary ingredients and their supplier information (price, availability, delivery lead time) and sends it to the terminal.

[1841] 3. Information on takeout and dining out options.

[1842] Based on the analysis results, the server sends suggested takeout and dining-in options to the terminal.

[1843] Terminal processing

[1844] User Interface

[1845] The device (smartphone or tablet) is responsible for user interaction:

[1846] 1. Sending notifications

[1847] The system notifies users of recipe video links and ingredient purchase information received from the server. Notification methods include push notifications and in-app notifications.

[1848] 2. Receiving and sending user input

[1849] The terminal receives input from the user (for example, selection of optional services or feedback) and sends it to the server.

[1850] Provision of optional services

[1851] If a user selects a paid optional service (such as shopping assistance, housekeeping assistance, or restaurant reservations), the terminal forwards the request to the server. The server processes the request and sends it to the appropriate service provider.

[1852] User actions

[1853] Information entry

[1854] Users input information about their family structure and event dates (e.g., birthdays and anniversaries) into the system. They can also update their food inventory information via the IoT refrigerator.

[1855] Acceptance of proposals and feedback

[1856] Users review and implement recipe suggestions, ingredient purchase information, and dining options received from the system. They then provide feedback on the services they used.

[1857] Specific example: When preparing dinner for the family

[1858] 1. The user enters into the system that their child's birthday is approaching. The system also updates the inventory information in the refrigerator.

[1859] 2. The server suggests party recipes based on the inventory information in the refrigerator, a list of seasonal ingredients, and the weather forecast.

[1860] 3. The server lists the missing ingredients and provides information on the best place to purchase them.

[1861] 4. The device notifies the user of the suggested recipe video link and purchase information.

[1862] 5. The user accepts the system's suggestions and uses the shopping assistance service as needed.

[1863] 6. The server processes the request and instructs a partner grocery delivery service to purchase the ingredients.

[1864] Using this specific example, users can efficiently manage and purchase ingredients, cook, and even choose restaurants, completing a series of tasks. As a result, the time and effort involved in daily meal preparation can be significantly reduced.

[1865] Examples of prompts to input into a generative AI model are as follows:

[1866] "My child's birthday is approaching. Please update the ingredients in the refrigerator and suggest party recipes based on online retail stores and weather forecasts. Based on the suggestions, please list any missing ingredients and provide information on the best places to buy them (price, availability, delivery lead time)."

[1867] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1868] Step 1: Data Collection

[1869] The server collects necessary information from multiple sources. Specifically, it accesses the websites of online retail stores and uses scraping techniques to obtain the latest flyer information. This flyer information includes data on discounts and special offers.

[1870] Input: URL of a retail store's website on the internet

[1871] Processing: Use scraping techniques to obtain flyer information and save it to a database.

[1872] Output: Update of flyer information database

[1873] Specifically, it uses Python libraries such as BeautifulSoup and Scrapy to analyze the HTML structure of a web page and extract the necessary information.

[1874] Step 2: Obtain inventory information from the refrigerator

[1875] The server periodically receives inventory information from IoT refrigerators installed in the user's home. MQTT and HTTP are used as communication protocols.

[1876] Input: Inventory information sent from IoT refrigerator

[1877] Processing: Analyze inventory information and save it to the database.

[1878] Output: Refrigerator inventory database update

[1879] Specifically, the IoT refrigerator sends inventory information to the cloud at regular intervals, and a server retrieves and analyzes this data.

[1880] Step 3: Obtain logistics lead time and inventory information.

[1881] The server calls APIs from partner online supermarkets and grocery delivery sites to obtain the latest inventory information and logistics lead times.

[1882] Input: API key and endpoint for online supermarkets and grocery delivery sites.

[1883] Processing: Make an API call and save the retrieved inventory information and logistics lead time to the database.

[1884] Output: Update of inventory and logistics lead time databases

[1885] Specifically, the process involves making HTTPS requests using a RESTful API, parsing the data returned in JSON format, and saving it.

[1886] Step 4: Gathering weather forecast information

[1887] The server periodically calls the weather forecast API to retrieve the latest weather information. Authentication is performed using the API key.

[1888] Input: API key and endpoint for the weather forecast API.

[1889] Process: Make an API call and save the retrieved weather information to the database.

[1890] Output: Weather forecast database update

[1891] Specifically, the process involves making an HTTPS request using an API client and then parsing the returned JSON data.

[1892] Step 5: Gathering information on partner restaurants

[1893] The server calls APIs from partner restaurants and food service providers to retrieve the latest takeout and menu information.

[1894] Input: API key and endpoint for partner restaurants and food service providers

[1895] Processing: Make an API call and save the retrieved takeout information and menu information to the database.

[1896] Output: Updates to the takeout and menu databases.

[1897] In terms of specific operations, it will similarly make HTTPS requests using a RESTful API, parse the data, and save it.

[1898] Step 6: Obtaining User Information

[1899] Users enter information such as family composition and event dates (birthdays, anniversaries, etc.) through the application's input form. This information is sent to the server and stored in the database.

[1900] Input: Family composition and event date information entered by the user.

[1901] Processing: Parse the input information and save it to the database.

[1902] Output: User information database update

[1903] In terms of specific operation, a POST request is sent from the frontend input form to the server, and the data is received and analyzed on the backend.

[1904] Step 7: Data Analysis and Recipe Selection

[1905] The server performs data analysis based on the collected information and selects the optimal recipe that matches the user's conditions. It uses AI models (e.g., TensorFlow, PyTorch).

[1906] Input: Refrigerator inventory information, flyer information, weather forecast, family composition, event date information

[1907] Processing: Use an AI model to select the optimal recipe.

[1908] Output: Selected recipes

[1909] In terms of specific operations, data is input into the AI ​​model, and the optimal recipe is obtained as a result of the model's prediction.

[1910] Step 8: List the missing ingredients

[1911] The server compares the ingredients required for the selected recipe with the ingredients currently in the refrigerator and lists any missing ingredients.

[1912] Input: Selected recipe, refrigerator inventory information

[1913] Processing: Use the Numpy library to list the missing ingredients.

[1914] Output: List of missing ingredients

[1915] Specifically, the process involves using a data frame to compare required ingredients with current inventory and extracting any missing ingredients.

[1916] Step 9: Selecting the best supplier and calculating the price.

[1917] The server selects the best supplier for any missing ingredients and calculates the total cost based on price information and logistics lead time.

[1918] Input: List of missing ingredients, online supermarket inventory information, price information, logistics lead time

[1919] Process: Use linear programming to calculate the optimal supplier and total cost.

[1920] Output: Optimal purchasing information and total cost

[1921] Specifically, the program performs linear programming using the Scipy library.

[1922] Step 10: Provide a link to a recommended recipe video.

[1923] The server generates links to video content related to the selected recipe and sends them to the device.

[1924] Input: Selected recipe

[1925] Processing: Generate video link and notify device.

[1926] Output: Recommended recipe video link notification

[1927] Specifically, the system retrieves video links related to the recipe from the database and notifies the user.

[1928] Step 11: Notifying and providing feedback to the user

[1929] The device notifies the user based on information received from the server. It also receives feedback from the user.

[1930] Input: Notification information from the server, user feedback

[1931] Processing: Sending notifications, receiving feedback and forwarding it to the server.

[1932] Output: User notification and feedback sent to the server

[1933] Specifically, the system sends push notifications via a mobile application and receives user feedback.

[1934] Step 12: Requesting and processing optional services

[1935] The server processes user requests for paid optional services and forwards the requests to the appropriate service providers.

[1936] Input: User's optional service request

[1937] Processing: Processing the request and forwarding the request to the service provider.

[1938] Output: Request to service provider

[1939] In terms of specific operation, the API is used to send a corresponding request to the service provider.

[1940] (Application Example 1)

[1941] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1942] In today's busy lifestyle, meal preparation has become a significant burden for users. In particular, keeping track of what's in the refrigerator, planning appropriate recipes, purchasing necessary ingredients, and finding restaurant or delivery options when time is limited are all cumbersome tasks. In this situation, there is a need for a system that centrally manages scattered information and provides optimal suggestions to help users efficiently plan everything from purchasing ingredients to cooking and dining out.

[1943] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1944] In this invention, the server includes means for collecting flyer information from online retail stores, means for receiving inventory information in the refrigerator, means for obtaining food purchase information based on logistics lead times, means for receiving weather forecast information, means for obtaining family composition and event date information, means for analyzing this information and providing optimal recipes and food purchase advice, means for suggesting optimal recipes based on the ingredients in the user's refrigerator and the weather forecast, means for listing missing ingredients and providing information on the best place to buy them, and means for providing menu information that can be ordered online from nearby restaurants. This enables the user to efficiently manage and execute everything from purchasing ingredients to cooking and planning meals out in a unified manner.

[1945] "Online retail store flyer information" refers to special sale and discount information provided by retail stores that can be accessed via the internet.

[1946] "Refrigerator inventory information" refers to information about the types and quantities of food and products stored inside the refrigerator.

[1947] "Logistics lead time" refers to the time from the time an order is placed until delivery is completed.

[1948] "Weather forecast information" refers to predicted weather conditions and includes data such as temperature, probability of precipitation, and wind speed.

[1949] "Family structure" refers to information such as the number of members in the user's household and their relationships.

[1950] "Event date information" refers to information about special days for the user, such as birthdays and anniversaries.

[1951] An "o...

Claims

1. Methods for collecting online retail store flyer information, A means of receiving inventory information inside the refrigerator, A means of obtaining food ingredient purchase information based on logistics lead time, A means of receiving weather forecast information, Means of obtaining information on family structure and event dates, A system that analyzes this information and provides optimal recipe and ingredient purchasing advice.

2. The system according to claim 1, including means for providing recipe videos.

3. The system according to claim 1, which includes means for collecting and analyzing takeout information and restaurant information to make restaurant dining recommendations.

4. The system according to claim 1, comprising means for providing, as paid options, shopping assistance services, housekeeping services, restaurant reservations, and related product introductions.

5. The system according to claim 1, comprising means for collecting and analyzing user feedback to improve the accuracy of the system's suggestions.

Images