system

Abstract

Provide a system. [Solution] A means for taking an image of the inside of a refrigerator; A means for uploading the captured image to a server; means for analyzing the uploaded image to identify the item; a means for storing data of the identified items in a database and updating inventory information; A means for suggesting recipes that can be made by calling the API of a recipe suggestion service based on the updated inventory information; a means for displaying on the terminal a list of ingredients for the suggested recipe; A system including:

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Conventional refrigerators require complicated internal inventory management, often resulting in food waste and ingredient shortages, resulting in inconvenience. Furthermore, they lack centralized functionality, such as optimal recipe suggestions based on inventory information or automatic ordering of missing ingredients, resulting in low user convenience. This results in increased food waste and unnecessary purchases, as well as inefficient food management within the home. [Means for solving the problem]

[0005] The present invention solves these problems by providing a system that includes a means for taking images of the inside of a refrigerator, a means for uploading the taken images to a server, a means for identifying items by analyzing the uploaded images, a means for saving data on the identified items in a database and updating inventory information, a means for suggesting possible recipes by calling an API of a recipe suggestion service based on the updated inventory information, and a means for displaying a list of ingredients for the suggested recipes on a terminal. Furthermore, by further including a means for preprocessing the uploaded images and a means for identifying missing ingredients from the list of ingredients for the suggested recipes and notifying the user of the missing ingredients, the system enables users to efficiently manage inventory in the refrigerator and select recipes, and further improves the convenience of purchasing missing ingredients and automatic ordering.

[0006] The "means for capturing images inside the refrigerator" is a camera device that captures images inside the refrigerator to recognize the items inside the refrigerator.

[0007] The "means for uploading captured images to a server" is a communication means for transferring images captured by the camera inside the refrigerator to a server via the Internet.

[0008] The "means for analyzing uploaded images and identifying items" refers to software or algorithms that use image recognition technology to analyze images received in the server and identify items in the refrigerator.

[0009] The "means for storing data of identified items in a database and updating inventory information" refers to a means for recording information of items identified by image analysis in a database and updating current inventory data in the refrigerator.

[0010] "Calling the API of the recipe suggestion service" refers to a means of calling an application program interface for linking with an external recipe database and obtaining the optimal recipe based on inventory information.

[0011] The "means for suggesting recipes that can be made" is a system for presenting to the user recipes that can be made using ingredients in the refrigerator based on the acquired recipe data.

[0012] The "means for displaying a list of ingredients for a proposed recipe on a terminal" is a means for displaying a list of ingredients required for a recipe on a terminal such as a user's smartphone or tablet.

[0013] "Means for pre-processing uploaded images" refers to means for performing pre-processing such as adjusting the brightness and contrast of images and removing noise in order to improve image recognition accuracy.

[0014] The "means for identifying missing ingredients and notifying the user of the missing ingredients" refers to a means for identifying ingredients that are required for a proposed recipe but are not in the refrigerator and notifying the user of that information. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8]FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0016] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0019] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0020] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0021] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0023] [First embodiment]

[0024] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0025] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0028] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0031] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0035] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0036] MODE FOR CARRYING OUT THE INVENTION

[0037] Configuration overview

[0038] This system manages the items in the refrigerator, suggests optimal recipes, and notifies users of any missing ingredients. The system includes a camera inside the refrigerator, a server connected to the Internet, and a user's device (such as a smartphone or tablet).

[0039] Basic operations

[0040] 1. Taking pictures inside the refrigerator

[0041] Device: A camera installed inside the refrigerator takes pictures of the interior periodically or upon user instruction.

[0042] 2. Upload an image

[0043] Terminal: Uploads captured images to a server via Wi-Fi or other communication means.

[0044] 3. Image analysis and item identification

[0045] Server: Analyzes the uploaded images and identifies the items in the refrigerator. This analysis uses machine learning models (e.g., TENSORFLOW (registered trademark), PyTorch).

[0046] 4. Updating inventory information

[0047] Server: Stores the information of identified items in a database and keeps the inventory information in the refrigerator up to date.

[0048] 5. Recipe suggestions

[0049] Server: Based on the updated inventory information, it calls an external recipe database (e.g., a recipe suggestion service API), matches the available ingredients, and retrieves recipes that can be made.

[0050] 6. Viewing Recipes

[0051] Device: The recipe information sent from the server is displayed on the user's smartphone or tablet. The displayed recipe also includes a list of required ingredients.

[0052] 7. Identification and notification of shortages

[0053] Terminal: Identify missing ingredients from the suggested recipe's ingredient list and notify the user. The list of missing ingredients is displayed on the terminal, allowing the user to review them.

[0054] 8. Check inventory on the go

[0055] Users can access the system from their smartphones or tablets while on the go, such as at the supermarket, and check real-time inventory information and a list of materials that are in short supply.

[0056] 9. Delivery service collaboration (when at home)

[0057] User: If there are missing materials while at home, the system will provide the option to automatically order the missing materials using a delivery service (e.g., a delivery service API).

[0058] 10. Notifications and Optimization

[0059] Server: Notifies the user that the purchase or delivery is complete, updates the refrigerator inventory again, and updates the algorithm to improve the accuracy of future recipe suggestions based on the user's purchase history and frequency of use.

[0060] Specific examples

[0061] User: There are carrots, milk, and eggs in the refrigerator.

[0062] Server: These items are identified through image analysis and stored in a database.

[0063] Server: Recipes such as "stir-fried vegetables" and "omelette" are obtained from the recipe suggestion service.

[0064] Device: The smartphone displays recipes for "Stir-fried vegetables" and "Omelette," along with a list of ingredients needed.

[0065] User: Checks the supermarket for a missing ingredient (e.g., green peppers) and adds it to a shopping list.

[0066] This completes the embodiment of the present invention. This system allows users to efficiently manage inventory in their refrigerators, select optimal recipes, and easily check, purchase, and order ingredients that are in short supply.

[0067] The processing flow will be explained below.

[0068] Step 1:

[0069] Device: A camera installed inside the refrigerator takes pictures of the inside of the refrigerator at regular intervals or as set by the user.

[0070] Step 2:

[0071] Terminal: Uploads captured images to a server via Wi-Fi or other communication means.

[0072] Step 3:

[0073] Server: Preprocesses the received images, adjusting brightness and contrast, and removing noise to improve the accuracy of image recognition.

[0074] Step 4:

[0075] Server: The preprocessed images are input into a machine learning model (e.g., TensorFlow, PyTorch) and analyzed to identify the items in the refrigerator.

[0076] Step 5:

[0077] Server: Stores information about items identified through image analysis in a database and updates the current inventory information in the refrigerator.

[0078] Step 6:

[0079] Server: Based on the updated inventory information, the API of the recipe suggestion service is called, and the available ingredients are matched to obtain recipes that can be made.

[0080] Step 7:

[0081] Server: Selects the optimal recipe from the retrieved recipes and sends that information to the user's device.

[0082] Step 8:

[0083] Device: Recipe information and a list of ingredients are displayed on the user's smartphone or tablet.

[0084] Step 9:

[0085] On the device: Based on the ingredients list of the suggested recipe, identify any missing ingredients and notify the user.

[0086] Step 10:

[0087] Users: Access the system from the supermarket or other locations to view real-time inventory information and shortage lists.

[0088] Step 11:

[0089] User: If there are missing materials while at home, select the delivery service integration option.

[0090] Step 12:

[0091] Server: Calls the delivery service's API to automatically order missing ingredients.

[0092] Step 13:

[0093] Server: Notifies the user once the order is complete and tracks the delivery progress.

[0094] Step 14:

[0095] Terminal: Once the delivery is complete, the user is notified again and the refrigerator inventory information is updated.

[0096] Step 15:

[0097] Server: Updates the algorithm to improve the accuracy of future suggestions based on the user's purchase history and frequency of use.

[0098] Through the above processing steps, users can seamlessly manage inventory in the refrigerator, select recipes, and even purchase missing ingredients or place automatic orders.

[0099] Example 1

[0100] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0101] Conventional refrigerator management systems require users to manually manage refrigerator inventory information, which can result in users losing track of what's in the refrigerator, leading to food waste and unnecessary purchases. Another issue is that checking and purchasing ingredients when they're running low is complicated and time-consuming, making efficient food management difficult.

[0102] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0103] In this invention, the server includes means for taking images of the inside of the refrigerator, means for uploading the taken images to the server, means for identifying items by analyzing the uploaded images, means for saving data on the identified items in a database and updating inventory information, means for calling a recipe suggestion service based on the updated inventory information and suggesting possible recipes, means for displaying an ingredient list for the suggested recipe on a terminal, means for identifying missing ingredients from the ingredient list for the suggested recipe and notifying the user, and means for checking inventory information and missing ingredients using a smart device while on the go. This enables automatic and efficient management of inventory in the refrigerator, allowing users to select optimal recipes and easily check and purchase missing ingredients.

[0104] "Means for taking images inside the refrigerator" refers to the camera installed to capture images inside the refrigerator and the system that controls it in general.

[0105] "Means for uploading captured images to a server" refers to a communication system and software for transmitting captured image data to a server via the Internet or the like.

[0106] "Means for identifying items by analyzing uploaded images" refers to algorithms or machine learning models that use image recognition technology to identify items in the refrigerator from uploaded images.

[0107] "Means for storing data of identified items in a database and updating inventory information" refers to software and hardware for recording recognized item information in a database and keeping inventory information in the refrigerator up to date based on that information.

[0108] "Means for calling a recipe suggestion service based on updated inventory information and suggesting recipes that can be made" refers to a system that obtains appropriate recipe information from an external service based on the latest inventory information in the refrigerator and provides it to the user.

[0109] "Means for displaying a list of ingredients for a proposed recipe on a terminal" refers to an application or interface for displaying recipe information provided by the server on a terminal such as a user's smartphone or tablet.

[0110] "Means for identifying missing ingredients from the ingredient list of a proposed recipe and notifying the user" refers to a system that compares ingredients required for a recipe with current inventory information, detects missing ingredients, and notifies the user.

[0111] "Means for checking inventory information and missing ingredients using smart devices while on the go" refers to a function that allows users to check inventory information and missing ingredients in their refrigerators in real time using a smartphone or tablet while on the go, such as at a supermarket.

[0112] "Means for pre-processing captured images" refers to algorithms or software for performing pre-processing such as resizing, noise removal, and color correction of images before image analysis.

[0113] "Means to order missing ingredients via a delivery service" refers to an interface or API for automatically ordering missing ingredients from a delivery service.

[0114] "Means for updating algorithms to improve the accuracy of recipe suggestions based on purchase history and frequency of use" refers to machine learning models and algorithms that optimize future recipe suggestions by taking into account a user's past purchase history and frequency of use of an item.

[0115] MODE FOR CARRYING OUT THE INVENTION

[0116] The system of the present invention aims to efficiently manage items in a refrigerator, provide optimal recipes to users, and facilitate checking and purchasing of ingredients that are in short supply. The system includes a camera in the refrigerator, a server connected to the Internet, and a user's device (such as a smartphone or tablet).

[0117] Taking pictures inside the refrigerator

[0118] Device: A camera installed inside the refrigerator takes pictures based on user instructions or a regular schedule. When a user presses the "take a picture" button on a smartphone application, the camera activates and captures an image of the inside of the refrigerator. For example, a user may issue a voice command such as "Take a picture of the inside of the refrigerator."

[0119] Uploading an image

[0120] Device: The captured images are automatically uploaded to a server using Wi-Fi or other communication methods. For example, a smartphone application can link with the refrigerator camera and send image data to the server.

[0121] Image analysis and item identification

[0122] Server: The server receives the uploaded images and performs image analysis using machine learning models such as TensorFlow and PyTorch. It identifies the items identified through the analysis and records that information in a database. For example, items such as "carrots," "milk," and "eggs" are identified.

[0123] Inventory updates

[0124] Server: The identified item information is stored in a database and the refrigerator inventory information is updated to the latest state, so that the inventory is always reflected in real time.

[0125] Recipe suggestions

[0126] Server: Based on the latest inventory information, the server calls an external recipe suggestion service API to obtain recipe information that can be made with the current inventory. For example, recipes for "stir-fried vegetables" and "omelette" are obtained from the recipe suggestion service.

[0127] Viewing Recipes

[0128] Device: The recipe information sent from the server is displayed on the user's smartphone or tablet. This information also includes a list of ingredients, which the user can check. Specifically, the smartphone app displays recipes for "stir-fried vegetables" and "omelette" along with their ingredient lists.

[0129] Identifying and notifying shortages

[0130] On the device: Based on the ingredients list of a suggested recipe, identify any missing ingredients and notify the user. For example, if a recipe requires a missing bell pepper, a pop-up notification will appear on the device stating, "You are missing bell peppers."

[0131] Check inventory on the go

[0132] User: While on the go (e.g., at the supermarket), they can use their smartphone to check real-time inventory information and a list of ingredients they are short on. For example, a user opens the app and sees that there are six eggs in the refrigerator.

[0133] Delivery service collaboration (when at home)

[0134] User: If a user is at home and is short on ingredients, the system provides the option to automatically order the missing ingredients using a delivery service API. For example, when a user presses the "Order Delivery" button on their smartphone, the delivery service API is called and green peppers are automatically ordered.

[0135] Optimization based on purchase history and frequency of use

[0136] Server: Once the purchase or delivery is complete, the user is notified and the refrigerator inventory is updated again. The server also updates the algorithm to improve the accuracy of future recipe suggestions based on the user's purchase history and frequency of use. For example, the server can retrain the algorithm based on the purchase data to make better suggestions next time.

[0137] Examples of prompt statements

[0138] "Suggest a recipe that can be made with the items in your refrigerator. Currently, we have carrots, milk, and eggs."

[0139] This invention allows users to efficiently manage their refrigerator inventory, select suitable recipes, and easily check and purchase ingredients they are running low on.

[0140] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0141] Program processing flow

[0142] Step 1:

[0143] Device: A camera installed inside the refrigerator takes pictures based on user instructions or set timing. The input is the user's instructions or schedule, and the output is the image taken inside the refrigerator. For example, when a user presses the "take a picture" button in the app, the camera is activated and an image of the inside of the refrigerator is captured.

[0144] Specific operation: When a user presses the "shoot" button on the smartphone app, the camera starts up and takes a picture of the inside of the refrigerator. At this time, the internal LED light turns on, allowing a clear image to be captured in bright light.

[0145] Step 2:

[0146] Terminal: Acquires captured images and uploads them to a server via Wi-Fi. The input is the captured image data, and the output is the image data uploaded to the server. After the camera captures the image, the smartphone app automatically collects the image data and sends it to the server.

[0147] What it does: After taking a photo, the app uploads the image data to a server via Wi-Fi. This process runs in the background, and users can monitor the progress via a progress bar within the app.

[0148] Step 3:

[0149] Server: The server receives the uploaded images and performs image analysis using a machine learning model (e.g., TensorFlow or PyTorch). The input is the uploaded image data, and the output is a list of identified items. Through analysis, the items in the refrigerator are identified.

[0150] Specific operation: When the server receives the image data, the machine learning model is activated and analyzes the items in the refrigerator. From the analysis results, items such as "carrots," "milk," and "eggs" are identified. The results are recorded in a database.

[0151] Step 4:

[0152] Server: Stores the identified item information in a database and updates the inventory information in the refrigerator. The input is a list of identified items, and the output is updated inventory information. This ensures that the inventory is always updated in real time.

[0153] What happens: The server updates the inventory table in the database, recording new information such as "3 carrots, 1 liter of milk, and 6 eggs."

[0154] Step 5:

[0155] Server: Based on the latest inventory information, it calls an external recipe suggestion service API to obtain recipe information that can be made. The input is the latest inventory information, and the output is the suggested recipe information.

[0156] Specific operation: The server calls the recipe suggestion service API and sends the current inventory information. For example, based on the inventory information for "carrots, milk, and eggs," recipe information for "stir-fried vegetables" and "omelette" is obtained.

[0157] Step 6:

[0158] Device: The recipe information sent from the server is displayed on the user's smartphone or tablet. The input is the suggested recipe information, and the output is the recipe information displayed on the device.

[0159] What it does: The app on the user's smartphone displays recipes for "stir-fried vegetables" and "omelette," along with a list of ingredients. A detailed explanation of the recipe and cooking instructions are also displayed.

[0160] Step 7:

[0161] Terminal: Based on the recipe ingredient list, it identifies missing ingredients and notifies the user. The input is the suggested recipe information and current inventory information, and the output is the list of missing ingredients and a notification.

[0162] What it does: The device checks the ingredients in the suggested recipe against the current inventory and displays a notification to the user saying, "You're low on bell peppers." The list of ingredients you're short on is available in the app.

[0163] Step 8:

[0164] User: Check real-time inventory information and material shortage lists on the go using a smartphone or tablet. The input is an access request from the smartphone, and the output is real-time inventory information.

[0165] Specific behavior: When a user opens the app at a supermarket and taps the stock check button, they can see that there are 6 eggs left.

[0166] Step 9:

[0167] User: When there are missing materials while at home, the system automatically orders the missing materials using the delivery service API. The input is a list of missing materials, and the output is a notification that the delivery order has been completed.

[0168] Specific operation: When a user presses the "Order Delivery" button in the app, the delivery service API is called and "green peppers" are automatically ordered. After the order is completed, a notification is displayed on the smartphone.

[0169] Step 10:

[0170] Server: Notifies the user that the purchase or delivery is complete and updates the refrigerator inventory information again. It also updates the algorithm to improve the accuracy of recipe suggestions based on purchase history and frequency of use. The input is the purchase completion notification, and the output is the updated inventory information and the algorithm.

[0171] Specific operation: When the delivery is complete, a notification saying "Peppers delivery completed" is displayed on the user's device. The server updates the inventory information in the database and retrains the algorithm based on the purchase data to improve the accuracy of the next recommendation.

[0172] (Application example 1)

[0173] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0174] In modern life, managing ingredients in the refrigerator is complicated, and the hassle of checking for missing ingredients and the stress of selecting the optimal recipe are issues. Many users also find it troublesome to check inventory while on the go and to purchase or order missing ingredients. Therefore, there is a need for a system that can efficiently manage refrigerator inventory information, suggest optimal recipes, and automatically order missing ingredients.

[0175] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0176] In this invention, the server includes a means for taking images of the inside of the refrigerator, a means for uploading the taken images to the server, a means for analyzing the uploaded images to identify items, and a means for automatically ordering missing ingredients from a food delivery service. This makes it possible to efficiently manage inventory information in the refrigerator, propose optimal recipes, and automatically order missing ingredients.

[0177] The "means for taking images inside the refrigerator" is a device that uses a camera installed inside the refrigerator to take images of items inside the refrigerator periodically or based on a user's instruction.

[0178] The "means for uploading captured images to a server" is a communication function for transmitting captured image data to a server via the Internet.

[0179] The "means of identifying items by analyzing uploaded images" refers to a technology that uses machine learning algorithms and image analysis technology on a server to recognize and identify each item in a refrigerator from uploaded images.

[0180] The "means for storing data on identified items in a database and updating inventory information" is a system for storing analyzed item information in a database and keeping the current inventory information in the refrigerator up to date.

[0181] "Means of suggesting recipes that can be made by calling the API of a recipe suggestion service based on updated inventory information" refers to a method of using the API of an external recipe suggestion service based on the latest inventory information to obtain recipes for dishes that can be made based on current inventory.

[0182] The "means for displaying the ingredient list of the proposed recipe on a terminal" is a means for displaying the acquired recipe information and the list of necessary ingredients on a terminal such as a user's smartphone or tablet.

[0183] The "means for automatically ordering missing ingredients from a food delivery service" is a technology that determines which ingredients are missing to create a recipe and automatically orders the missing ingredients from a food delivery service via the Internet.

[0184] "Means for pre-processing uploaded images" refers to methods for pre-processing captured images, such as filtering and resizing, in order to improve image quality or extract necessary information before image analysis.

[0185] The "means for notifying the user of missing ingredients" is a technology that sends a notification message to a smartphone, tablet, etc. to inform the user of any missing ingredients needed for a recipe proposal.

[0186] An embodiment of the present invention will now be described. The system of the present invention efficiently manages items in a refrigerator, proposes optimal recipes, and automatically orders ingredients that are in short supply. This system includes the following components: a camera in the refrigerator, a server connected to the Internet, and a user terminal (such as a smartphone or tablet).

[0187] First, the basic operation steps of the system will be explained.

[0188] Basic operations

[0189] 1. Taking pictures inside the refrigerator

[0190] Terminal: A camera installed inside the refrigerator takes pictures of the inside periodically or based on the user's instructions.

[0191] 2. Upload an image

[0192] Terminal: Uploads captured images to a server via Wi-Fi or other communication means.

[0193] 3. Image analysis and item identification

[0194] Server: Analyzes the uploaded images and identifies the items in the refrigerator. This analysis uses machine learning models such as TensorFlow and PyTorch.

[0195] 4. Updating inventory information

[0196] Server: Stores the information of identified items in a database and keeps the inventory information in the refrigerator up to date.

[0197] 5. Recipe suggestions

[0198] Server: Based on the updated inventory information, it calls an external recipe database (e.g., a recipe suggestion service API), matches the available ingredients, and retrieves recipes that can be made.

[0199] 6. Viewing Recipes

[0200] Device: The recipe information sent from the server is displayed on the user's smartphone or tablet. The displayed recipe also includes a list of necessary ingredients.

[0201] 7. Identification and notification of shortages

[0202] Terminal: Identify missing ingredients from the suggested recipe's ingredient list and notify the user. The list of missing ingredients is displayed on the terminal so the user can review them.

[0203] 8. Automatic ordering of missing materials

[0204] Server: Automatically orders missing ingredients from a food delivery service, using the food delivery service's API.

[0205] Hardware and software used

[0206] 1. Refrigerator camera

[0207] Example: Camera using the Raspberry Pi camera module

[0208] 2. Internet connection environment

[0209] Example: Wi-Fi router

[0210] 3. User's Device

[0211] Example: Smartphones and tablets

[0212] 4. Server

[0213] Programming: Python

[0214] Image processing: OpenCV

[0215] Item identification: Machine learning libraries such as TensorFlow and PyTorch

[0216] Communication:requests library

[0217] Specific examples

[0218] A user uses the "SmartKitchen" app while there is chicken, onions, and carrots in the refrigerator. The camera inside the refrigerator automatically takes a picture, and the app suggests recipes for "chicken teriyaki" and "curry." Because the user is low on onions, the app automatically places an order with the nearest food delivery service and notifies the user.

[0219] Prompt Sentence Examples

[0220] Prompt for an application that determines what ingredients are in the refrigerator, suggests recipes to the user, and automatically orders any missing ingredients:

[0221] Identify ingredients from images of your refrigerator and suggest recipes based on your current inventory, including automatically ordering missing ingredients through a food delivery service.

[0222] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0223] Step 1:

[0224] Taking pictures inside the refrigerator

[0225] The device takes pictures using a camera installed inside the refrigerator. In this process, the camera module captures images of the inside of the refrigerator periodically or based on the user's instructions. The input is the current state of the inside of the refrigerator, and the output is a captured image file.

[0226] Step 2:

[0227] Uploading an image

[0228] The device uploads the captured image file to the server. In this process, the image data is sent to the server via an internet connection. The input is the captured image file, and the output is the image data stored on the server.

[0229] Step 3:

[0230] Image analysis and item identification

[0231] The server analyzes the uploaded image data and identifies the items in the refrigerator. This process uses machine learning models (e.g., TensorFlow, PyTorch) to analyze the image data and identify specific items. The input is the uploaded image data, and the output is a list of identified items.

[0232] Step 4:

[0233] Inventory updates

[0234] The server saves the identified item list in a database and updates the inventory information. This process writes the identified item data to the database and updates the refrigerator inventory information. The input is the identified item list, and the output is the updated inventory information.

[0235] Step 5:

[0236] Recipe suggestions

[0237] Based on the updated inventory information, the server calls the API of the recipe suggestion service to suggest recipes that can be made. This process calls an external recipe database API to obtain appropriate recipes based on the inventory information in the refrigerator. The input is the updated inventory information, and the output is a list of suggested recipes.

[0238] Step 6:

[0239] Viewing Recipes

[0240] The device displays the recipe information sent from the server on the user's smartphone or tablet. In this process, the suggested recipe and the list of required ingredients are displayed on the device. The input is the suggested recipe list, and the output is the recipe information displayed on the user's device.

[0241] Step 7:

[0242] Identifying and notifying shortages

[0243] The terminal identifies missing ingredients from the ingredient list of the proposed recipe and notifies the user. This process compares the missing ingredients with current inventory information and sends a notification message to the user. The input is the ingredient list of the proposed recipe and current inventory information, and the output is a missing ingredient list and a notification message.

[0244] Step 8:

[0245] Automatic ordering of missing materials

[0246] The server automatically orders the missing ingredients from a food delivery service. This process uses the list of missing ingredients to call the food delivery service's API and confirm the order. The input is the list of missing ingredients, and the output is order confirmation information.

[0247] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0248] MODE FOR CARRYING OUT THE INVENTION

[0249] Configuration overview

[0250] This system manages the items in the refrigerator, suggests optimal recipes, notifies users of missing ingredients, and also incorporates a user emotion recognition engine. The system includes a camera in the refrigerator, an internet-connected server, a user device (such as a smartphone or tablet), and an emotion recognition engine.

[0251] Basic operations

[0252] 1. Taking pictures inside the refrigerator

[0253] Device: A camera installed inside the refrigerator takes pictures of the interior periodically or upon user instruction.

[0254] 2. Upload an image

[0255] Terminal: Uploads captured images to a server via Wi-Fi or other communication means.

[0256] 3. Image analysis and item identification

[0257] Server: Preprocesses the received images, adjusting brightness and contrast, and removing noise to improve the accuracy of image recognition.

[0258] Server: The preprocessed images are input into a machine learning model (e.g., TensorFlow, PyTorch) and analyzed to identify the items in the refrigerator.

[0259] 4. Updating inventory information

[0260] Server: Stores information about items identified through image analysis in a database and updates the current inventory information in the refrigerator.

[0261] 5. Recipe suggestions

[0262] Server: Based on the updated inventory information, the API of the recipe suggestion service is called, and the available ingredients are matched to obtain recipes that can be made.

[0263] 6. Emotion recognition

[0264] Device: The camera and microphone on the user's device are used to collect facial and voice data.

[0265] Emotion engine: Analyzes collected data and determines the user's emotional state (e.g., joy, sadness, stress, etc.).

[0266] 7. Optimizing recipe suggestions

[0267] Server: Based on the emotional information determined by the emotion engine, the server suggests recipes that suit the user's current emotional state.

[0268] 8. Viewing Recipes

[0269] Device: The recipe information sent from the server is displayed on the user's smartphone or tablet. The displayed recipe also includes a list of required ingredients.

[0270] 9. Identification and Notification of Shortage Materials

[0271] Terminal: Identify missing ingredients from the suggested recipe's ingredient list and notify the user of this information.

[0272] 10. Check inventory on the go

[0273] Users: Access the system from their smartphones or tablets while on the go, such as at the supermarket, to check real-time inventory information and a list of materials that are in short supply.

[0274] 11. Delivery service collaboration (when at home)

[0275] User: If there are missing materials while at home, select the delivery service integration option.

[0276] Server: Calls the delivery service's API to automatically order missing ingredients.

[0277] 12. Notifications and Optimization

[0278] Server: Notifies the user that the purchase or delivery is complete and tracks the delivery progress.

[0279] Terminal: Once the delivery is complete, the user is notified again and the refrigerator inventory information is updated.

[0280] Server: Updates the algorithm to improve the accuracy of future suggestions based on the user's purchase history and frequency of use.

[0281] By incorporating a user emotion recognition engine, recipes can be suggested based on the user's current emotions, providing a more personalized experience.

[0282] Specific examples

[0283] User: There are carrots, milk, and eggs in the refrigerator. The user comes home from work and is stressed.

[0284] Server: These items are identified through image analysis and stored in a database.

[0285] Emotion engine: Determines the user's stress level from facial expressions and voice data, and prioritizes recipes with a relaxing effect.

[0286] Server: Recipes that contribute to relaxation, such as "carbonara," are obtained from a recipe suggestion service.

[0287] Device: The smartphone will display a recipe for "Carbonara" along with a list of ingredients needed.

[0288] User: Checks the supermarket for a missing ingredient (e.g., bacon) and adds it to a shopping list.

[0289] In this way, this system allows users to efficiently manage their refrigerator inventory and also suggests optimal recipes that suit their emotional state at the time, resulting in a more satisfying cooking experience.

[0290] The processing flow will be explained below.

[0291] Step 1:

[0292] Device: A camera installed inside the refrigerator takes pictures of the inside of the refrigerator at regular intervals or as set by the user.

[0293] Step 2:

[0294] Terminal: Uploads captured images to a server via Wi-Fi or other communication means.

[0295] Step 3:

[0296] Server: Preprocesses the received images, adjusting brightness and contrast, and removing noise to improve the accuracy of image recognition.

[0297] Step 4:

[0298] Server: The preprocessed images are input into a machine learning model (e.g., TensorFlow, PyTorch) and analyzed to identify the items in the refrigerator.

[0299] Step 5:

[0300] Server: Stores information about items identified through image analysis in a database and updates the current inventory information in the refrigerator.

[0301] Step 6:

[0302] Server: Based on the updated inventory information, the API of the recipe suggestion service is called, and the available ingredients are matched to obtain recipes that can be made.

[0303] Step 7:

[0304] Device: The camera and microphone on the user's device are used to collect facial and voice data.

[0305] Step 8:

[0306] Emotion engine: Analyzes collected data and determines the user's emotional state (e.g., joy, sadness, stress, etc.).

[0307] Step 9:

[0308] Server: Based on the emotional information determined by the emotion engine, the server optimizes and suggests recipes that suit the user's current emotional state.

[0309] Step 10:

[0310] Device: The recipe information sent from the server is displayed on the user's smartphone or tablet. The displayed recipe also includes a list of required ingredients.

[0311] Step 11:

[0312] On the device: Based on the ingredients list of the suggested recipe, identify any missing ingredients and notify the user.

[0313] Step 12:

[0314] Users: Access the system from their smartphones or tablets while on the go, such as at the supermarket, to check real-time inventory information and a list of materials that are in short supply.

[0315] Step 13:

[0316] User: If there are missing materials while at home, select the delivery service integration option.

[0317] Step 14:

[0318] Server: Calls the delivery service's API to automatically order missing ingredients.

[0319] Step 15:

[0320] Server: Notifies the user once the order is complete and tracks the delivery progress.

[0321] Step 16:

[0322] Terminal: Once the delivery is complete, the user is notified again and the refrigerator inventory information is updated.

[0323] Step 17:

[0324] Server: Updates the algorithm to improve the accuracy of future suggestions based on the user's purchase history and frequency of use.

[0325] Example 2

[0326] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0327] Conventional refrigerator management systems are limited in their ability to manage refrigerator inventory and suggest recipes, making it difficult to provide personalized services that reflect the user's emotional state and circumstances. Furthermore, there are few ways to check inventory information while on the go, making it difficult to take appropriate action when necessary ingredients are unavailable.

[0328] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for capturing images of the inside of the refrigerator, a means for uploading the captured images to the server, a means for preprocessing and analyzing the uploaded images to identify items, a means for saving data on the identified items in a database and updating inventory information, a means for suggesting possible recipes by calling an API of a recipe suggestion service based on the updated inventory information, a means for recognizing the user's emotional state and selecting an optimal recipe, and a means for displaying an ingredient list for the suggested recipe on a terminal. This allows the user to efficiently manage inventory in the refrigerator and receive personalized recipe suggestions based on their emotional state. Furthermore, by checking inventory information and receiving notifications of missing ingredients while on the go, the user can appropriately procure the necessary ingredients.

[0329] The "means for taking images inside the refrigerator" is a mechanism for taking images inside the refrigerator using a camera installed inside the refrigerator.

[0330] The "means for uploading captured images to a server" is a mechanism for transmitting captured image data to a server via a communication means.

[0331] The "means for preprocessing and analyzing uploaded images to identify items" refers to a mechanism that performs preprocessing such as adjusting the brightness and contrast of received image data and removing noise, and then identifies items using a machine learning model.

[0332] The "means for saving data on identified items in a database and updating inventory information" is a mechanism for saving information on items identified by analysis in a database and updating the latest inventory status.

[0333] "Means for calling the API of a recipe suggestion service based on updated inventory information and suggesting recipes that can be made" is a mechanism for calling the API of an external recipe suggestion service based on the latest inventory information and obtaining recipe information that can be made.

[0334] The "means for recognizing the user's emotional state and selecting the optimal recipe" is a mechanism for analyzing the user's facial expressions and voice data to determine their emotional state and select a recipe that is appropriate for that state.

[0335] The "means for displaying the ingredient list of the proposed recipe on the terminal" is a mechanism for displaying the recipe information and ingredient list sent from the server on the user's smartphone or tablet.

[0336] The present invention relates to a system that manages items in a refrigerator, suggests optimal recipes based on the user's emotional state, and notifies the user of any missing ingredients. The system includes a camera in the refrigerator, a server connected to the Internet, a user device (such as a smartphone or tablet), and an emotion engine.

[0337] System Configuration

[0338] 1. Camera in the refrigerator

[0339] Hardware: Camera device

[0340] Software: Image acquisition program

[0341] Function: Takes pictures of the inside of the refrigerator periodically or at the user's command.

[0342] 2. Server

[0343] Hardware: Server Device

[0344] Software: Image preprocessing program, machine learning model (e.g., TensorFlow, PyTorch), recipe suggestion service API, database management system

[0345] Function: Preprocesses received images to identify and specify items. Stores information about identified items in a database and updates inventory information in the refrigerator. Calls the API of a recipe suggestion service based on the updated inventory information and obtains information about recipes that can be created.

[0346] 3. User Device

[0347] Hardware: Smartphones and tablet devices

[0348] Software: Display program, notification program, emotion recognition function

[0349] Function: Displays recipe information sent from the server to the user and notifies them of missing ingredients. Collects the user's facial expressions and voice data and analyzes them with the emotion engine.

[0350] 4. Emotion Engine

[0351] Software: Emotional state analysis algorithm

[0352] Function: Analyzes collected user facial and voice data to determine the user's emotional state. Based on the determined emotional information, selects a recipe that is appropriate for the user's current emotional state.

[0353] Specific examples

[0354] Example 1:

[0355] User: There are carrots, milk, and eggs in the refrigerator. The user comes home from work and is stressed.

[0356] Server: These items are identified through image analysis and stored in a database.

[0357] Emotion engine: Determines the user's stress level from facial expressions and voice data, and prioritizes recipes with a relaxing effect.

[0358] Server: Recipes that contribute to relaxation, such as "carbonara," are obtained from a recipe suggestion service.

[0359] Device: The smartphone will display a recipe for "Carbonara" along with a list of ingredients needed.

[0360] User: Checks the supermarket for a missing ingredient (e.g., bacon) and adds it to a shopping list.

[0361] Example 2 (An example of a prompt using a generative AI model):

[0362] Prompt: "The ingredients in my fridge are carrots, potatoes, cheese, and eggs. I'm feeling relaxed today, so can you recommend a recipe that will fit that mood?"

[0363] Server: Performs image analysis and emotion recognition to suggest relaxing recipes such as "cheese omelet."

[0364] This allows the system to provide a more satisfying cooking experience by allowing users to efficiently manage inventory in the refrigerator, suggesting optimal recipes based on their emotional state at the time, and notifying them of any missing ingredients.

[0365] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0366] Step 1:

[0367] Device: A camera installed inside the refrigerator takes images of the inside of the refrigerator periodically or based on the user's instructions.

[0368] Input: User instruction or timer event.

[0369] Data processing: Acquisition of image data.

[0370] Output: A captured image of the inside of a refrigerator.

[0371] Step 2:

[0372] Terminal: Uploads captured images to a server via Wi-Fi or other communication means.

[0373] Input: Captured image data.

[0374] Data processing: Transfer of image data.

[0375] Output: Image data sent to the server.

[0376] Step 3:

[0377] Server: Preprocesses the received images, adjusting brightness and contrast and removing noise.

[0378] Input: Image data sent to the server.

[0379] Data processing: Image preprocessing (brightness, contrast adjustment, noise removal).

[0380] Output: Preprocessed image data.

[0381] Step 4:

[0382] Server: The preprocessed image data is input into a machine learning model (e.g., TensorFlow, PyTorch) and analyzed to identify the items in the refrigerator.

[0383] Input: Preprocessed image data.

[0384] Data processing: Identification and labeling of items.

[0385] Output: A list of identified items.

[0386] Step 5:

[0387] Server: Stores information about items identified through analysis in a database and updates the inventory information in the refrigerator.

[0388] Input: A list of identified items.

[0389] Data processing: storing information in the database and updating inventory information.

[0390] Output: Updated inventory information.

[0391] Step 6:

[0392] Server: Based on the updated inventory information, the API of the recipe suggestion service is called, and the available ingredients are matched to obtain recipes that can be made.

[0393] Input: Updated inventory information.

[0394] Data processing: Calling the recipe suggestion service API and obtaining recipes.

[0395] Output: A list of suggested recipes.

[0396] Step 7:

[0397] Terminal: Collects the user's facial expressions and voice data and sends them to the emotion engine to recognize the user's emotional state.

[0398] Input: User's facial and voice data.

[0399] Data processing: Collecting and transmitting emotion data.

[0400] Output: Data sent to the emotion engine.

[0401] Step 8:

[0402] Emotion engine: Analyzes collected data and determines the user's emotional state.

[0403] Input: User's facial and voice data.

[0404] Data processing: Analysis of emotional state (e.g., stress, joy).

[0405] Output: Determined emotional state.

[0406] Step 9:

[0407] Server: Based on the emotional information determined by the emotion engine, selects a recipe that is appropriate for the user's current emotional state.

[0408] Input: A list of identified emotional states and suggested recipes.

[0409] Data processing: Recipe selection based on emotional state.

[0410] Output: The best recipe for the user.

[0411] Step 10:

[0412] Terminal: The optimal recipe information sent from the server is displayed on the terminal, and the user is notified of the list of necessary ingredients.

[0413] Input: The selected optimal recipe.

[0414] Data processing: Display recipe information.

[0415] Output: Recipe and ingredients list displayed on the user's device.

[0416] Step 11:

[0417] On the device: Based on the ingredients list of a suggested recipe, identify any missing ingredients and notify the user.

[0418] Input: Materials list and current inventory information.

[0419] Data processing: Identifying missing materials.

[0420] Output: Notify user of missing material.

[0421] Step 12:

[0422] User: Accesses the system from a terminal while on the go, such as at a supermarket, to check refrigerator inventory information and a list of ingredients that are in short supply.

[0423] Input: The user's access request.

[0424] Data processing: Providing real-time inventory information.

[0425] Output: Inventory information and a list of missing materials displayed on the user's terminal.

[0426] Step 13:

[0427] User: When at home, select the delivery service integration option to automatically order missing ingredients.

[0428] Input: User's delivery service selection.

[0429] Data processing: Sending orders to delivery services.

[0430] Output: Delivery order confirmation.

[0431] Step 14:

[0432] Server: Notifies the user that the purchase or delivery is complete and tracks the delivery progress.

[0433] Input: Delivery completion information.

[0434] Data Processing: Delivery progress tracking and notification.

[0435] Output: Delivery completion notification to the user.

[0436] Step 15:

[0437] Terminal: Once the delivery is complete, the user is notified again and the refrigerator inventory is updated.

[0438] Input: Delivery completion information for snooze.

[0439] Data processing: Updating refrigerator inventory information.

[0440] Output: Notification of updated inventory information.

[0441] Step 16:

[0442] Server: Updates the algorithm to improve the accuracy of future suggestions based on the user's purchase history and frequency of use.

[0443] Input: Purchase history and usage frequency data.

[0444] Data processing: Algorithm updates and optimizations.

[0445] Output: Improved accuracy of suggestions for the next time.

[0446] (Application example 2)

[0447] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0448] The present invention aims to improve user satisfaction and convenience by providing a refrigerator inventory management and recipe suggestion system that not only efficiently manages items in a refrigerator, but also suggests optimal cooking instructions based on the user's emotional state and provides a series of processes including automatic ordering of missing ingredients. However, conventional refrigerator management systems have problems in that they do not take into account responses based on the user's emotions and do not have a function for automatically ordering missing ingredients.

[0449] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0450] In this invention, the server includes means for taking images of the inside of the refrigerator, means for uploading the taken images to the server, means for analyzing the uploaded images to identify items, means for saving data on the identified items in a database and updating inventory information, means for calling an API of a recipe suggestion service based on the updated inventory information to provide creatable cooking instructions, means for collecting facial expression and voice data using a camera and microphone of the terminal to acquire the user's emotional state, means for analyzing the collected data to determine the user's emotion, means for proposing cooking instructions according to the determined emotion, means for displaying an ingredient list for the proposed cooking instructions on the terminal, and means for identifying missing items from the suggested ingredient list and automatically ordering them. This allows the user to not only manage inventory in the refrigerator, but also to suggest cooking instructions according to the user's emotional state and automatically order missing ingredients.

[0451] The "means for taking images inside the refrigerator" is a device that uses a camera installed inside the refrigerator to take images of items inside the refrigerator periodically or in response to instructions.

[0452] The "means for uploading captured images to a server" is a communication device that transfers captured images to a server via the Internet.

[0453] The "means for analyzing uploaded images to identify items" is a device that analyzes the images received on the server and uses image recognition algorithms to identify the items in the refrigerator.

[0454] The "means for saving data on identified items in a database and updating inventory information" is a device that saves information on items identified through analysis in a database and updates the inventory status in the refrigerator to the latest status.

[0455] "Means for providing cooking instructions that can be created by calling the API of a recipe suggestion service based on updated inventory information" refers to a device that uses the latest inventory information to access the API of a recipe suggestion service and obtains cooking instructions based on that information.

[0456] "Means for collecting facial expression and voice data using a camera or microphone on a terminal to obtain the emotional state of a user" refers to a device that collects facial expression and voice data using a camera or microphone installed on a terminal in order to grasp the emotional state of a user.

[0457] The "means for analyzing collected data to determine the user's emotions" is a device that analyzes collected facial and voice data and uses emotion recognition algorithms to identify the user's emotional state.

[0458] The "means for proposing cooking instructions according to the determined emotion" is a device that selects and provides optimal cooking instructions based on the user's emotional state.

[0459] The "means for displaying an ingredient list for the proposed cooking instructions on a terminal" is a device that displays an ingredient list based on the proposed cooking instructions on a user's terminal.

[0460] The "means for identifying missing items in the proposed ingredient list and automatically ordering them" refers to a device that verifies the ingredient list based on the proposed cooking instructions, identifies missing items, and automatically orders them.

[0461] System Configuration

[0462] The present invention relates to a "refrigerator item management and recipe suggestion system based on emotional state" and uses the following hardware and software.

[0463] Hardware used:

[0464] Refrigerator camera (connectable Wi-Fi camera)

[0465] User devices (smartphones, tablets)

[0466] server

[0467] Software used:

[0468] Emotion recognition engine (e.g., Affectiva, Microsoft® Emotion API)

[0469] Image analysis engine (e.g. TensorFlow, PyTorch)

[0470] Cloud databases (e.g., AWS (registered trademark) DynamoDB, Google (registered trademark) Cloud Firestore)

[0471] Delivery service APIs (e.g. UberEats, DoorDash)

[0472] Basic operations

[0473] Taking and uploading images of the inside of the refrigerator

[0474] The device sends instructions to a camera installed inside the refrigerator, taking pictures of the inside of the refrigerator periodically or based on the user's instructions. The captured image data is then uploaded to a server via Wi-Fi.

[0475] Image analysis and item identification

[0476] The server preprocesses the uploaded images and uses image recognition algorithms (such as TensorFlow or PyTorch) to identify the items in the refrigerator. The identified item data is stored in a cloud database and inventory information is updated.

[0477] Acquiring and analyzing emotional states

[0478] The device collects facial expression and voice data of the user to obtain the user's emotional state, and the collected data is analyzed using an emotion recognition engine to determine the user's emotional state.

[0479] Recipe suggestions and ingredient lists

[0480] The server then calls the API of the recipe suggestion service to obtain optimal cooking instructions based on the updated inventory information and emotional state. The obtained cooking instructions and ingredient list are then displayed on the user's device.

[0481] Automatic ordering of missing materials

[0482] The terminal identifies any missing items from the proposed materials list and automatically orders them through the delivery service API, with the order status tracked and notified to the user in real time.

[0483] Specific examples

[0484] After the user returns home from work, an image of the contents of the refrigerator is automatically uploaded to the app. The app then acquires the user's facial expression data and determines their emotional state as "tired." Based on this information, the server analyzes the inventory and determines that carbonara is ready to be made, but that there is a shortage of bacon. When the user selects the delivery option, the bacon is automatically ordered, and the progress is displayed in the app.

[0485] Prompt Sentence Examples

[0486] After the user returns home, an image of the inside of the refrigerator and the user's facial expression data are collected from the smartphone. Image analysis is used to check the inventory in the refrigerator, and if fatigue is determined from the facial expression data, what recipes should be suggested and how to order missing ingredients via a delivery service? Please explain this with specific steps.

[0487] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0488] Step 1:

[0489] The user issues a command to take a picture of the refrigerator from the terminal. The input is the user's command, and the output is image data of the inside of the refrigerator. Specifically, the terminal sends a command to take a picture to the camera installed inside the refrigerator, and the camera takes a picture of the current inside of the refrigerator.

[0490] Step 2:

[0491] The captured image data is uploaded from the device to the server. The input is the image data of the inside of the refrigerator, and the output is an image file saved on the server. Specifically, the device transfers the image data to the server via Wi-Fi.

[0492] Step 3:

[0493] The server preprocesses the uploaded image data and performs image analysis. The input is the image file stored on the server, and the output is the data of the identified items in the refrigerator. Specifically, the server adjusts the brightness and contrast of the image and uses a machine learning model to identify the items.

[0494] Step 4:

[0495] The identified item data is stored in a database, and inventory information is updated. The input is the identified item data in the refrigerator, and the output is the updated inventory information. Specifically, the server connects to the database, stores the latest item information, and updates the inventory information.

[0496] Step 5:

[0497] The user's device collects facial and voice data to acquire the user's emotional state. The input is the user's real-time facial and voice data, and the output is the collected data. Specifically, the device collects the data using a camera and microphone.

[0498] Step 6:

[0499] The collected data is sent to a server and analyzed by an emotion recognition engine. The input is the collected facial and voice data, and the output is the user's emotional state. Specifically, the server uses an emotion recognition algorithm to determine the user's emotions.

[0500] Step 7:

[0501] Based on the updated inventory information and emotional state, the server calls the API of the recipe suggestion service to obtain optimal cooking instructions. The input is the updated inventory information and the user's emotional state, and the output is cooking instructions. Specifically, the server calls the API of the recipe suggestion service to obtain cooking instructions.

[0502] Step 8:

[0503] The acquired cooking instructions and ingredient list are displayed on the user's terminal. The input is the cooking instructions and ingredient list, and the output is the information displayed on the terminal. Specifically, the server sends this information to the terminal, and the terminal displays it.

[0504] Step 9:

[0505] The user's device identifies missing items from the proposed materials list and automatically places an order. The input is the materials list and inventory information, and the output is order information for the delivery service. Specifically, the device checks the inventory information to identify missing items and calls the delivery service's API to place the order.

[0506] Step 10:

[0507] The user is notified of the order progress and the inventory information is updated after delivery is completed. The input is the progress of the delivery service, and the output is the updated inventory information and a notification to the user. Specifically, the server receives information from the delivery service, updates the inventory information, and notifies the user.

[0508] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0509] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0510] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0511] [Second embodiment]

[0512] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0513] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0514] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0515] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0516] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0517] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0518] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0519] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0520] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0521] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0522] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0523] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0524] MODE FOR CARRYING OUT THE INVENTION

[0525] Configuration overview

[0526] This system manages the items in the refrigerator, suggests optimal recipes, and notifies users of any missing ingredients. The system includes a camera inside the refrigerator, a server connected to the Internet, and a user's device (such as a smartphone or tablet).

[0527] Basic operations

[0528] 1. Taking pictures inside the refrigerator

[0529] Device: A camera installed inside the refrigerator takes pictures of the interior periodically or upon user instruction.

[0530] 2. Upload an image

[0531] Terminal: Uploads captured images to a server via Wi-Fi or other communication means.

[0532] 3. Image analysis and item identification

[0533] Server: Analyzes the uploaded images and identifies the items in the refrigerator. This analysis uses machine learning models (e.g., TensorFlow, PyTorch).

[0534] 4. Updating inventory information

[0535] Server: Stores the information of identified items in a database and keeps the inventory information in the refrigerator up to date.

[0536] 5. Recipe suggestions

[0537] Server: Based on the updated inventory information, it calls an external recipe database (e.g., a recipe suggestion service API), matches the available ingredients, and retrieves recipes that can be made.

[0538] 6. Viewing Recipes

[0539] Device: The recipe information sent from the server is displayed on the user's smartphone or tablet. The displayed recipe also includes a list of required ingredients.

[0540] 7. Identification and notification of shortages

[0541] Terminal: Identify missing ingredients from the suggested recipe's ingredient list and notify the user. The list of missing ingredients is displayed on the terminal so the user can review them.

[0542] 8. Check inventory on the go

[0543] Users can access the system from their smartphones or tablets while on the go, such as at the supermarket, and check real-time inventory information and a list of materials that are in short supply.

[0544] 9. Delivery service collaboration (when at home)

[0545] User: If there are missing materials while at home, the system will provide the option to automatically order the missing materials using a delivery service (e.g., a delivery service API).

[0546] 10. Notifications and Optimization

[0547] Server: Notifies the user that the purchase or delivery is complete, updates the refrigerator inventory again, and updates the algorithm to improve the accuracy of future recipe suggestions based on the user's purchase history and frequency of use.

[0548] Specific examples

[0549] User: There are carrots, milk, and eggs in the refrigerator.

[0550] Server: These items are identified through image analysis and stored in a database.

[0551] Server: Recipes such as "stir-fried vegetables" and "omelette" are obtained from the recipe suggestion service.

[0552] Device: The smartphone displays recipes for "Stir-fried vegetables" and "Omelette," along with a list of ingredients needed.

[0553] User: Checks the supermarket for a missing ingredient (e.g., green peppers) and adds it to a shopping list.

[0554] This completes the embodiment of the present invention. This system allows users to efficiently manage inventory in their refrigerators, select optimal recipes, and easily check, purchase, and order ingredients that are in short supply.

[0555] The processing flow will be explained below.

[0556] Step 1:

[0557] Device: A camera installed inside the refrigerator takes pictures of the inside of the refrigerator at regular intervals or as set by the user.

[0558] Step 2:

[0559] Terminal: Uploads captured images to a server via Wi-Fi or other communication means.

[0560] Step 3:

[0561] Server: Preprocesses the received images, adjusting brightness and contrast, and removing noise to improve the accuracy of image recognition.

[0562] Step 4:

[0563] Server: The preprocessed images are input into a machine learning model (e.g., TensorFlow, PyTorch) and analyzed to identify the items in the refrigerator.

[0564] Step 5:

[0565] Server: Stores information about items identified through image analysis in a database and updates the current inventory information in the refrigerator.

[0566] Step 6:

[0567] Server: Based on the updated inventory information, the API of the recipe suggestion service is called, and the available ingredients are matched to obtain recipes that can be made.

[0568] Step 7:

[0569] Server: Selects the optimal recipe from the retrieved recipes and sends that information to the user's device.

[0570] Step 8:

[0571] Device: Recipe information and a list of ingredients are displayed on the user's smartphone or tablet.

[0572] Step 9:

[0573] On the device: Based on the ingredients list of the suggested recipe, identify any missing ingredients and notify the user.

[0574] Step 10:

[0575] Users: Access the system from the supermarket or other locations to view real-time inventory information and shortage lists.

[0576] Step 11:

[0577] User: If there are missing materials while at home, select the delivery service integration option.

[0578] Step 12:

[0579] Server: Calls the delivery service's API to automatically order missing ingredients.

[0580] Step 13:

[0581] Server: Notifies the user once the order is complete and tracks the delivery progress.

[0582] Step 14:

[0583] Terminal: Once the delivery is complete, the user is notified again and the refrigerator inventory information is updated.

[0584] Step 15:

[0585] Server: Updates the algorithm to improve the accuracy of future suggestions based on the user's purchase history and frequency of use.

[0586] Through the above processing steps, users can seamlessly manage inventory in the refrigerator, select recipes, and even purchase missing ingredients or place automatic orders.

[0587] Example 1

[0588] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0589] Conventional refrigerator management systems require users to manually manage refrigerator inventory information, which can result in users losing track of what's in the refrigerator, leading to food waste and unnecessary purchases. Another issue is that checking and purchasing ingredients when they're running low is complicated and time-consuming, making efficient food management difficult.

[0590] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0591] In this invention, the server includes means for taking images of the inside of the refrigerator, means for uploading the taken images to the server, means for identifying items by analyzing the uploaded images, means for saving data on the identified items in a database and updating inventory information, means for calling a recipe suggestion service based on the updated inventory information and suggesting possible recipes, means for displaying an ingredient list for the suggested recipe on a terminal, means for identifying missing ingredients from the ingredient list for the suggested recipe and notifying the user, and means for checking inventory information and missing ingredients using a smart device while on the go. This enables automatic and efficient management of inventory in the refrigerator, allowing users to select optimal recipes and easily check and purchase missing ingredients.

[0592] "Means for taking images inside the refrigerator" refers to the camera installed to capture images inside the refrigerator and the system that controls it in general.

[0593] "Means for uploading captured images to a server" refers to a communication system and software for transmitting captured image data to a server via the Internet or the like.

[0594] "Means for identifying items by analyzing uploaded images" refers to algorithms or machine learning models that use image recognition technology to identify items in the refrigerator from uploaded images.

[0595] "Means for storing data of identified items in a database and updating inventory information" refers to software and hardware for recording recognized item information in a database and keeping inventory information in the refrigerator up to date based on that information.

[0596] "Means for calling a recipe suggestion service based on updated inventory information and suggesting recipes that can be made" refers to a system that obtains appropriate recipe information from an external service based on the latest inventory information in the refrigerator and provides it to the user.

[0597] "Means for displaying a list of ingredients for a proposed recipe on a terminal" refers to an application or interface for displaying recipe information provided by the server on a terminal such as a user's smartphone or tablet.

[0598] "Means for identifying missing ingredients from the ingredient list of a proposed recipe and notifying the user" refers to a system that compares ingredients required for a recipe with current inventory information, detects missing ingredients, and notifies the user.

[0599] "Means for checking inventory information and missing ingredients using smart devices while on the go" refers to a function that allows users to check inventory information and missing ingredients in their refrigerators in real time using a smartphone or tablet while on the go, such as at a supermarket.

[0600] "Means for pre-processing captured images" refers to algorithms or software for performing pre-processing such as resizing, noise removal, and color correction of images before image analysis.

[0601] "Means to order missing ingredients via a delivery service" refers to an interface or API for automatically ordering missing ingredients from a delivery service.

[0602] "Means for updating algorithms to improve the accuracy of recipe suggestions based on purchase history and frequency of use" refers to machine learning models and algorithms that optimize future recipe suggestions by taking into account a user's past purchase history and frequency of use of an item.

[0603] MODE FOR CARRYING OUT THE INVENTION

[0604] The system of the present invention aims to efficiently manage items in a refrigerator, provide optimal recipes to users, and facilitate checking and purchasing of ingredients that are in short supply. The system includes a camera in the refrigerator, a server connected to the Internet, and a user's device (such as a smartphone or tablet).

[0605] Taking pictures inside the refrigerator

[0606] Device: A camera installed inside the refrigerator takes pictures based on user instructions or a regular schedule. When a user presses the "take a picture" button on a smartphone application, the camera activates and captures an image of the inside of the refrigerator. For example, a user may issue a voice command such as "Take a picture of the inside of the refrigerator."

[0607] Uploading an image

[0608] Device: The captured images are automatically uploaded to a server using Wi-Fi or other communication methods. For example, a smartphone application can link with the refrigerator camera and send image data to the server.

[0609] Image analysis and item identification

[0610] Server: The server receives the uploaded images and performs image analysis using machine learning models such as TensorFlow and PyTorch. It identifies the items identified through the analysis and records that information in a database. For example, items such as "carrots," "milk," and "eggs" are identified.

[0611] Inventory updates

[0612] Server: The identified item information is stored in a database and the refrigerator inventory information is updated to the latest state, so that the inventory is always reflected in real time.

[0613] Recipe suggestions

[0614] Server: Based on the latest inventory information, the server calls an external recipe suggestion service API to obtain recipe information that can be made with the current inventory. For example, recipes for "stir-fried vegetables" and "omelette" are obtained from the recipe suggestion service.

[0615] Viewing Recipes

[0616] Device: The recipe information sent from the server is displayed on the user's smartphone or tablet. This information also includes a list of ingredients, which the user can check. Specifically, the smartphone app displays recipes for "stir-fried vegetables" and "omelette" along with their ingredient lists.

[0617] Identifying and notifying shortages

[0618] On the device: Based on the ingredients list of a suggested recipe, identify any missing ingredients and notify the user. For example, if a recipe requires a missing bell pepper, a pop-up notification will appear on the device stating, "You are missing bell peppers."

[0619] Check inventory on the go

[0620] User: While on the go (e.g., at the supermarket), they can use their smartphone to check real-time inventory information and a list of ingredients they are short on. For example, a user opens the app and sees that there are six eggs in the refrigerator.

[0621] Delivery service collaboration (when at home)

[0622] User: If a user is at home and is short on ingredients, the system provides the option to automatically order the missing ingredients using a delivery service API. For example, when a user presses the "Order Delivery" button on their smartphone, the delivery service API is called and green peppers are automatically ordered.

[0623] Optimization based on purchase history and frequency of use

[0624] Server: Once the purchase or delivery is complete, the user is notified and the refrigerator inventory is updated again. The server also updates the algorithm to improve the accuracy of future recipe suggestions based on the user's purchase history and frequency of use. For example, the server can retrain the algorithm based on the purchase data to make better suggestions next time.

[0625] Examples of prompt statements

[0626] "Suggest a recipe that can be made with the items in your refrigerator. Currently, we have carrots, milk, and eggs."

[0627] This invention allows users to efficiently manage their refrigerator inventory, select suitable recipes, and easily check and purchase ingredients they are running low on.

[0628] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0629] Program processing flow

[0630] Step 1:

[0631] Device: A camera installed inside the refrigerator takes pictures based on user instructions or set timing. The input is the user's instructions or schedule, and the output is the image taken inside the refrigerator. For example, when a user presses the "take a picture" button in the app, the camera is activated and an image of the inside of the refrigerator is captured.

[0632] Specific operation: When a user presses the "shoot" button on the smartphone app, the camera starts up and takes a picture of the inside of the refrigerator. At this time, the internal LED light turns on, allowing a clear image to be captured in bright light.

[0633] Step 2:

[0634] Terminal: Acquires captured images and uploads them to a server via Wi-Fi. The input is the captured image data, and the output is the image data uploaded to the server. After the camera captures the image, the smartphone app automatically collects the image data and sends it to the server.

[0635] What it does: After taking a photo, the app uploads the image data to a server via Wi-Fi. This process runs in the background, and users can monitor the progress via a progress bar within the app.

[0636] Step 3:

[0637] Server: The server receives the uploaded images and performs image analysis using a machine learning model (e.g., TensorFlow or PyTorch). The input is the uploaded image data, and the output is a list of identified items. Through analysis, the items in the refrigerator are identified.

[0638] Specific operation: When the server receives the image data, the machine learning model is activated and analyzes the items in the refrigerator. From the analysis results, items such as "carrots," "milk," and "eggs" are identified. The results are recorded in a database.

[0639] Step 4:

[0640] Server: Stores the identified item information in a database and updates the inventory information in the refrigerator. The input is a list of identified items, and the output is updated inventory information. This ensures that the inventory is always updated in real time.

[0641] What happens: The server updates the inventory table in the database, recording new information such as "3 carrots, 1 liter of milk, and 6 eggs."

[0642] Step 5:

[0643] Server: Based on the latest inventory information, it calls an external recipe suggestion service API to obtain recipe information that can be made. The input is the latest inventory information, and the output is the suggested recipe information.

[0644] Specific operation: The server calls the recipe suggestion service API and sends the current inventory information. For example, based on the inventory information for "carrots, milk, and eggs," recipe information for "stir-fried vegetables" and "omelette" is obtained.

[0645] Step 6:

[0646] Device: The recipe information sent from the server is displayed on the user's smartphone or tablet. The input is the suggested recipe information, and the output is the recipe information displayed on the device.

[0647] What it does: The app on the user's smartphone displays recipes for "stir-fried vegetables" and "omelette," along with a list of ingredients. A detailed explanation of the recipe and cooking instructions are also displayed.

[0648] Step 7:

[0649] Terminal: Based on the recipe ingredient list, it identifies missing ingredients and notifies the user. The input is the suggested recipe information and current inventory information, and the output is the list of missing ingredients and a notification.

[0650] What it does: The device checks the ingredients in the suggested recipe against the current inventory and displays a notification to the user saying, "You're low on bell peppers." The list of ingredients you're short on is available in the app.

[0651] Step 8:

[0652] User: Check real-time inventory information and material shortage lists on the go using a smartphone or tablet. The input is an access request from the smartphone, and the output is real-time inventory information.

[0653] Specific behavior: When a user opens the app at a supermarket and taps the stock check button, they can see that there are 6 eggs left.

[0654] Step 9:

[0655] User: When there are missing materials while at home, the system automatically orders the missing materials using the delivery service API. The input is a list of missing materials, and the output is a notification that the delivery order has been completed.

[0656] Specific operation: When a user presses the "Order Delivery" button in the app, the delivery service API is called and "green peppers" are automatically ordered. After the order is completed, a notification is displayed on the smartphone.

[0657] Step 10:

[0658] Server: Notifies the user that the purchase or delivery is complete and updates the refrigerator inventory information again. It also updates the algorithm to improve the accuracy of recipe suggestions based on purchase history and frequency of use. The input is the purchase completion notification, and the output is the updated inventory information and the algorithm.

[0659] Specific operation: When the delivery is complete, a notification saying "Peppers delivery completed" is displayed on the user's device. The server updates the inventory information in the database and retrains the algorithm based on the purchase data to improve the accuracy of the next recommendation.

[0660] (Application example 1)

[0661] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0662] In modern life, managing ingredients in the refrigerator is complicated, and the hassle of checking for missing ingredients and the stress of selecting the optimal recipe are issues. Many users also find it troublesome to check inventory while on the go and to purchase or order missing ingredients. Therefore, there is a need for a system that can efficiently manage refrigerator inventory information, suggest optimal recipes, and automatically order missing ingredients.

[0663] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0664] In this invention, the server includes a means for taking images of the inside of the refrigerator, a means for uploading the taken images to the server, a means for analyzing the uploaded images to identify items, and a means for automatically ordering missing ingredients from a food delivery service. This makes it possible to efficiently manage inventory information in the refrigerator, propose optimal recipes, and automatically order missing ingredients.

[0665] The "means for taking images inside the refrigerator" is a device that uses a camera installed inside the refrigerator to take images of items inside the refrigerator periodically or based on a user's instruction.

[0666] The "means for uploading captured images to a server" is a communication function for transmitting captured image data to a server via the Internet.

[0667] The "means of identifying items by analyzing uploaded images" refers to a technology that uses machine learning algorithms and image analysis technology on a server to recognize and identify each item in a refrigerator from uploaded images.

[0668] The "means for storing data on identified items in a database and updating inventory information" is a system for storing analyzed item information in a database and keeping the current inventory information in the refrigerator up to date.

[0669] "Means of suggesting recipes that can be made by calling the API of a recipe suggestion service based on updated inventory information" refers to a method of using the API of an external recipe suggestion service based on the latest inventory information to obtain recipes for dishes that can be made based on current inventory.

[0670] The "means for displaying the ingredient list of the proposed recipe on a terminal" is a means for displaying the acquired recipe information and the list of necessary ingredients on a terminal such as a user's smartphone or tablet.

[0671] The "means for automatically ordering missing ingredients from a food delivery service" is a technology that determines which ingredients are missing to create a recipe and automatically orders the missing ingredients from a food delivery service via the Internet.

[0672] "Means for pre-processing uploaded images" refers to methods for pre-processing captured images, such as filtering and resizing, in order to improve image quality or extract necessary information before image analysis.

[0673] The "means for notifying the user of missing ingredients" is a technology that sends a notification message to a smartphone, tablet, etc. to inform the user of any missing ingredients needed for a recipe proposal.

[0674] An embodiment of the present invention will now be described. The system of the present invention efficiently manages items in a refrigerator, proposes optimal recipes, and automatically orders ingredients that are in short supply. This system includes the following components: a camera in the refrigerator, a server connected to the Internet, and a user terminal (such as a smartphone or tablet).

[0675] First, the basic operation steps of the system will be explained.

[0676] Basic operations

[0677] 1. Taking pictures inside the refrigerator

[0678] Terminal: A camera installed inside the refrigerator takes pictures of the inside periodically or based on the user's instructions.

[0679] 2. Upload an image

[0680] Terminal: Uploads captured images to a server via Wi-Fi or other communication means.

[0681] 3. Image analysis and item identification

[0682] Server: Analyzes the uploaded images and identifies the items in the refrigerator. This analysis uses machine learning models such as TensorFlow and PyTorch.

[0683] 4. Updating inventory information

[0684] Server: Stores the information of identified items in a database and keeps the inventory information in the refrigerator up to date.

[0685] 5. Recipe suggestions

[0686] Server: Based on the updated inventory information, it calls an external recipe database (e.g., a recipe suggestion service API), matches the available ingredients, and retrieves recipes that can be made.

[0687] 6. Viewing Recipes

[0688] Device: The recipe information sent from the server is displayed on the user's smartphone or tablet. The displayed recipe also includes a list of necessary ingredients.

[0689] 7. Identification and notification of shortages

[0690] Terminal: Identify missing ingredients from the suggested recipe's ingredient list and notify the user. The list of missing ingredients is displayed on the terminal so the user can review them.

[0691] 8. Automatic ordering of missing materials

[0692] Server: Automatically orders missing ingredients from a food delivery service, using the food delivery service's API.

[0693] Hardware and software used

[0694] 1. Refrigerator camera

[0695] Example: Camera using the Raspberry Pi camera module

[0696] 2. Internet connection environment

[0697] Example: Wi-Fi router

[0698] 3. User's Device

[0699] Example: Smartphones and tablets

[0700] 4. Server

[0701] Programming: Python

[0702] Image processing: OpenCV

[0703] Item identification: Machine learning libraries such as TensorFlow and PyTorch

[0704] Communication:requests library

[0705] Specific examples

[0706] A user uses the "SmartKitchen" app while there is chicken, onions, and carrots in the refrigerator. The camera inside the refrigerator automatically takes a picture, and the app suggests recipes for "chicken teriyaki" and "curry." Because the user is low on onions, the app automatically places an order with the nearest food delivery service and notifies the user.

[0707] Prompt Sentence Examples

[0708] Prompt for an application that determines what ingredients are in the refrigerator, suggests recipes to the user, and automatically orders any missing ingredients:

[0709] Identify ingredients from images of your refrigerator and suggest recipes based on your current inventory, including automatically ordering missing ingredients through a food delivery service.

[0710] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0711] Step 1:

[0712] Taking pictures inside the refrigerator

[0713] The device takes pictures using a camera installed inside the refrigerator. In this process, the camera module captures images of the inside of the refrigerator periodically or based on the user's instructions. The input is the current state of the inside of the refrigerator, and the output is a captured image file.

[0714] Step 2:

[0715] Uploading an image

[0716] The device uploads the captured image file to the server. In this process, the image data is sent to the server via an internet connection. The input is the captured image file, and the output is the image data stored on the server.

[0717] Step 3:

[0718] Image analysis and item identification

[0719] The server analyzes the uploaded image data and identifies the items in the refrigerator. This process uses machine learning models (e.g., TensorFlow, PyTorch) to analyze the image data and identify specific items. The input is the uploaded image data, and the output is a list of identified items.

[0720] Step 4:

[0721] Inventory updates

[0722] The server saves the identified item list in a database and updates the inventory information. This process writes the identified item data to the database and updates the refrigerator inventory information. The input is the identified item list, and the output is the updated inventory information.

[0723] Step 5:

[0724] Recipe suggestions

[0725] Based on the updated inventory information, the server calls the API of the recipe suggestion service to suggest recipes that can be made. This process calls an external recipe database API to obtain appropriate recipes based on the inventory information in the refrigerator. The input is the updated inventory information, and the output is a list of suggested recipes.

[0726] Step 6:

[0727] Viewing Recipes

[0728] The device displays the recipe information sent from the server on the user's smartphone or tablet. In this process, the suggested recipe and the list of required ingredients are displayed on the device. The input is the suggested recipe list, and the output is the recipe information displayed on the user's device.

[0729] Step 7:

[0730] Identifying and notifying shortages

[0731] The terminal identifies missing ingredients from the ingredient list of the proposed recipe and notifies the user. This process compares the missing ingredients with current inventory information and sends a notification message to the user. The input is the ingredient list of the proposed recipe and current inventory information, and the output is a missing ingredient list and a notification message.

[0732] Step 8:

[0733] Automatic ordering of missing materials

[0734] The server automatically orders the missing ingredients from a food delivery service. This process uses the list of missing ingredients to call the food delivery service's API and confirm the order. The input is the list of missing ingredients, and the output is order confirmation information.

[0735] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0736] MODE FOR CARRYING OUT THE INVENTION

[0737] Configuration overview

[0738] This system manages the items in the refrigerator, suggests optimal recipes, notifies users of missing ingredients, and also incorporates a user emotion recognition engine. The system includes a camera in the refrigerator, an internet-connected server, a user device (such as a smartphone or tablet), and an emotion recognition engine.

[0739] Basic operations

[0740] 1. Taking pictures inside the refrigerator

[0741] Device: A camera installed inside the refrigerator takes pictures of the interior periodically or upon user instruction.

[0742] 2. Upload an image

[0743] Terminal: Uploads captured images to a server via Wi-Fi or other communication means.

[0744] 3. Image analysis and item identification

[0745] Server: Preprocesses the received images, adjusting brightness and contrast, and removing noise to improve the accuracy of image recognition.

[0746] Server: The preprocessed images are input into a machine learning model (e.g., TensorFlow, PyTorch) and analyzed to identify the items in the refrigerator.

[0747] 4. Updating inventory information

[0748] Server: Stores information about items identified through image analysis in a database and updates the current inventory information in the refrigerator.

[0749] 5. Recipe suggestions

[0750] Server: Based on the updated inventory information, the API of the recipe suggestion service is called, and the available ingredients are matched to obtain recipes that can be made.

[0751] 6. Emotion recognition

[0752] Device: The camera and microphone on the user's device are used to collect facial and voice data.

[0753] Emotion engine: Analyzes collected data and determines the user's emotional state (e.g., joy, sadness, stress, etc.).

[0754] 7. Optimizing recipe suggestions

[0755] Server: Based on the emotional information determined by the emotion engine, the server suggests recipes that suit the user's current emotional state.

[0756] 8. Viewing Recipes

[0757] Device: The recipe information sent from the server is displayed on the user's smartphone or tablet. The displayed recipe also includes a list of required ingredients.

[0758] 9. Identification and Notification of Shortage Materials

[0759] Terminal: Identify missing ingredients from the suggested recipe's ingredient list and notify the user of this information.

[0760] 10. Check inventory on the go

[0761] Users: Access the system from their smartphones or tablets while on the go, such as at the supermarket, to check real-time inventory information and a list of materials that are in short supply.

[0762] 11. Delivery service collaboration (when at home)

[0763] User: If there are missing materials while at home, select the delivery service integration option.

[0764] Server: Calls the delivery service's API to automatically order missing ingredients.

[0765] 12. Notifications and Optimization

[0766] Server: Notifies the user that the purchase or delivery is complete and tracks the delivery progress.

[0767] Terminal: Once the delivery is complete, the user is notified again and the refrigerator inventory information is updated.

[0768] Server: Updates the algorithm to improve the accuracy of future suggestions based on the user's purchase history and frequency of use.

[0769] By incorporating a user emotion recognition engine, recipes can be suggested based on the user's current emotions, providing a more personalized experience.

[0770] Specific examples

[0771] User: There are carrots, milk, and eggs in the refrigerator. The user comes home from work and is stressed.

[0772] Server: These items are identified through image analysis and stored in a database.

[0773] Emotion engine: Determines the user's stress level from facial expressions and voice data, and prioritizes recipes with a relaxing effect.

[0774] Server: Recipes that contribute to relaxation, such as "carbonara," are obtained from a recipe suggestion service.

[0775] Device: The smartphone will display a recipe for "Carbonara" along with a list of ingredients needed.

[0776] User: Checks the supermarket for a missing ingredient (e.g., bacon) and adds it to a shopping list.

[0777] In this way, this system allows users to efficiently manage their refrigerator inventory and also suggests optimal recipes that suit their emotional state at the time, resulting in a more satisfying cooking experience.

[0778] The processing flow will be explained below.

[0779] Step 1:

[0780] Device: A camera installed inside the refrigerator takes pictures of the inside of the refrigerator at regular intervals or as set by the user.

[0781] Step 2:

[0782] Terminal: Uploads captured images to a server via Wi-Fi or other communication means.

[0783] Step 3:

[0784] Server: Preprocesses the received images, adjusting brightness and contrast, and removing noise to improve the accuracy of image recognition.

[0785] Step 4:

[0786] Server: The preprocessed images are input into a machine learning model (e.g., TensorFlow, PyTorch) and analyzed to identify the items in the refrigerator.

[0787] Step 5:

[0788] Server: Stores information about items identified through image analysis in a database and updates the current inventory information in the refrigerator.

[0789] Step 6:

[0790] Server: Based on the updated inventory information, the API of the recipe suggestion service is called, and the available ingredients are matched to obtain recipes that can be made.

[0791] Step 7:

[0792] Device: The camera and microphone on the user's device are used to collect facial and voice data.

[0793] Step 8:

[0794] Emotion engine: Analyzes collected data and determines the user's emotional state (e.g., joy, sadness, stress, etc.).

[0795] Step 9:

[0796] Server: Based on the emotional information determined by the emotion engine, the server optimizes and suggests recipes that suit the user's current emotional state.

[0797] Step 10:

[0798] Device: The recipe information sent from the server is displayed on the user's smartphone or tablet. The displayed recipe also includes a list of required ingredients.

[0799] Step 11:

[0800] On the device: Based on the ingredients list of the suggested recipe, identify any missing ingredients and notify the user.

[0801] Step 12:

[0802] Users: Access the system from their smartphones or tablets while on the go, such as at the supermarket, to check real-time inventory information and a list of materials that are in short supply.

[0803] Step 13:

[0804] User: If there are missing materials while at home, select the delivery service integration option.

[0805] Step 14:

[0806] Server: Calls the delivery service's API to automatically order missing ingredients.

[0807] Step 15:

[0808] Server: Notifies the user once the order is complete and tracks the delivery progress.

[0809] Step 16:

[0810] Terminal: Once the delivery is complete, the user is notified again and the refrigerator inventory information is updated.

[0811] Step 17:

[0812] Server: Updates the algorithm to improve the accuracy of future suggestions based on the user's purchase history and frequency of use.

[0813] Example 2

[0814] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0815] Conventional refrigerator management systems are limited in their ability to manage refrigerator inventory and suggest recipes, making it difficult to provide personalized services that reflect the user's emotional state and circumstances. Furthermore, there are few ways to check inventory information while on the go, making it difficult to take appropriate action when necessary ingredients are unavailable.

[0816] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a means for capturing images of the inside of the refrigerator, a means for uploading the captured images to the server, a means for preprocessing and analyzing the uploaded images to identify items, a means for saving data on the identified items in a database and updating inventory information, a means for suggesting possible recipes by calling an API of a recipe suggestion service based on the updated inventory information, a means for recognizing the user's emotional state and selecting an optimal recipe, and a means for displaying an ingredient list for the suggested recipe on a terminal. This allows the user to efficiently manage inventory in the refrigerator and receive personalized recipe suggestions based on their emotional state. Furthermore, by checking inventory information while on the go and receiving notifications of missing ingredients, the user can appropriately procure the necessary ingredients.

[0817] The "means for taking images inside the refrigerator" is a mechanism for taking images inside the refrigerator using a camera installed inside the refrigerator.

[0818] The "means for uploading captured images to a server" is a mechanism for transmitting captured image data to a server via a communication means.

[0819] The "means for preprocessing and analyzing uploaded images to identify items" refers to a mechanism that performs preprocessing such as adjusting the brightness and contrast of received image data and removing noise, and then identifies items using a machine learning model.

[0820] The "means for saving data on identified items in a database and updating inventory information" is a mechanism for saving information on items identified by analysis in a database and updating the latest inventory status.

[0821] "Means for calling the API of a recipe suggestion service based on updated inventory information and suggesting recipes that can be made" is a mechanism for calling the API of an external recipe suggestion service based on the latest inventory information and obtaining recipe information that can be made.

[0822] The "means for recognizing the user's emotional state and selecting the optimal recipe" is a mechanism for analyzing the user's facial expressions and voice data to determine their emotional state and select a recipe that is appropriate for that state.

[0823] The "means for displaying the ingredient list of the proposed recipe on the terminal" is a mechanism for displaying the recipe information and ingredient list sent from the server on the user's smartphone or tablet.

[0824] The present invention relates to a system that manages items in a refrigerator, suggests optimal recipes based on the user's emotional state, and notifies the user of any missing ingredients. The system includes a camera in the refrigerator, a server connected to the Internet, a user device (such as a smartphone or tablet), and an emotion engine.

[0825] System Configuration

[0826] 1. Camera in the refrigerator

[0827] Hardware: Camera device

[0828] Software: Image acquisition program

[0829] Function: Takes pictures of the inside of the refrigerator periodically or at the user's command.

[0830] 2. Server

[0831] Hardware: Server Device

[0832] Software: Image preprocessing program, machine learning model (e.g., TensorFlow, PyTorch), recipe suggestion service API, database management system

[0833] Function: Preprocesses received images to identify and specify items. Stores information about identified items in a database and updates inventory information in the refrigerator. Calls the API of a recipe suggestion service based on the updated inventory information and obtains information about recipes that can be created.

[0834] 3. User Device

[0835] Hardware: Smartphones and tablet devices

[0836] Software: Display program, notification program, emotion recognition function

[0837] Function: Displays recipe information sent from the server to the user and notifies them of missing ingredients. Collects the user's facial expressions and voice data and analyzes them with the emotion engine.

[0838] 4. Emotion Engine

[0839] Software: Emotional state analysis algorithm

[0840] Function: Analyzes collected user facial and voice data to determine the user's emotional state. Based on the determined emotional information, selects a recipe that is appropriate for the user's current emotional state.

[0841] Specific examples

[0842] Example 1:

[0843] User: There are carrots, milk, and eggs in the refrigerator. The user comes home from work and is stressed.

[0844] Server: These items are identified through image analysis and stored in a database.

[0845] Emotion engine: Determines the user's stress level from facial expressions and voice data, and prioritizes recipes with a relaxing effect.

[0846] Server: Recipes that contribute to relaxation, such as "carbonara," are obtained from a recipe suggestion service.

[0847] Device: The smartphone will display a recipe for "Carbonara" along with a list of ingredients needed.

[0848] User: Checks the supermarket for a missing ingredient (e.g., bacon) and adds it to a shopping list.

[0849] Example 2 (An example of a prompt using a generative AI model):

[0850] Prompt: "The ingredients in my fridge are carrots, potatoes, cheese, and eggs. I'm feeling relaxed today, so can you recommend a recipe that will fit that mood?"

[0851] Server: Performs image analysis and emotion recognition to suggest relaxing recipes such as "cheese omelet."

[0852] This allows the system to provide a more satisfying cooking experience by allowing users to efficiently manage inventory in the refrigerator, suggesting optimal recipes based on their emotional state at the time, and notifying them of any missing ingredients.

[0853] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0854] Step 1:

[0855] Device: A camera installed inside the refrigerator takes images of the inside of the refrigerator periodically or based on the user's instructions.

[0856] Input: User instruction or timer event.

[0857] Data processing: Acquisition of image data.

[0858] Output: A captured image of the inside of a refrigerator.

[0859] Step 2:

[0860] Terminal: Uploads captured images to a server via Wi-Fi or other communication means.

[0861] Input: Captured image data.

[0862] Data processing: Transfer of image data.

[0863] Output: Image data sent to the server.

[0864] Step 3:

[0865] Server: Preprocesses the received images, adjusting brightness and contrast and removing noise.

[0866] Input: Image data sent to the server.

[0867] Data processing: Image preprocessing (brightness, contrast adjustment, noise removal).

[0868] Output: Preprocessed image data.

[0869] Step 4:

[0870] Server: The preprocessed image data is input into a machine learning model (e.g., TensorFlow, PyTorch) and analyzed to identify the items in the refrigerator.

[0871] Input: Preprocessed image data.

[0872] Data processing: Identification and labeling of items.

[0873] Output: A list of identified items.

[0874] Step 5:

[0875] Server: Stores information about items identified through analysis in a database and updates the inventory information in the refrigerator.

[0876] Input: A list of identified items.

[0877] Data processing: storing information in the database and updating inventory information.

[0878] Output: Updated inventory information.

[0879] Step 6:

[0880] Server: Based on the updated inventory information, the API of the recipe suggestion service is called, and the available ingredients are matched to obtain recipes that can be made.

[0881] Input: Updated inventory information.

[0882] Data processing: Calling the recipe suggestion service API and obtaining recipes.

[0883] Output: A list of suggested recipes.

[0884] Step 7:

[0885] Terminal: Collects the user's facial expressions and voice data and sends them to the emotion engine to recognize the user's emotional state.

[0886] Input: User's facial and voice data.

[0887] Data processing: Collecting and transmitting emotion data.

[0888] Output: Data sent to the emotion engine.

[0889] Step 8:

[0890] Emotion engine: Analyzes collected data and determines the user's emotional state.

[0891] Input: User's facial and voice data.

[0892] Data processing: Analysis of emotional state (e.g., stress, joy).

[0893] Output: Determined emotional state.

[0894] Step 9:

[0895] Server: Based on the emotional information determined by the emotion engine, selects a recipe that is appropriate for the user's current emotional state.

[0896] Input: A list of identified emotional states and suggested recipes.

[0897] Data processing: Recipe selection based on emotional state.

[0898] Output: The best recipe for the user.

[0899] Step 10:

[0900] Terminal: The optimal recipe information sent from the server is displayed on the terminal, and the user is notified of the list of necessary ingredients.

[0901] Input: The selected optimal recipe.

[0902] Data processing: Display recipe information.

[0903] Output: Recipe and ingredients list displayed on the user's device.

[0904] Step 11:

[0905] On the device: Based on the ingredients list of a suggested recipe, identify any missing ingredients and notify the user.

[0906] Input: Materials list and current inventory information.

[0907] Data processing: Identifying missing materials.

[0908] Output: Notify user of missing material.

[0909] Step 12:

[0910] User: Accesses the system from a terminal while on the go, such as at a supermarket, to check refrigerator inventory information and a list of ingredients that are in short supply.

[0911] Input: The user's access request.

[0912] Data processing: Providing real-time inventory information.

[0913] Output: Inventory information and a list of missing materials displayed on the user's terminal.

[0914] Step 13:

[0915] User: When at home, select the delivery service integration option to automatically order missing ingredients.

[0916] Input: User's delivery service selection.

[0917] Data processing: Sending orders to delivery services.

[0918] Output: Delivery order confirmation.

[0919] Step 14:

[0920] Server: Notifies the user that the purchase or delivery is complete and tracks the delivery progress.

[0921] Input: Delivery completion information.

[0922] Data Processing: Delivery progress tracking and notification.

[0923] Output: Delivery completion notification to the user.

[0924] Step 15:

[0925] Terminal: Once the delivery is complete, the user is notified again and the refrigerator inventory is updated.

[0926] Input: Delivery completion information for snooze.

[0927] Data processing: Updating refrigerator inventory information.

[0928] Output: Notification of updated inventory information.

[0929] Step 16:

[0930] Server: Updates the algorithm to improve the accuracy of future suggestions based on the user's purchase history and frequency of use.

[0931] Input: Purchase history and usage frequency data.

[0932] Data processing: Algorithm updates and optimizations.

[0933] Output: Improved accuracy of suggestions for the next time.

[0934] (Application example 2)

[0935] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0936] The present invention aims to improve user satisfaction and convenience by providing a refrigerator inventory management and recipe suggestion system that not only efficiently manages items in a refrigerator, but also suggests optimal cooking instructions based on the user's emotional state and provides a series of processes including automatic ordering of missing ingredients. However, conventional refrigerator management systems have problems in that they do not take into account responses based on the user's emotions and do not have a function for automatically ordering missing ingredients.

[0937] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0938] In this invention, the server includes means for taking images of the inside of the refrigerator, means for uploading the taken images to the server, means for analyzing the uploaded images to identify items, means for saving data on the identified items in a database and updating inventory information, means for calling an API of a recipe suggestion service based on the updated inventory information to provide creatable cooking instructions, means for collecting facial expression and voice data using a camera and microphone of the terminal to acquire the user's emotional state, means for analyzing the collected data to determine the user's emotion, means for proposing cooking instructions according to the determined emotion, means for displaying an ingredient list for the proposed cooking instructions on the terminal, and means for identifying missing items from the suggested ingredient list and automatically ordering them. This allows the user to not only manage inventory in the refrigerator, but also to suggest cooking instructions according to the user's emotional state and automatically order missing ingredients.

[0939] The "means for taking images inside the refrigerator" is a device that uses a camera installed inside the refrigerator to take images of items inside the refrigerator periodically or in response to instructions.

[0940] The "means for uploading captured images to a server" is a communication device that transfers captured images to a server via the Internet.

[0941] The "means for analyzing uploaded images to identify items" is a device that analyzes the images received on the server and uses image recognition algorithms to identify the items in the refrigerator.

[0942] The "means for saving data on identified items in a database and updating inventory information" is a device that saves information on items identified through analysis in a database and updates the inventory status in the refrigerator to the latest status.

[0943] "Means for providing cooking instructions that can be created by calling the API of a recipe suggestion service based on updated inventory information" refers to a device that uses the latest inventory information to access the API of a recipe suggestion service and obtains cooking instructions based on that information.

[0944] "Means for collecting facial expression and voice data using a camera or microphone on a terminal to obtain the emotional state of a user" refers to a device that collects facial expression and voice data using a camera or microphone installed on a terminal in order to grasp the emotional state of a user.

[0945] The "means for analyzing collected data to determine the user's emotions" is a device that analyzes collected facial and voice data and uses emotion recognition algorithms to identify the user's emotional state.

[0946] The "means for proposing cooking instructions according to the determined emotion" is a device that selects and provides optimal cooking instructions based on the user's emotional state.

[0947] The "means for displaying an ingredient list for the proposed cooking instructions on a terminal" is a device that displays an ingredient list based on the proposed cooking instructions on a user's terminal.

[0948] The "means for identifying missing items in the proposed ingredient list and automatically ordering them" refers to a device that verifies the ingredient list based on the proposed cooking instructions, identifies missing items, and automatically orders them.

[0949] System Configuration

[0950] The present invention relates to a "refrigerator item management and recipe suggestion system based on emotional state" and uses the following hardware and software.

[0951] Hardware used:

[0952] Refrigerator camera (connectable Wi-Fi camera)

[0953] User devices (smartphones, tablets)

[0954] server

[0955] Software used:

[0956] Emotion recognition engine (e.g. Affectiva, Microsoft Emotion API)

[0957] Image analysis engine (e.g. TensorFlow, PyTorch)

[0958] Cloud databases (e.g. AWS DynamoDB, Google Cloud Firestore)

[0959] Delivery service APIs (e.g. UberEats, DoorDash)

[0960] Basic operations

[0961] Taking and uploading images of the inside of the refrigerator

[0962] The device sends instructions to a camera installed inside the refrigerator, taking pictures of the inside of the refrigerator periodically or based on the user's instructions. The captured image data is then uploaded to a server via Wi-Fi.

[0963] Image analysis and item identification

[0964] The server preprocesses the uploaded images and uses image recognition algorithms (such as TensorFlow or PyTorch) to identify the items in the refrigerator. The identified item data is stored in a cloud database and inventory information is updated.

[0965] Acquiring and analyzing emotional states

[0966] The device collects facial expression and voice data of the user to obtain the user's emotional state, and the collected data is analyzed using an emotion recognition engine to determine the user's emotional state.

[0967] Recipe suggestions and ingredient lists

[0968] The server then calls the API of the recipe suggestion service to obtain optimal cooking instructions based on the updated inventory information and emotional state. The obtained cooking instructions and ingredient list are then displayed on the user's device.

[0969] Automatic ordering of missing materials

[0970] The terminal identifies any missing items from the proposed materials list and automatically orders them through the delivery service API, with the order status tracked and notified to the user in real time.

[0971] Specific examples

[0972] After the user returns home from work, an image of the contents of the refrigerator is automatically uploaded to the app. The app then acquires the user's facial expression data and determines their emotional state as "tired." Based on this information, the server analyzes the inventory and determines that carbonara is ready to be made, but that there is a shortage of bacon. When the user selects the delivery option, the bacon is automatically ordered, and the progress is displayed in the app.

[0973] Prompt Sentence Examples

[0974] After the user returns home, an image of the inside of the refrigerator and the user's facial expression data are collected from the smartphone. Image analysis is used to check the inventory in the refrigerator, and if fatigue is determined from the facial expression data, what recipes should be suggested and how to order missing ingredients via a delivery service? Please explain this with specific steps.

[0975] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0976] Step 1:

[0977] The user issues a command to take a picture of the refrigerator from the terminal. The input is the user's command, and the output is image data of the inside of the refrigerator. Specifically, the terminal sends a command to take a picture to the camera installed inside the refrigerator, and the camera takes a picture of the current inside of the refrigerator.

[0978] Step 2:

[0979] The captured image data is uploaded from the device to the server. The input is the image data of the inside of the refrigerator, and the output is an image file saved on the server. Specifically, the device transfers the image data to the server via Wi-Fi.

[0980] Step 3:

[0981] The server preprocesses the uploaded image data and performs image analysis. The input is the image file stored on the server, and the output is the data of the identified items in the refrigerator. Specifically, the server adjusts the brightness and contrast of the image and uses a machine learning model to identify the items.

[0982] Step 4:

[0983] The identified item data is stored in a database, and inventory information is updated. The input is the identified item data in the refrigerator, and the output is the updated inventory information. Specifically, the server connects to the database, stores the latest item information, and updates the inventory information.

[0984] Step 5:

[0985] The user's device collects facial and voice data to acquire the user's emotional state. The input is the user's real-time facial and voice data, and the output is the collected data. Specifically, the device collects the data using a camera and microphone.

[0986] Step 6:

[0987] The collected data is sent to a server and analyzed by an emotion recognition engine. The input is the collected facial and voice data, and the output is the user's emotional state. Specifically, the server uses an emotion recognition algorithm to determine the user's emotions.

[0988] Step 7:

[0989] Based on the updated inventory information and emotional state, the server calls the API of the recipe suggestion service to obtain optimal cooking instructions. The input is the updated inventory information and the user's emotional state, and the output is cooking instructions. Specifically, the server calls the API of the recipe suggestion service to obtain cooking instructions.

[0990] Step 8:

[0991] The acquired cooking instructions and ingredient list are displayed on the user's terminal. The input is the cooking instructions and ingredient list, and the output is the information displayed on the terminal. Specifically, the server sends this information to the terminal, and the terminal displays it.

[0992] Step 9:

[0993] The user's device identifies missing items from the proposed materials list and automatically places an order. The input is the materials list and inventory information, and the output is order information for the delivery service. Specifically, the device checks the inventory information to identify missing items and calls the delivery service's API to place the order.

[0994] Step 10:

[0995] The user is notified of the order progress and the inventory information is updated after delivery is completed. The input is the progress of the delivery service, and the output is the updated inventory information and a notification to the user. Specifically, the server receives information from the delivery service, updates the inventory information, and notifies the user.

[0996] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0997] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0998] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0999] [Third embodiment]

[1000] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1001] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1002] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1003] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1004] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1005] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1006] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1007] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1008] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1009] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1010] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1011] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1012] MODE FOR CARRYING OUT THE INVENTION

[1013] Configuration overview

[1014] This system manages the items in the refrigerator, suggests optimal recipes, and notifies users of any missing ingredients. The system includes a camera inside the refrigerator, a server connected to the Internet, and a user's device (such as a smartphone or tablet).

[1015] Basic operations

[1016] 1. Taking pictures inside the refrigerator

[1017] Device: A camera installed inside the refrigerator takes pictures of the interior periodically or upon user instruction.

[1018] 2. Upload an image

[1019] Terminal: Uploads captured images to a server via Wi-Fi or other communication means.

[1020] 3. Image analysis and item identification

[1021] Server: Analyzes the uploaded images and identifies the items in the refrigerator. This analysis uses machine learning models (e.g., TensorFlow, PyTorch).

[1022] 4. Updating inventory information

[1023] Server: Stores the information of identified items in a database and keeps the inventory information in the refrigerator up to date.

[1024] 5. Recipe suggestions

[1025] Server: Based on the updated inventory information, it calls an external recipe database (e.g., a recipe suggestion service API), matches the available ingredients, and retrieves recipes that can be made.

[1026] 6. Viewing Recipes

[1027] Device: The recipe information sent from the server is displayed on the user's smartphone or tablet. The displayed recipe also includes a list of required ingredients.

[1028] 7. Identification and notification of shortages

[1029] Terminal: Identify missing ingredients from the suggested recipe's ingredient list and notify the user. The list of missing ingredients is displayed on the terminal, allowing the user to review them.

[1030] 8. Check inventory on the go

[1031] Users can access the system from their smartphones or tablets while on the go, such as at the supermarket, and check real-time inventory information and a list of materials that are in short supply.

[1032] 9. Delivery service collaboration (when at home)

[1033] User: If there are missing materials while at home, the system will provide the option to automatically order the missing materials using a delivery service (e.g., a delivery service API).

[1034] 10. Notifications and Optimization

[1035] Server: Notifies the user that the purchase or delivery is complete, updates the refrigerator inventory again, and updates the algorithm to improve the accuracy of future recipe suggestions based on the user's purchase history and frequency of use.

[1036] Specific examples

[1037] User: There are carrots, milk, and eggs in the refrigerator.

[1038] Server: These items are identified through image analysis and stored in a database.

[1039] Server: Recipes such as "stir-fried vegetables" and "omelette" are obtained from the recipe suggestion service.

[1040] Device: The smartphone displays recipes for "Stir-fried vegetables" and "Omelette," along with a list of ingredients needed.

[1041] User: Checks the supermarket for a missing ingredient (e.g., green peppers) and adds it to a shopping list.

[1042] This completes the embodiment of the present invention. This system allows users to efficiently manage inventory in their refrigerators, select optimal recipes, and easily check, purchase, and order ingredients that are in short supply.

[1043] The processing flow will be explained below.

[1044] Step 1:

[1045] Device: A camera installed inside the refrigerator takes pictures of the inside of the refrigerator at regular intervals or as set by the user.

[1046] Step 2:

[1047] Terminal: Uploads captured images to a server via Wi-Fi or other communication means.

[1048] Step 3:

[1049] Server: Preprocesses the received images, adjusting brightness and contrast, and removing noise to improve the accuracy of image recognition.

[1050] Step 4:

[1051] Server: The preprocessed images are input into a machine learning model (e.g., TensorFlow, PyTorch) and analyzed to identify the items in the refrigerator.

[1052] Step 5:

[1053] Server: Stores information about items identified through image analysis in a database and updates the current inventory information in the refrigerator.

[1054] Step 6:

[1055] Server: Based on the updated inventory information, the API of the recipe suggestion service is called, and the available ingredients are matched to obtain recipes that can be made.

[1056] Step 7:

[1057] Server: Selects the optimal recipe from the retrieved recipes and sends that information to the user's device.

[1058] Step 8:

[1059] Device: Recipe information and a list of ingredients are displayed on the user's smartphone or tablet.

[1060] Step 9:

[1061] On the device: Based on the ingredients list of the suggested recipe, identify any missing ingredients and notify the user.

[1062] Step 10:

[1063] Users: Access the system from the supermarket or other locations to view real-time inventory information and shortage lists.

[1064] Step 11:

[1065] User: If there are missing materials while at home, select the delivery service integration option.

[1066] Step 12:

[1067] Server: Calls the delivery service's API to automatically order missing ingredients.

[1068] Step 13:

[1069] Server: Notifies the user once the order is complete and tracks the delivery progress.

[1070] Step 14:

[1071] Terminal: Once the delivery is complete, the user is notified again and the refrigerator inventory information is updated.

[1072] Step 15:

[1073] Server: Updates the algorithm to improve the accuracy of future suggestions based on the user's purchase history and frequency of use.

[1074] Through the above processing steps, users can seamlessly manage inventory in the refrigerator, select recipes, and even purchase missing ingredients or place automatic orders.

[1075] Example 1

[1076] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1077] Conventional refrigerator management systems require users to manually manage refrigerator inventory information, which can result in users losing track of what's in the refrigerator, leading to food waste and unnecessary purchases. Another issue is that checking and purchasing ingredients when they're running low is complicated and time-consuming, making efficient food management difficult.

[1078] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1079] In this invention, the server includes means for taking images of the inside of the refrigerator, means for uploading the taken images to the server, means for identifying items by analyzing the uploaded images, means for saving data on the identified items in a database and updating inventory information, means for calling a recipe suggestion service based on the updated inventory information and suggesting possible recipes, means for displaying an ingredient list for the suggested recipe on a terminal, means for identifying missing ingredients from the ingredient list for the suggested recipe and notifying the user, and means for checking inventory information and missing ingredients using a smart device while on the go. This enables automatic and efficient management of inventory in the refrigerator, allowing users to select optimal recipes and easily check and purchase missing ingredients.

[1080] "Means for taking images inside the refrigerator" refers to the camera installed to capture images inside the refrigerator and the system that controls it in general.

[1081] "Means for uploading captured images to a server" refers to a communication system and software for transmitting captured image data to a server via the Internet or the like.

[1082] "Means for identifying items by analyzing uploaded images" refers to algorithms or machine learning models that use image recognition technology to identify items in the refrigerator from uploaded images.

[1083] "Means for storing data of identified items in a database and updating inventory information" refers to software and hardware for recording recognized item information in a database and keeping inventory information in the refrigerator up to date based on that information.

[1084] "Means for calling a recipe suggestion service based on updated inventory information and suggesting recipes that can be made" refers to a system that obtains appropriate recipe information from an external service based on the latest inventory information in the refrigerator and provides it to the user.

[1085] "Means for displaying a list of ingredients for a proposed recipe on a terminal" refers to an application or interface for displaying recipe information provided by the server on a terminal such as a user's smartphone or tablet.

[1086] "Means for identifying missing ingredients from the ingredient list of a proposed recipe and notifying the user" refers to a system that compares ingredients required for a recipe with current inventory information, detects missing ingredients, and notifies the user.

[1087] "Means for checking inventory information and missing ingredients using smart devices while on the go" refers to a function that allows users to check inventory information and missing ingredients in their refrigerators in real time using a smartphone or tablet while on the go, such as at a supermarket.

[1088] "Means for pre-processing captured images" refers to algorithms or software for performing pre-processing such as resizing, noise removal, and color correction of images before image analysis.

[1089] "Means to order missing ingredients via a delivery service" refers to an interface or API for automatically ordering missing ingredients from a delivery service.

[1090] "Means for updating algorithms to improve the accuracy of recipe suggestions based on purchase history and frequency of use" refers to machine learning models and algorithms that optimize future recipe suggestions by taking into account a user's past purchase history and frequency of use of an item.

[1091] MODE FOR CARRYING OUT THE INVENTION

[1092] The system of the present invention aims to efficiently manage items in a refrigerator, provide optimal recipes to users, and facilitate checking and purchasing of ingredients that are in short supply. The system includes a camera in the refrigerator, a server connected to the Internet, and a user's device (such as a smartphone or tablet).

[1093] Taking pictures inside the refrigerator

[1094] Device: A camera installed inside the refrigerator takes pictures based on user instructions or a regular schedule. When a user presses the "take a picture" button on a smartphone application, the camera activates and captures an image of the inside of the refrigerator. For example, a user may issue a voice command such as "Take a picture of the inside of the refrigerator."

[1095] Uploading an image

[1096] Device: The captured images are automatically uploaded to a server using Wi-Fi or other communication methods. For example, a smartphone application can link with the refrigerator camera and send image data to the server.

[1097] Image analysis and item identification

[1098] Server: The server receives the uploaded images and performs image analysis using machine learning models such as TensorFlow and PyTorch. It identifies the items identified through the analysis and records that information in a database. For example, items such as "carrots," "milk," and "eggs" are identified.

[1099] Inventory updates

[1100] Server: The identified item information is stored in a database and the refrigerator inventory information is updated to the latest state, so that the inventory is always reflected in real time.

[1101] Recipe suggestions

[1102] Server: Based on the latest inventory information, the server calls an external recipe suggestion service API to obtain recipe information that can be made with the current inventory. For example, recipes for "stir-fried vegetables" and "omelette" are obtained from the recipe suggestion service.

[1103] Viewing Recipes

[1104] Device: The recipe information sent from the server is displayed on the user's smartphone or tablet. This information also includes a list of ingredients, which the user can check. Specifically, the smartphone app displays recipes for "stir-fried vegetables" and "omelette" along with their ingredient lists.

[1105] Identifying and notifying shortages

[1106] On the device: Based on the ingredients list of a suggested recipe, identify any missing ingredients and notify the user. For example, if a recipe requires a missing bell pepper, a pop-up notification will appear on the device stating, "You are missing bell peppers."

[1107] Check inventory on the go

[1108] User: While on the go (e.g., at the supermarket), they can use their smartphone to check real-time inventory information and a list of ingredients they are short on. For example, a user opens the app and sees that there are six eggs in the refrigerator.

[1109] Delivery service collaboration (when at home)

[1110] User: If a user is at home and is short on ingredients, the system provides the option to automatically order the missing ingredients using a delivery service API. For example, when a user presses the "Order Delivery" button on their smartphone, the delivery service API is called and green peppers are automatically ordered.

[1111] Optimization based on purchase history and frequency of use

[1112] Server: Once the purchase or delivery is complete, the user is notified and the refrigerator inventory is updated again. The server also updates the algorithm to improve the accuracy of future recipe suggestions based on the user's purchase history and frequency of use. For example, the server can retrain the algorithm based on the purchase data to make better suggestions next time.

[1113] Examples of prompt statements

[1114] "Suggest a recipe that can be made with the items in your refrigerator. Currently, we have carrots, milk, and eggs."

[1115] This invention allows users to efficiently manage their refrigerator inventory, select suitable recipes, and easily check and purchase ingredients they are running low on.

[1116] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1117] Program processing flow

[1118] Step 1:

[1119] Device: A camera installed inside the refrigerator takes pictures based on user instructions or set timing. The input is the user's instructions or schedule, and the output is the image taken inside the refrigerator. For example, when a user presses the "take a picture" button in the app, the camera is activated and an image of the inside of the refrigerator is captured.

[1120] Specific operation: When a user presses the "shoot" button on the smartphone app, the camera starts up and takes a picture of the inside of the refrigerator. At this time, the internal LED light turns on, allowing a clear image to be captured in bright light.

[1121] Step 2:

[1122] Terminal: Acquires captured images and uploads them to a server via Wi-Fi. The input is the captured image data, and the output is the image data uploaded to the server. After the camera captures the image, the smartphone app automatically collects the image data and sends it to the server.

[1123] What it does: After taking a photo, the app uploads the image data to a server via Wi-Fi. This process runs in the background, and users can monitor the progress via a progress bar within the app.

[1124] Step 3:

[1125] Server: The server receives the uploaded images and performs image analysis using a machine learning model (e.g., TensorFlow or PyTorch). The input is the uploaded image data, and the output is a list of identified items. Through analysis, the items in the refrigerator are identified.

[1126] Specific operation: When the server receives the image data, the machine learning model is activated and analyzes the items in the refrigerator. From the analysis results, items such as "carrots," "milk," and "eggs" are identified. The results are recorded in a database.

[1127] Step 4:

[1128] Server: Stores the identified item information in a database and updates the inventory information in the refrigerator. The input is a list of identified items, and the output is updated inventory information. This ensures that the inventory is always updated in real time.

[1129] What happens: The server updates the inventory table in the database, recording new information such as "3 carrots, 1 liter of milk, and 6 eggs."

[1130] Step 5:

[1131] Server: Based on the latest inventory information, it calls an external recipe suggestion service API to obtain recipe information that can be made. The input is the latest inventory information, and the output is the suggested recipe information.

[1132] Specific operation: The server calls the recipe suggestion service API and sends the current inventory information. For example, based on the inventory information for "carrots, milk, and eggs," recipe information for "stir-fried vegetables" and "omelette" is obtained.

[1133] Step 6:

[1134] Device: The recipe information sent from the server is displayed on the user's smartphone or tablet. The input is the suggested recipe information, and the output is the recipe information displayed on the device.

[1135] What it does: The app on the user's smartphone displays recipes for "stir-fried vegetables" and "omelette," along with a list of ingredients. A detailed explanation of the recipe and cooking instructions are also displayed.

[1136] Step 7:

[1137] Terminal: Based on the recipe ingredient list, it identifies missing ingredients and notifies the user. The input is the suggested recipe information and current inventory information, and the output is the list of missing ingredients and a notification.

[1138] What it does: The device checks the ingredients in the suggested recipe against the current inventory and displays a notification to the user saying, "You're low on bell peppers." The list of ingredients you're short on is available in the app.

[1139] Step 8:

[1140] User: Check real-time inventory information and material shortage lists on the go using a smartphone or tablet. The input is an access request from the smartphone, and the output is real-time inventory information.

[1141] Specific behavior: When a user opens the app at a supermarket and taps the stock check button, they can see that there are 6 eggs left.

[1142] Step 9:

[1143] User: When there are missing materials while at home, the system automatically orders the missing materials using the delivery service API. The input is a list of missing materials, and the output is a notification that the delivery order has been completed.

[1144] Specific operation: When a user presses the "Order Delivery" button in the app, the delivery service API is called and "green peppers" are automatically ordered. After the order is completed, a notification is displayed on the smartphone.

[1145] Step 10:

[1146] Server: Notifies the user that the purchase or delivery is complete and updates the refrigerator inventory information again. It also updates the algorithm to improve the accuracy of recipe suggestions based on purchase history and frequency of use. The input is the purchase completion notification, and the output is the updated inventory information and the algorithm.

[1147] Specific operation: When the delivery is complete, a notification saying "Peppers delivery completed" is displayed on the user's device. The server updates the inventory information in the database and retrains the algorithm based on the purchase data to improve the accuracy of the next recommendation.

[1148] (Application example 1)

[1149] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1150] In modern life, managing ingredients in the refrigerator is complicated, and the hassle of checking for missing ingredients and the stress of selecting the optimal recipe are issues. Many users also find it troublesome to check inventory while on the go and to purchase or order missing ingredients. Therefore, there is a need for a system that can efficiently manage refrigerator inventory information, suggest optimal recipes, and automatically order missing ingredients.

[1151] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1152] In this invention, the server includes a means for taking images of the inside of the refrigerator, a means for uploading the taken images to the server, a means for analyzing the uploaded images to identify items, and a means for automatically ordering missing ingredients from a food delivery service. This makes it possible to efficiently manage inventory information in the refrigerator, propose optimal recipes, and automatically order missing ingredients.

[1153] The "means for taking images inside the refrigerator" is a device that uses a camera installed inside the refrigerator to take images of items inside the refrigerator periodically or based on a user's instruction.

[1154] The "means for uploading captured images to a server" is a communication function for transmitting captured image data to a server via the Internet.

[1155] The "means of identifying items by analyzing uploaded images" refers to a technology that uses machine learning algorithms and image analysis technology on a server to recognize and identify each item in a refrigerator from uploaded images.

[1156] The "means for storing data on identified items in a database and updating inventory information" is a system for storing analyzed item information in a database and keeping the current inventory information in the refrigerator up to date.

[1157] "Means of calling the API of a recipe suggestion service based on updated inventory information to suggest recipes that can be made" refers to a method of using the API of an external recipe suggestion service based on the latest inventory information to obtain recipes for dishes that can be made based on current inventory.

[1158] The "means for displaying a list of ingredients for a proposed recipe on a terminal" refers to a means for displaying the acquired recipe information and list of necessary ingredients on a terminal such as a user's smartphone or tablet.

[1159] The "means for automatically ordering missing ingredients from a food delivery service" is a technology that determines which ingredients are missing to create a recipe and automatically orders the missing ingredients from a food delivery service via the Internet.

[1160] "Means for pre-processing uploaded images" refers to methods for pre-processing captured images, such as filtering and resizing, in order to improve image quality or extract necessary information before image analysis.

[1161] The "means for notifying the user of missing ingredients" is a technology that sends a notification message to a smartphone, tablet, etc. to inform the user of any missing ingredients needed for a recipe proposal.

[1162] An embodiment of the present invention will now be described. The system of the present invention efficiently manages items in a refrigerator, proposes optimal recipes, and automatically orders ingredients that are in short supply. This system includes the following components: a camera in the refrigerator, a server connected to the Internet, and a user terminal (such as a smartphone or tablet).

[1163] First, the basic operation steps of the system will be explained.

[1164] Basic operations

[1165] 1. Taking pictures inside the refrigerator

[1166] Terminal: A camera installed inside the refrigerator takes pictures of the inside periodically or based on the user's instructions.

[1167] 2. Upload an image

[1168] Terminal: Uploads captured images to a server via Wi-Fi or other communication means.

[1169] 3. Image analysis and item identification

[1170] Server: Analyzes the uploaded images and identifies the items in the refrigerator. This analysis uses machine learning models such as TensorFlow and PyTorch.

[1171] 4. Updating inventory information

[1172] Server: Stores the information of identified items in a database and keeps the inventory information in the refrigerator up to date.

[1173] 5. Recipe suggestions

[1174] Server: Based on the updated inventory information, it calls an external recipe database (e.g., a recipe suggestion service API), matches the available ingredients, and retrieves recipes that can be made.

[1175] 6. Viewing Recipes

[1176] Device: The recipe information sent from the server is displayed on the user's smartphone or tablet. The displayed recipe also includes a list of necessary ingredients.

[1177] 7. Identification and notification of shortages

[1178] Terminal: Identify missing ingredients from the suggested recipe's ingredient list and notify the user. The list of missing ingredients is displayed on the terminal so the user can review them.

[1179] 8. Automatic ordering of missing materials

[1180] Server: Automatically orders missing ingredients from a food delivery service, using the food delivery service's API.

[1181] Hardware and software used

[1182] 1. Refrigerator camera

[1183] Example: Camera using the Raspberry Pi camera module

[1184] 2. Internet connection environment

[1185] Example: Wi-Fi router

[1186] 3. User's Device

[1187] Example: Smartphones and tablets

[1188] 4. Server

[1189] Programming: Python

[1190] Image processing: OpenCV

[1191] Item identification: Machine learning libraries such as TensorFlow and PyTorch

[1192] Communication:requests library

[1193] Specific examples

[1194] A user uses the "SmartKitchen" app while there is chicken, onions, and carrots in the refrigerator. The camera inside the refrigerator automatically takes a picture, and the app suggests recipes for "chicken teriyaki" and "curry." Because the user is low on onions, the app automatically places an order with the nearest food delivery service and notifies the user.

[1195] Prompt Sentence Examples

[1196] Prompt for an application that determines what ingredients are in the refrigerator, suggests recipes to the user, and automatically orders any missing ingredients:

[1197] Identify ingredients from images of your refrigerator and suggest recipes based on your current inventory, including automatically ordering missing ingredients through a food delivery service.

[1198] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1199] Step 1:

[1200] Taking pictures inside the refrigerator

[1201] The device takes pictures using a camera installed inside the refrigerator. In this process, the camera module captures images of the inside of the refrigerator periodically or based on the user's instructions. The input is the current state of the inside of the refrigerator, and the output is a captured image file.

[1202] Step 2:

[1203] Uploading an image

[1204] The device uploads the captured image file to the server. In this process, the image data is sent to the server via an internet connection. The input is the captured image file, and the output is the image data stored on the server.

[1205] Step 3:

[1206] Image analysis and item identification

[1207] The server analyzes the uploaded image data and identifies the items in the refrigerator. This process uses machine learning models (e.g., TensorFlow, PyTorch) to analyze the image data and identify specific items. The input is the uploaded image data, and the output is a list of identified items.

[1208] Step 4:

[1209] Inventory updates

[1210] The server saves the identified item list in a database and updates the inventory information. This process writes the identified item data to the database and updates the refrigerator inventory information. The input is the identified item list, and the output is the updated inventory information.

[1211] Step 5:

[1212] Recipe suggestions

[1213] Based on the updated inventory information, the server calls the API of the recipe suggestion service to suggest recipes that can be made. This process calls an external recipe database API to obtain appropriate recipes based on the inventory information in the refrigerator. The input is the updated inventory information, and the output is a list of suggested recipes.

[1214] Step 6:

[1215] Viewing Recipes

[1216] The device displays the recipe information sent from the server on the user's smartphone or tablet. In this process, the suggested recipe and the list of required ingredients are displayed on the device. The input is the suggested recipe list, and the output is the recipe information displayed on the user's device.

[1217] Step 7:

[1218] Identifying and notifying shortages

[1219] The terminal identifies missing ingredients from the ingredient list of the proposed recipe and notifies the user. This process compares the missing ingredients with current inventory information and sends a notification message to the user. The input is the ingredient list of the proposed recipe and current inventory information, and the output is a missing ingredient list and a notification message.

[1220] Step 8:

[1221] Auto-ordering of missing materials

[1222] The server automatically orders the missing ingredients from a food delivery service. This process uses the list of missing ingredients to call the food delivery service's API and confirm the order. The input is the list of missing ingredients, and the output is order confirmation information.

[1223] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1224] MODE FOR CARRYING OUT THE INVENTION

[1225] Configuration overview

[1226] This system manages the items in the refrigerator, suggests optimal recipes, notifies users of missing ingredients, and also incorporates a user emotion recognition engine. The system includes a camera in the refrigerator, an internet-connected server, a user device (such as a smartphone or tablet), and an emotion recognition engine.

[1227] Basic operations

[1228] 1. Taking pictures inside the refrigerator

[1229] Device: A camera installed inside the refrigerator takes pictures of the interior periodically or upon user instruction.

[1230] 2. Upload an image

[1231] Terminal: Uploads captured images to a server via Wi-Fi or other communication means.

[1232] 3. Image analysis and item identification

[1233] Server: Preprocesses the received images, adjusting brightness and contrast, and removing noise to improve the accuracy of image recognition.

[1234] Server: The preprocessed images are input into a machine learning model (e.g., TensorFlow, PyTorch) and analyzed to identify the items in the refrigerator.

[1235] 4. Updating inventory information

[1236] Server: Stores information about items identified through image analysis in a database and updates the current inventory information in the refrigerator.

[1237] 5. Recipe suggestions

[1238] Server: Based on the updated inventory information, the API of the recipe suggestion service is called, and the available ingredients are matched to obtain recipes that can be made.

[1239] 6. Emotion recognition

[1240] Device: The camera and microphone on the user's device are used to collect facial and voice data.

[1241] Emotion engine: Analyzes collected data and determines the user's emotional state (e.g., joy, sadness, stress, etc.).

[1242] 7. Optimizing recipe suggestions

[1243] Server: Based on the emotional information determined by the emotion engine, the server suggests recipes that suit the user's current emotional state.

[1244] 8. Viewing Recipes

[1245] Device: The recipe information sent from the server is displayed on the user's smartphone or tablet. The displayed recipe also includes a list of required ingredients.

[1246] 9. Identification and Notification of Shortage Materials

[1247] Terminal: Identify missing ingredients from the suggested recipe's ingredient list and notify the user of this information.

[1248] 10. Check inventory on the go

[1249] Users: Access the system from their smartphones or tablets while on the go, such as at the supermarket, to check real-time inventory information and a list of materials that are in short supply.

[1250] 11. Delivery service collaboration (when at home)

[1251] User: If there are missing materials while at home, select the delivery service integration option.

[1252] Server: Calls the delivery service's API to automatically order missing ingredients.

[1253] 12. Notifications and Optimization

[1254] Server: Notifies the user that the purchase or delivery is complete and tracks the delivery progress.

[1255] Terminal: Once the delivery is complete, the user is notified again and the refrigerator inventory information is updated.

[1256] Server: Updates the algorithm to improve the accuracy of future suggestions based on the user's purchase history and frequency of use.

[1257] By incorporating a user emotion recognition engine, recipes can be suggested based on the user's current emotions, providing a more personalized experience.

[1258] Specific examples

[1259] User: There are carrots, milk, and eggs in the refrigerator. The user comes home from work and is stressed.

[1260] Server: These items are identified through image analysis and stored in a database.

[1261] Emotion engine: Determines the user's stress level from facial expressions and voice data, and prioritizes recipes with a relaxing effect.

[1262] Server: Recipes that contribute to relaxation, such as "carbonara," are obtained from a recipe suggestion service.

[1263] Device: The smartphone will display a recipe for "Carbonara" along with a list of ingredients needed.

[1264] User: Checks the supermarket for a missing ingredient (e.g., bacon) and adds it to a shopping list.

[1265] In this way, this system allows users to efficiently manage their refrigerator inventory and also suggests optimal recipes that suit their emotional state at the time, resulting in a more satisfying cooking experience.

[1266] The processing flow will be explained below.

[1267] Step 1:

[1268] Device: A camera installed inside the refrigerator takes pictures of the inside of the refrigerator at regular intervals or as set by the user.

[1269] Step 2:

[1270] Terminal: Uploads captured images to a server via Wi-Fi or other communication means.

[1271] Step 3:

[1272] Server: Preprocesses the received images, adjusting brightness and contrast, and removing noise to improve the accuracy of image recognition.

[1273] Step 4:

[1274] Server: The preprocessed images are input into a machine learning model (e.g., TensorFlow, PyTorch) and analyzed to identify the items in the refrigerator.

[1275] Step 5:

[1276] Server: Stores information about items identified through image analysis in a database and updates the current inventory information in the refrigerator.

[1277] Step 6:

[1278] Server: Based on the updated inventory information, the API of the recipe suggestion service is called, and the available ingredients are matched to obtain recipes that can be made.

[1279] Step 7:

[1280] Device: The camera and microphone on the user's device are used to collect facial and voice data.

[1281] Step 8:

[1282] Emotion engine: Analyzes collected data and determines the user's emotional state (e.g., joy, sadness, stress, etc.).

[1283] Step 9:

[1284] Server: Based on the emotional information determined by the emotion engine, the server optimizes and suggests recipes that suit the user's current emotional state.

[1285] Step 10:

[1286] Device: The recipe information sent from the server is displayed on the user's smartphone or tablet. The displayed recipe also includes a list of required ingredients.

[1287] Step 11:

[1288] On the device: Based on the ingredients list of the suggested recipe, identify any missing ingredients and notify the user.

[1289] Step 12:

[1290] Users: Access the system from their smartphones or tablets while on the go, such as at the supermarket, to check real-time inventory information and a list of materials that are in short supply.

[1291] Step 13:

[1292] User: If there are missing materials while at home, select the delivery service integration option.

[1293] Step 14:

[1294] Server: Calls the delivery service's API to automatically order missing ingredients.

[1295] Step 15:

[1296] Server: Notifies the user once the order is complete and tracks the delivery progress.

[1297] Step 16:

[1298] Terminal: Once the delivery is complete, the user is notified again and the refrigerator inventory information is updated.

[1299] Step 17:

[1300] Server: Updates the algorithm to improve the accuracy of future suggestions based on the user's purchase history and frequency of use.

[1301] Example 2

[1302] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1303] Conventional refrigerator management systems are limited in their ability to manage refrigerator inventory and suggest recipes, making it difficult to provide personalized services that reflect the user's emotional state and circumstances. Furthermore, there are few ways to check inventory information while on the go, making it difficult to take appropriate action when necessary ingredients are unavailable.

[1304] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for capturing images of the inside of the refrigerator, a means for uploading the captured images to the server, a means for preprocessing and analyzing the uploaded images to identify items, a means for saving data on the identified items in a database and updating inventory information, a means for suggesting possible recipes by calling an API of a recipe suggestion service based on the updated inventory information, a means for recognizing the user's emotional state and selecting an optimal recipe, and a means for displaying an ingredient list for the suggested recipe on a terminal. This allows the user to efficiently manage inventory in the refrigerator and receive personalized recipe suggestions based on their emotional state. Furthermore, by checking inventory information and receiving notifications of missing ingredients while on the go, the user can appropriately procure the necessary ingredients.

[1305] The "means for taking images inside the refrigerator" is a mechanism for taking images inside the refrigerator using a camera installed inside the refrigerator.

[1306] The "means for uploading captured images to a server" is a mechanism for transmitting captured image data to a server via a communication means.

[1307] The "means for preprocessing and analyzing uploaded images to identify items" refers to a mechanism that performs preprocessing such as adjusting the brightness and contrast of received image data and removing noise, and then identifies items using a machine learning model.

[1308] The "means for saving data on identified items in a database and updating inventory information" is a mechanism for saving information on items identified by analysis in a database and updating the latest inventory status.

[1309] "Means for calling the API of a recipe suggestion service based on updated inventory information and suggesting recipes that can be made" is a mechanism for calling the API of an external recipe suggestion service based on the latest inventory information and obtaining recipe information that can be made.

[1310] The "means for recognizing the user's emotional state and selecting the optimal recipe" is a mechanism for analyzing the user's facial expressions and voice data to determine their emotional state and select a recipe that is appropriate for that state.

[1311] The "means for displaying the ingredient list of the proposed recipe on the terminal" is a mechanism for displaying the recipe information and ingredient list sent from the server on the user's smartphone or tablet.

[1312] The present invention relates to a system that manages items in a refrigerator, suggests optimal recipes based on the user's emotional state, and notifies the user of any missing ingredients. The system includes a camera in the refrigerator, a server connected to the Internet, a user device (such as a smartphone or tablet), and an emotion engine.

[1313] System Configuration

[1314] 1. Camera in the refrigerator

[1315] Hardware: Camera device

[1316] Software: Image acquisition program

[1317] Function: Takes pictures of the inside of the refrigerator periodically or at the user's command.

[1318] 2. Server

[1319] Hardware: Server Device

[1320] Software: Image preprocessing program, machine learning model (e.g., TensorFlow, PyTorch), recipe suggestion service API, database management system

[1321] Function: Preprocesses received images to identify and specify items. Stores information about identified items in a database and updates inventory information in the refrigerator. Calls the API of a recipe suggestion service based on the updated inventory information and obtains information about recipes that can be created.

[1322] 3. User Device

[1323] Hardware: Smartphones and tablet devices

[1324] Software: Display program, notification program, emotion recognition function

[1325] Function: Displays recipe information sent from the server to the user and notifies them of missing ingredients. Collects the user's facial expressions and voice data and analyzes them with the emotion engine.

[1326] 4. Emotion Engine

[1327] Software: Emotional state analysis algorithm

[1328] Function: Analyzes collected user facial and voice data to determine the user's emotional state. Based on the determined emotional information, selects a recipe that is appropriate for the user's current emotional state.

[1329] Specific examples

[1330] Example 1:

[1331] User: There are carrots, milk, and eggs in the refrigerator. The user comes home from work and is stressed.

[1332] Server: These items are identified through image analysis and stored in a database.

[1333] Emotion engine: Determines the user's stress level from facial expressions and voice data, and prioritizes recipes with a relaxing effect.

[1334] Server: Recipes that contribute to relaxation, such as "carbonara," are obtained from a recipe suggestion service.

[1335] Device: The smartphone will display a recipe for "Carbonara" along with a list of ingredients needed.

[1336] User: Checks the supermarket for a missing ingredient (e.g., bacon) and adds it to a shopping list.

[1337] Example 2 (An example of a prompt using a generative AI model):

[1338] Prompt: "The ingredients in my fridge are carrots, potatoes, cheese, and eggs. I'm feeling relaxed today, so can you recommend a recipe that will fit that mood?"

[1339] Server: Performs image analysis and emotion recognition to suggest relaxing recipes such as "cheese omelet."

[1340] This allows the system to provide a more satisfying cooking experience by allowing users to efficiently manage inventory in the refrigerator, suggesting optimal recipes based on their emotional state at the time, and notifying them of any missing ingredients.

[1341] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1342] Step 1:

[1343] Device: A camera installed inside the refrigerator takes images of the inside of the refrigerator periodically or based on the user's instructions.

[1344] Input: User instruction or timer event.

[1345] Data processing: Acquisition of image data.

[1346] Output: A captured image of the inside of a refrigerator.

[1347] Step 2:

[1348] Terminal: Uploads captured images to a server via Wi-Fi or other communication means.

[1349] Input: Captured image data.

[1350] Data processing: Transfer of image data.

[1351] Output: Image data sent to the server.

[1352] Step 3:

[1353] Server: Preprocesses the received images, adjusting brightness and contrast and removing noise.

[1354] Input: Image data sent to the server.

[1355] Data processing: Image preprocessing (brightness, contrast adjustment, noise removal).

[1356] Output: Preprocessed image data.

[1357] Step 4:

[1358] Server: The preprocessed image data is input into a machine learning model (e.g., TensorFlow, PyTorch) and analyzed to identify the items in the refrigerator.

[1359] Input: Preprocessed image data.

[1360] Data processing: Identification and labeling of items.

[1361] Output: A list of identified items.

[1362] Step 5:

[1363] Server: Stores information about items identified through analysis in a database and updates the inventory information in the refrigerator.

[1364] Input: A list of identified items.

[1365] Data processing: storing information in the database and updating inventory information.

[1366] Output: Updated inventory information.

[1367] Step 6:

[1368] Server: Based on the updated inventory information, the API of the recipe suggestion service is called, and the available ingredients are matched to obtain recipes that can be made.

[1369] Input: Updated inventory information.

[1370] Data processing: Calling the recipe suggestion service API and obtaining recipes.

[1371] Output: A list of suggested recipes.

[1372] Step 7:

[1373] Terminal: Collects the user's facial expressions and voice data and sends them to the emotion engine to recognize the user's emotional state.

[1374] Input: User's facial and voice data.

[1375] Data processing: Collecting and transmitting emotion data.

[1376] Output: Data sent to the emotion engine.

[1377] Step 8:

[1378] Emotion engine: Analyzes collected data and determines the user's emotional state.

[1379] Input: User's facial and voice data.

[1380] Data processing: Analysis of emotional state (e.g., stress, joy).

[1381] Output: Determined emotional state.

[1382] Step 9:

[1383] Server: Based on the emotional information determined by the emotion engine, selects a recipe that is appropriate for the user's current emotional state.

[1384] Input: A list of identified emotional states and suggested recipes.

[1385] Data processing: Recipe selection based on emotional state.

[1386] Output: The best recipe for the user.

[1387] Step 10:

[1388] Terminal: The optimal recipe information sent from the server is displayed on the terminal, and the user is notified of the list of necessary ingredients.

[1389] Input: The selected optimal recipe.

[1390] Data processing: Display recipe information.

[1391] Output: Recipe and ingredients list displayed on the user's device.

[1392] Step 11:

[1393] On the device: Based on the ingredients list of a suggested recipe, identify any missing ingredients and notify the user.

[1394] Input: Materials list and current inventory information.

[1395] Data processing: Identifying missing materials.

[1396] Output: Notify user of missing material.

[1397] Step 12:

[1398] User: Accesses the system from a terminal while on the go, such as at a supermarket, to check refrigerator inventory information and a list of ingredients that are in short supply.

[1399] Input: The user's access request.

[1400] Data processing: Providing real-time inventory information.

[1401] Output: Inventory information and a list of missing materials displayed on the user's terminal.

[1402] Step 13:

[1403] User: When at home, select the delivery service integration option to automatically order missing ingredients.

[1404] Input: User's delivery service selection.

[1405] Data processing: Sending orders to delivery services.

[1406] Output: Delivery order confirmation.

[1407] Step 14:

[1408] Server: Notifies the user that the purchase or delivery is complete and tracks the delivery progress.

[1409] Input: Delivery completion information.

[1410] Data Processing: Delivery progress tracking and notification.

[1411] Output: Delivery completion notification to the user.

[1412] Step 15:

[1413] Terminal: Once the delivery is complete, the user is notified again and the refrigerator inventory is updated.

[1414] Input: Delivery completion information for snooze.

[1415] Data processing: Updating refrigerator inventory information.

[1416] Output: Notification of updated inventory information.

[1417] Step 16:

[1418] Server: Updates the algorithm to improve the accuracy of future suggestions based on the user's purchase history and frequency of use.

[1419] Input: Purchase history and usage frequency data.

[1420] Data processing: Algorithm updates and optimizations.

[1421] Output: Improved accuracy of suggestions for the next time.

[1422] (Application example 2)

[1423] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1424] The present invention aims to improve user satisfaction and convenience by providing a refrigerator inventory management and recipe suggestion system that not only efficiently manages items in a refrigerator, but also suggests optimal cooking instructions based on the user's emotional state and provides a series of processes including automatic ordering of missing ingredients. However, conventional refrigerator management systems have problems in that they do not take into account responses based on the user's emotions and do not have a function for automatically ordering missing ingredients.

[1425] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1426] In this invention, the server includes means for taking images of the inside of the refrigerator, means for uploading the taken images to the server, means for analyzing the uploaded images to identify items, means for saving data on the identified items in a database and updating inventory information, means for calling an API of a recipe suggestion service based on the updated inventory information to provide creatable cooking instructions, means for collecting facial expression and voice data using a camera and microphone of the terminal to acquire the user's emotional state, means for analyzing the collected data to determine the user's emotion, means for proposing cooking instructions according to the determined emotion, means for displaying an ingredient list for the proposed cooking instructions on the terminal, and means for identifying missing items from the suggested ingredient list and automatically ordering them. This allows the user to not only manage inventory in the refrigerator, but also to suggest cooking instructions according to the user's emotional state and automatically order missing ingredients.

[1427] The "means for taking images inside the refrigerator" is a device that uses a camera installed inside the refrigerator to take images of items inside the refrigerator periodically or in response to instructions.

[1428] The "means for uploading captured images to a server" is a communication device that transfers captured images to a server via the Internet.

[1429] The "means for analyzing uploaded images to identify items" is a device that analyzes the images received on the server and uses image recognition algorithms to identify the items in the refrigerator.

[1430] The "means for saving data on identified items in a database and updating inventory information" is a device that saves information on items identified through analysis in a database and updates the inventory status in the refrigerator to the latest status.

[1431] "Means for providing cooking instructions that can be created by calling the API of a recipe suggestion service based on updated inventory information" refers to a device that uses the latest inventory information to access the API of a recipe suggestion service and obtains cooking instructions based on that information.

[1432] "Means for collecting facial expression and voice data using a camera or microphone on a terminal to obtain the emotional state of a user" refers to a device that collects facial expression and voice data using a camera or microphone installed on a terminal in order to grasp the emotional state of a user.

[1433] The "means for analyzing collected data to determine the user's emotions" is a device that analyzes collected facial and voice data and uses emotion recognition algorithms to identify the user's emotional state.

[1434] The "means for proposing cooking instructions according to the determined emotion" is a device that selects and provides optimal cooking instructions based on the user's emotional state.

[1435] The "means for displaying an ingredient list for the proposed cooking instructions on a terminal" is a device that displays an ingredient list based on the proposed cooking instructions on a user's terminal.

[1436] The "means for identifying missing items in the proposed ingredient list and automatically ordering them" refers to a device that verifies the ingredient list based on the proposed cooking instructions, identifies missing items, and automatically orders them.

[1437] System Configuration

[1438] The present invention relates to a "refrigerator item management and recipe suggestion system based on emotional state" and uses the following hardware and software.

[1439] Hardware used:

[1440] Refrigerator camera (connectable Wi-Fi camera)

[1441] User devices (smartphones, tablets)

[1442] server

[1443] Software used:

[1444] Emotion recognition engine (e.g. Affectiva, Microsoft Emotion API)

[1445] Image analysis engine (e.g. TensorFlow, PyTorch)

[1446] Cloud databases (e.g. AWS DynamoDB, Google Cloud Firestore)

[1447] Delivery service APIs (e.g. UberEats, DoorDash)

[1448] Basic operations

[1449] Taking and uploading images of the inside of the refrigerator

[1450] The device sends instructions to a camera installed inside the refrigerator, taking pictures of the inside of the refrigerator periodically or based on the user's instructions. The captured image data is then uploaded to a server via Wi-Fi.

[1451] Image analysis and item identification

[1452] The server preprocesses the uploaded images and uses image recognition algorithms (such as TensorFlow or PyTorch) to identify the items in the refrigerator. The identified item data is stored in a cloud database and inventory information is updated.

[1453] Acquiring and analyzing emotional states

[1454] The device collects facial expression and voice data of the user to obtain the user's emotional state, and the collected data is analyzed using an emotion recognition engine to determine the user's emotional state.

[1455] Recipe suggestions and ingredient lists

[1456] The server then calls the API of the recipe suggestion service to obtain optimal cooking instructions based on the updated inventory information and emotional state. The obtained cooking instructions and ingredient list are then displayed on the user's device.

[1457] Automatic ordering of missing materials

[1458] The terminal identifies any missing items from the proposed materials list and automatically orders them through the delivery service API, with the order status tracked and notified to the user in real time.

[1459] Specific examples

[1460] After the user returns home from work, an image of the contents of the refrigerator is automatically uploaded to the app. The app then acquires the user's facial expression data and determines their emotional state as "tired." Based on this information, the server analyzes the inventory and determines that carbonara is ready to be made, but that there is a shortage of bacon. When the user selects the delivery option, the bacon is automatically ordered, and the progress is displayed in the app.

[1461] Prompt Sentence Examples

[1462] After the user returns home, an image of the inside of the refrigerator and the user's facial expression data are collected from the smartphone. Image analysis is used to check the inventory in the refrigerator, and if fatigue is determined from the facial expression data, what recipes should be suggested and how to order missing ingredients via a delivery service? Please explain this with specific steps.

[1463] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1464] Step 1:

[1465] The user issues a command to take a picture of the refrigerator from the terminal. The input is the user's command, and the output is image data of the inside of the refrigerator. Specifically, the terminal sends a command to take a picture to the camera installed inside the refrigerator, and the camera takes a picture of the current inside of the refrigerator.

[1466] Step 2:

[1467] The captured image data is uploaded from the device to the server. The input is the image data of the inside of the refrigerator, and the output is an image file saved on the server. Specifically, the device transfers the image data to the server via Wi-Fi.

[1468] Step 3:

[1469] The server preprocesses the uploaded image data and performs image analysis. The input is the image file stored on the server, and the output is the data of the identified items in the refrigerator. Specifically, the server adjusts the brightness and contrast of the image and uses a machine learning model to identify the items.

[1470] Step 4:

[1471] The identified item data is stored in a database, and inventory information is updated. The input is the identified item data in the refrigerator, and the output is the updated inventory information. Specifically, the server connects to the database, stores the latest item information, and updates the inventory information.

[1472] Step 5:

[1473] The user's device collects facial and voice data to acquire the user's emotional state. The input is the user's real-time facial and voice data, and the output is the collected data. Specifically, the device collects the data using a camera and microphone.

[1474] Step 6:

[1475] The collected data is sent to a server and analyzed by an emotion recognition engine. The input is the collected facial and voice data, and the output is the user's emotional state. Specifically, the server uses an emotion recognition algorithm to determine the user's emotions.

[1476] Step 7:

[1477] Based on the updated inventory information and emotional state, the server calls the API of the recipe suggestion service to obtain optimal cooking instructions. The input is the updated inventory information and the user's emotional state, and the output is cooking instructions. Specifically, the server calls the API of the recipe suggestion service to obtain cooking instructions.

[1478] Step 8:

[1479] The acquired cooking instructions and ingredient list are displayed on the user's terminal. The input is the cooking instructions and ingredient list, and the output is the information displayed on the terminal. Specifically, the server sends this information to the terminal, and the terminal displays it.

[1480] Step 9:

[1481] The user's device identifies missing items from the proposed materials list and automatically places an order. The input is the materials list and inventory information, and the output is order information for the delivery service. Specifically, the device checks the inventory information to identify missing items and calls the delivery service's API to place the order.

[1482] Step 10:

[1483] The user is notified of the order progress and the inventory information is updated after delivery is completed. The input is the progress of the delivery service, and the output is the updated inventory information and a notification to the user. Specifically, the server receives information from the delivery service, updates the inventory information, and notifies the user.

[1484] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1485] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1486] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1487] [Fourth embodiment]

[1488] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1489] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1490] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1491] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1492] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1493] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1494] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1495] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1496] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1497] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1498] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1499] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1500] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1501] MODE FOR CARRYING OUT THE INVENTION

[1502] Configuration overview

[1503] This system manages the items in the refrigerator, suggests optimal recipes, and notifies users of any missing ingredients. The system includes a camera inside the refrigerator, a server connected to the Internet, and a user's device (such as a smartphone or tablet).

[1504] Basic operations

[1505] 1. Taking pictures inside the refrigerator

[1506] Device: A camera installed inside the refrigerator takes pictures of the interior periodically or upon user instruction.

[1507] 2. Upload an image

[1508] Terminal: Uploads captured images to a server via Wi-Fi or other communication means.

[1509] 3. Image analysis and item identification

[1510] Server: Analyzes the uploaded images and identifies the items in the refrigerator. This analysis uses machine learning models (e.g., TensorFlow, PyTorch).

[1511] 4. Updating inventory information

[1512] Server: Stores the information of identified items in a database and keeps the inventory information in the refrigerator up to date.

[1513] 5. Recipe suggestions

[1514] Server: Based on the updated inventory information, it calls an external recipe database (e.g., a recipe suggestion service API), matches the available ingredients, and retrieves recipes that can be made.

[1515] 6. Viewing Recipes

[1516] Device: The recipe information sent from the server is displayed on the user's smartphone or tablet. The displayed recipe also includes a list of required ingredients.

[1517] 7. Identification and notification of shortages

[1518] Terminal: Identify missing ingredients from the suggested recipe's ingredient list and notify the user. The list of missing ingredients is displayed on the terminal, allowing the user to review them.

[1519] 8. Check inventory on the go

[1520] Users can access the system from their smartphones or tablets while on the go, such as at the supermarket, and check real-time inventory information and a list of materials that are in short supply.

[1521] 9. Delivery service collaboration (when at home)

[1522] User: If there are missing materials while at home, the system will provide the option to automatically order the missing materials using a delivery service (e.g., a delivery service API).

[1523] 10. Notifications and Optimization

[1524] Server: Notifies the user that the purchase or delivery is complete, updates the refrigerator inventory again, and updates the algorithm to improve the accuracy of future recipe suggestions based on the user's purchase history and frequency of use.

[1525] Specific examples

[1526] User: There are carrots, milk, and eggs in the refrigerator.

[1527] Server: These items are identified through image analysis and stored in a database.

[1528] Server: Recipes such as "stir-fried vegetables" and "omelette" are obtained from the recipe suggestion service.

[1529] Device: The smartphone displays recipes for "Stir-fried vegetables" and "Omelette," along with a list of ingredients needed.

[1530] User: Checks the supermarket for a missing ingredient (e.g., green peppers) and adds it to a shopping list.

[1531] This completes the embodiment of the present invention. This system allows users to efficiently manage inventory in their refrigerators, select optimal recipes, and easily check, purchase, and order ingredients that are in short supply.

[1532] The processing flow will be explained below.

[1533] Step 1:

[1534] Device: A camera installed inside the refrigerator takes pictures of the inside of the refrigerator at regular intervals or as set by the user.

[1535] Step 2:

[1536] Terminal: Uploads captured images to a server via Wi-Fi or other communication means.

[1537] Step 3:

[1538] Server: Preprocesses the received images, adjusting brightness and contrast, and removing noise to improve the accuracy of image recognition.

[1539] Step 4:

[1540] Server: The preprocessed images are input into a machine learning model (e.g., TensorFlow, PyTorch) and analyzed to identify the items in the refrigerator.

[1541] Step 5:

[1542] Server: Stores information about items identified through image analysis in a database and updates the current inventory information in the refrigerator.

[1543] Step 6:

[1544] Server: Based on the updated inventory information, the API of the recipe suggestion service is called, and the available ingredients are matched to obtain recipes that can be made.

[1545] Step 7:

[1546] Server: Selects the optimal recipe from the retrieved recipes and sends that information to the user's device.

[1547] Step 8:

[1548] Device: Recipe information and a list of ingredients are displayed on the user's smartphone or tablet.

[1549] Step 9:

[1550] On the device: Based on the ingredients list of the suggested recipe, identify any missing ingredients and notify the user.

[1551] Step 10:

[1552] Users: Access the system from the supermarket or other locations to view real-time inventory information and shortage lists.

[1553] Step 11:

[1554] User: If there are missing materials while at home, select the delivery service integration option.

[1555] Step 12:

[1556] Server: Calls the delivery service's API to automatically order missing ingredients.

[1557] Step 13:

[1558] Server: Notifies the user once the order is complete and tracks the delivery progress.

[1559] Step 14:

[1560] Terminal: Once the delivery is complete, the user is notified again and the refrigerator inventory information is updated.

[1561] Step 15:

[1562] Server: Updates the algorithm to improve the accuracy of future suggestions based on the user's purchase history and frequency of use.

[1563] Through the above processing steps, users can seamlessly manage inventory in the refrigerator, select recipes, and even purchase missing ingredients or place automatic orders.

[1564] Example 1

[1565] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1566] Conventional refrigerator management systems require users to manually manage refrigerator inventory information, which can result in users losing track of what's in the refrigerator, leading to food waste and unnecessary purchases. Another issue is that checking and purchasing ingredients when they're running low is complicated and time-consuming, making efficient food management difficult.

[1567] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1568] In this invention, the server includes means for taking images of the inside of the refrigerator, means for uploading the taken images to the server, means for identifying items by analyzing the uploaded images, means for saving data on the identified items in a database and updating inventory information, means for calling a recipe suggestion service based on the updated inventory information and suggesting possible recipes, means for displaying an ingredient list for the suggested recipe on a terminal, means for identifying missing ingredients from the ingredient list for the suggested recipe and notifying the user, and means for checking inventory information and missing ingredients using a smart device while on the go. This enables automatic and efficient management of inventory in the refrigerator, allowing users to select optimal recipes and easily check and purchase missing ingredients.

[1569] "Means for taking images inside the refrigerator" refers to the camera installed to capture images inside the refrigerator and the system that controls it in general.

[1570] "Means for uploading captured images to a server" refers to a communication system and software for transmitting captured image data to a server via the Internet or the like.

[1571] "Means for identifying items by analyzing uploaded images" refers to algorithms or machine learning models that use image recognition technology to identify items in the refrigerator from uploaded images.

[1572] "Means for storing data of identified items in a database and updating inventory information" refers to software and hardware for recording recognized item information in a database and keeping inventory information in the refrigerator up to date based on that information.

[1573] "Means for calling a recipe suggestion service based on updated inventory information and suggesting recipes that can be made" refers to a system that obtains appropriate recipe information from an external service based on the latest inventory information in the refrigerator and provides it to the user.

[1574] "Means for displaying a list of ingredients for a proposed recipe on a terminal" refers to an application or interface for displaying recipe information provided by the server on a terminal such as a user's smartphone or tablet.

[1575] "Means for identifying missing ingredients from the ingredient list of a proposed recipe and notifying the user" refers to a system that compares ingredients required for a recipe with current inventory information, detects missing ingredients, and notifies the user.

[1576] "Means for checking inventory information and missing ingredients using smart devices while on the go" refers to a function that allows users to check inventory information and missing ingredients in their refrigerators in real time using a smartphone or tablet while on the go, such as at a supermarket.

[1577] "Means for pre-processing captured images" refers to algorithms or software for performing pre-processing such as resizing, noise removal, and color correction of images before image analysis.

[1578] "Means to order missing ingredients via a delivery service" refers to an interface or API for automatically ordering missing ingredients from a delivery service.

[1579] "Means for updating algorithms to improve the accuracy of recipe suggestions based on purchase history and frequency of use" refers to machine learning models and algorithms that optimize future recipe suggestions by taking into account a user's past purchase history and frequency of use of an item.

[1580] MODE FOR CARRYING OUT THE INVENTION

[1581] The system of the present invention aims to efficiently manage items in a refrigerator, provide optimal recipes to users, and facilitate checking and purchasing of ingredients that are in short supply. The system includes a camera in the refrigerator, a server connected to the Internet, and a user's device (such as a smartphone or tablet).

[1582] Taking pictures inside the refrigerator

[1583] Device: A camera installed inside the refrigerator takes pictures based on user instructions or a regular schedule. When a user presses the "take a picture" button on a smartphone application, the camera activates and captures an image of the inside of the refrigerator. For example, a user may issue a voice command such as "Take a picture of the inside of the refrigerator."

[1584] Uploading an image

[1585] Device: The captured images are automatically uploaded to a server using Wi-Fi or other communication methods. For example, a smartphone application can link with the refrigerator camera and send image data to the server.

[1586] Image analysis and item identification

[1587] Server: The server receives the uploaded images and performs image analysis using machine learning models such as TensorFlow and PyTorch. It identifies the items identified through the analysis and records that information in a database. For example, items such as "carrots," "milk," and "eggs" are identified.

[1588] Inventory updates

[1589] Server: The identified item information is stored in a database and the refrigerator inventory information is updated to the latest state, so that the inventory is always reflected in real time.

[1590] Recipe suggestions

[1591] Server: Based on the latest inventory information, the server calls an external recipe suggestion service API to obtain recipe information that can be made with the current inventory. For example, recipes for "stir-fried vegetables" and "omelette" are obtained from the recipe suggestion service.

[1592] Viewing Recipes

[1593] Device: The recipe information sent from the server is displayed on the user's smartphone or tablet. This information also includes a list of ingredients, which the user can check. Specifically, the smartphone app displays recipes for "stir-fried vegetables" and "omelette" along with their ingredient lists.

[1594] Identifying and notifying shortages

[1595] On the device: Based on the ingredients list of a suggested recipe, identify any missing ingredients and notify the user. For example, if a recipe requires a missing bell pepper, a pop-up notification will appear on the device stating, "You are missing bell peppers."

[1596] Check inventory on the go

[1597] User: While on the go (e.g., at the supermarket), they can use their smartphone to check real-time inventory information and a list of ingredients they are short on. For example, a user opens the app and sees that there are six eggs in the refrigerator.

[1598] Delivery service collaboration (when at home)

[1599] User: If a user is at home and is short on ingredients, the system provides the option to automatically order the missing ingredients using a delivery service API. For example, when a user presses the "Order Delivery" button on their smartphone, the delivery service API is called and green peppers are automatically ordered.

[1600] Optimization based on purchase history and frequency of use

[1601] Server: Once the purchase or delivery is complete, the user is notified and the refrigerator inventory is updated again. The server also updates the algorithm to improve the accuracy of future recipe suggestions based on the user's purchase history and frequency of use. For example, the server can retrain the algorithm based on the purchase data to make better suggestions next time.

[1602] Examples of prompt statements

[1603] "Suggest a recipe that can be made with the items in your refrigerator. Currently, we have carrots, milk, and eggs."

[1604] This invention allows users to efficiently manage their refrigerator inventory, select suitable recipes, and easily check and purchase ingredients they are running low on.

[1605] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1606] Program processing flow

[1607] Step 1:

[1608] Device: A camera installed inside the refrigerator takes pictures based on user instructions or set timing. The input is the user's instructions or schedule, and the output is the image taken inside the refrigerator. For example, when a user presses the "take a picture" button in the app, the camera is activated and an image of the inside of the refrigerator is captured.

[1609] Specific operation: When a user presses the "shoot" button on the smartphone app, the camera starts up and takes a picture of the inside of the refrigerator. At this time, the internal LED light turns on, allowing a clear image to be captured in bright light.

[1610] Step 2:

[1611] Terminal: Acquires captured images and uploads them to a server via Wi-Fi. The input is the captured image data, and the output is the image data uploaded to the server. After the camera captures the image, the smartphone app automatically collects the image data and sends it to the server.

[1612] What it does: After taking a photo, the app uploads the image data to a server via Wi-Fi. This process runs in the background, and users can monitor the progress via a progress bar within the app.

[1613] Step 3:

[1614] Server: The server receives the uploaded images and performs image analysis using a machine learning model (e.g., TensorFlow or PyTorch). The input is the uploaded image data, and the output is a list of identified items. Through analysis, the items in the refrigerator are identified.

[1615] Specific operation: When the server receives the image data, the machine learning model is activated and analyzes the items in the refrigerator. From the analysis results, items such as "carrots," "milk," and "eggs" are identified. The results are recorded in a database.

[1616] Step 4:

[1617] Server: Stores the identified item information in a database and updates the inventory information in the refrigerator. The input is a list of identified items, and the output is updated inventory information. This ensures that the inventory is always updated in real time.

[1618] What happens: The server updates the inventory table in the database, recording new information such as "3 carrots, 1 liter of milk, and 6 eggs."

[1619] Step 5:

[1620] Server: Based on the latest inventory information, it calls an external recipe suggestion service API to obtain recipe information that can be made. The input is the latest inventory information, and the output is the suggested recipe information.

[1621] Specific operation: The server calls the recipe suggestion service API and sends the current inventory information. For example, based on the inventory information for "carrots, milk, and eggs," recipe information for "stir-fried vegetables" and "omelette" is obtained.

[1622] Step 6:

[1623] Device: The recipe information sent from the server is displayed on the user's smartphone or tablet. The input is the suggested recipe information, and the output is the recipe information displayed on the device.

[1624] What it does: The app on the user's smartphone displays recipes for "stir-fried vegetables" and "omelette," along with a list of ingredients. A detailed explanation of the recipe and cooking instructions are also displayed.

[1625] Step 7:

[1626] Terminal: Based on the recipe ingredient list, it identifies missing ingredients and notifies the user. The input is the suggested recipe information and current inventory information, and the output is the list of missing ingredients and a notification.

[1627] What it does: The device checks the ingredients in the suggested recipe against the current inventory and displays a notification to the user saying, "You're low on bell peppers." The list of ingredients you're short on is available in the app.

[1628] Step 8:

[1629] User: Check real-time inventory information and material shortage lists on the go using a smartphone or tablet. The input is an access request from the smartphone, and the output is real-time inventory information.

[1630] Specific behavior: When a user opens the app at a supermarket and taps the stock check button, they can see that there are 6 eggs left.

[1631] Step 9:

[1632] User: When there are missing materials while at home, the system automatically orders the missing materials using the delivery service API. The input is a list of missing materials, and the output is a notification that the delivery order has been completed.

[1633] Specific operation: When a user presses the "Order Delivery" button in the app, the delivery service API is called and "green peppers" are automatically ordered. After the order is completed, a notification is displayed on the smartphone.

[1634] Step 10:

[1635] Server: Notifies the user that the purchase or delivery is complete and updates the refrigerator inventory information again. It also updates the algorithm to improve the accuracy of recipe suggestions based on purchase history and frequency of use. The input is the purchase completion notification, and the output is the updated inventory information and the algorithm.

[1636] Specific operation: When the delivery is complete, a notification saying "Peppers delivery completed" is displayed on the user's device. The server updates the inventory information in the database and retrains the algorithm based on the purchase data to improve the accuracy of the next recommendation.

[1637] (Application example 1)

[1638] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1639] In modern life, managing ingredients in the refrigerator is complicated, and the hassle of checking for missing ingredients and the stress of selecting the optimal recipe are issues. Many users also find it troublesome to check inventory while on the go and to purchase or order missing ingredients. Therefore, there is a need for a system that can efficiently manage refrigerator inventory information, suggest optimal recipes, and automatically order missing ingredients.

[1640] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1641] In this invention, the server includes a means for taking images of the inside of the refrigerator, a means for uploading the taken images to the server, a means for analyzing the uploaded images to identify items, and a means for automatically ordering missing ingredients from a food delivery service. This makes it possible to efficiently manage inventory information in the refrigerator, propose optimal recipes, and automatically order missing ingredients.

[1642] The "means for taking images inside the refrigerator" is a device that uses a camera installed inside the refrigerator to take images of items inside the refrigerator periodically or based on a user's instruction.

[1643] The "means for uploading captured images to a server" is a communication function for transmitting captured image data to a server via the Internet.

[1644] The "means of identifying items by analyzing uploaded images" refers to a technology that uses machine learning algorithms and image analysis technology on a server to recognize and identify each item in a refrigerator from uploaded images.

[1645] The "means for storing data on identified items in a database and updating inventory information" is a system for storing analyzed item information in a database and keeping the current inventory information in the refrigerator up to date.

[1646] "Means of suggesting recipes that can be made by calling the API of a recipe suggestion service based on updated inventory information" refers to a method of using the API of an external recipe suggestion service based on the latest inventory information to obtain recipes for dishes that can be made based on current inventory.

[1647] The "means for displaying the ingredient list of the proposed recipe on a terminal" is a means for displaying the acquired recipe information and the list of necessary ingredients on a terminal such as a user's smartphone or tablet.

[1648] The "means for automatically ordering missing ingredients from a food delivery service" is a technology that determines which ingredients are missing to create a recipe and automatically orders the missing ingredients from a food delivery service via the Internet.

[1649] "Means for pre-processing uploaded images" refers to methods for pre-processing captured images, such as filtering and resizing, in order to improve image quality or extract necessary information before image analysis.

[1650] The "means for notifying the user of missing ingredients" is a technology that sends a notification message to a smartphone, tablet, etc. to inform the user of any missing ingredients needed for a recipe proposal.

[1651] An embodiment of the present invention will now be described. The system of the present invention efficiently manages items in a refrigerator, proposes optimal recipes, and automatically orders ingredients that are in short supply. This system includes the following components: a camera in the refrigerator, a server connected to the Internet, and a user terminal (such as a smartphone or tablet).

[1652] First, the basic operation steps of the system will be explained.

[1653] Basic operations

[1654] 1. Taking pictures inside the refrigerator

[1655] Terminal: A camera installed inside the refrigerator takes pictures of the inside periodically or based on the user's instructions.

[1656] 2. Upload an image

[1657] Terminal: Uploads captured images to a server via Wi-Fi or other communication means.

[1658] 3. Image analysis and item identification

[1659] Server: Analyzes the uploaded images and identifies the items in the refrigerator. This analysis uses machine learning models such as TensorFlow and PyTorch.

[1660] 4. Updating inventory information

[1661] Server: Stores the information of identified items in a database and keeps the inventory information in the refrigerator up to date.

[1662] 5. Recipe suggestions

[1663] Server: Based on the updated inventory information, it calls an external recipe database (e.g., a recipe suggestion service API), matches the available ingredients, and retrieves recipes that can be made.

[1664] 6. Viewing Recipes

[1665] Device: The recipe information sent from the server is displayed on the user's smartphone or tablet. The displayed recipe also includes a list of necessary ingredients.

[1666] 7. Identification and notification of shortages

[1667] Terminal: Identify missing ingredients from the suggested recipe's ingredient list and notify the user. The list of missing ingredients is displayed on the terminal so the user can review them.

[1668] 8. Automatic ordering of missing materials

[1669] Server: Automatically orders missing ingredients from a food delivery service, using the food delivery service's API.

[1670] Hardware and software used

[1671] 1. Refrigerator camera

[1672] Example: Camera using the Raspberry Pi camera module

[1673] 2. Internet connection environment

[1674] Example: Wi-Fi router

[1675] 3. User's Device

[1676] Example: Smartphones and tablets

[1677] 4. Server

[1678] Programming: Python

[1679] Image processing: OpenCV

[1680] Item identification: Machine learning libraries such as TensorFlow and PyTorch

[1681] Communication:requests library

[1682] Specific examples

[1683] A user uses the "SmartKitchen" app while there is chicken, onions, and carrots in the refrigerator. The camera inside the refrigerator automatically takes a picture, and the app suggests recipes for "chicken teriyaki" and "curry." Because the user is low on onions, the app automatically places an order with the nearest food delivery service and notifies the user.

[1684] Prompt Sentence Examples

[1685] Prompt for an application that determines what ingredients are in the refrigerator, suggests recipes to the user, and automatically orders any missing ingredients:

[1686] Identify ingredients from images of your refrigerator and suggest recipes based on your current inventory, including automatically ordering missing ingredients through a food delivery service.

[1687] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1688] Step 1:

[1689] Taking pictures inside the refrigerator

[1690] The device takes pictures using a camera installed inside the refrigerator. In this process, the camera module captures images of the inside of the refrigerator periodically or based on the user's instructions. The input is the current state of the inside of the refrigerator, and the output is a captured image file.

[1691] Step 2:

[1692] Uploading an image

[1693] The device uploads the captured image file to the server. In this process, the image data is sent to the server via an internet connection. The input is the captured image file, and the output is the image data stored on the server.

[1694] Step 3:

[1695] Image analysis and item identification

[1696] The server analyzes the uploaded image data and identifies the items in the refrigerator. This process uses machine learning models (e.g., TensorFlow, PyTorch) to analyze the image data and identify specific items. The input is the uploaded image data, and the output is a list of identified items.

[1697] Step 4:

[1698] Inventory updates

[1699] The server saves the identified item list in a database and updates the inventory information. This process writes the identified item data to the database and updates the refrigerator inventory information. The input is the identified item list, and the output is the updated inventory information.

[1700] Step 5:

[1701] Recipe suggestions

[1702] Based on the updated inventory information, the server calls the API of the recipe suggestion service to suggest recipes that can be made. This process calls an external recipe database API to obtain appropriate recipes based on the inventory information in the refrigerator. The input is the updated inventory information, and the output is a list of suggested recipes.

[1703] Step 6:

[1704] Viewing Recipes

[1705] The device displays the recipe information sent from the server on the user's smartphone or tablet. In this process, the suggested recipe and the list of required ingredients are displayed on the device. The input is the suggested recipe list, and the output is the recipe information displayed on the user's device.

[1706] Step 7:

[1707] Identifying and notifying shortages

[1708] The terminal identifies missing ingredients from the ingredient list of the proposed recipe and notifies the user. This process compares the missing ingredients with current inventory information and sends a notification message to the user. The input is the ingredient list of the proposed recipe and current inventory information, and the output is a missing ingredient list and a notification message.

[1709] Step 8:

[1710] Automatic ordering of missing materials

[1711] The server automatically orders the missing ingredients from a food delivery service. This process uses the list of missing ingredients to call the food delivery service's API and confirm the order. The input is the list of missing ingredients, and the output is order confirmation information.

[1712] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1713] MODE FOR CARRYING OUT THE INVENTION

[1714] Configuration overview

[1715] This system manages the items in the refrigerator, suggests optimal recipes, notifies users of missing ingredients, and also incorporates a user emotion recognition engine. The system includes a camera in the refrigerator, an internet-connected server, a user device (such as a smartphone or tablet), and an emotion recognition engine.

[1716] Basic operations

[1717] 1. Taking pictures inside the refrigerator

[1718] Device: A camera installed inside the refrigerator takes pictures of the interior periodically or upon user instruction.

[1719] 2. Upload an image

[1720] Terminal: Uploads captured images to a server via Wi-Fi or other communication means.

[1721] 3. Image analysis and item identification

[1722] Server: Preprocesses the received images, adjusting brightness and contrast, and removing noise to improve the accuracy of image recognition.

[1723] Server: The preprocessed images are input into a machine learning model (e.g., TensorFlow, PyTorch) and analyzed to identify the items in the refrigerator.

[1724] 4. Updating inventory information

[1725] Server: Stores information about items identified through image analysis in a database and updates the current inventory information in the refrigerator.

[1726] 5. Recipe suggestions

[1727] Server: Based on the updated inventory information, the API of the recipe suggestion service is called, and the available ingredients are matched to obtain recipes that can be made.

[1728] 6. Emotion recognition

[1729] Device: The camera and microphone on the user's device are used to collect facial and voice data.

[1730] Emotion engine: Analyzes collected data and determines the user's emotional state (e.g., joy, sadness, stress, etc.).

[1731] 7. Optimizing recipe suggestions

[1732] Server: Based on the emotional information determined by the emotion engine, the server suggests recipes that suit the user's current emotional state.

[1733] 8. Viewing Recipes

[1734] Device: The recipe information sent from the server is displayed on the user's smartphone or tablet. The displayed recipe also includes a list of required ingredients.

[1735] 9. Identification and Notification of Shortage Materials

[1736] Terminal: Identify missing ingredients from the suggested recipe's ingredient list and notify the user of this information.

[1737] 10. Check inventory on the go

[1738] Users: Access the system from their smartphones or tablets while on the go, such as at the supermarket, to check real-time inventory information and a list of materials that are in short supply.

[1739] 11. Delivery service collaboration (when at home)

[1740] User: If there are missing materials while at home, select the delivery service integration option.

[1741] Server: Calls the delivery service's API to automatically order missing ingredients.

[1742] 12. Notifications and Optimization

[1743] Server: Notifies the user that the purchase or delivery is complete and tracks the delivery progress.

[1744] Terminal: Once the delivery is complete, the user is notified again and the refrigerator inventory information is updated.

[1745] Server: Updates the algorithm to improve the accuracy of future suggestions based on the user's purchase history and frequency of use.

[1746] By incorporating a user emotion recognition engine, recipes can be suggested based on the user's current emotions, providing a more personalized experience.

[1747] Specific examples

[1748] User: There are carrots, milk, and eggs in the refrigerator. The user comes home from work and is stressed.

[1749] Server: These items are identified through image analysis and stored in a database.

[1750] Emotion engine: Determines the user's stress level from facial expressions and voice data, and prioritizes recipes with a relaxing effect.

[1751] Server: Recipes that contribute to relaxation, such as "carbonara," are obtained from a recipe suggestion service.

[1752] Device: The smartphone will display a recipe for "Carbonara" along with a list of ingredients needed.

[1753] User: Checks the supermarket for a missing ingredient (e.g., bacon) and adds it to a shopping list.

[1754] In this way, this system allows users to efficiently manage their refrigerator inventory and also suggests optimal recipes that suit their emotional state at the time, resulting in a more satisfying cooking experience.

[1755] The processing flow will be explained below.

[1756] Step 1:

[1757] Device: A camera installed inside the refrigerator takes pictures of the inside of the refrigerator at regular intervals or as set by the user.

[1758] Step 2:

[1759] Terminal: Uploads captured images to a server via Wi-Fi or other communication means.

[1760] Step 3:

[1761] Server: Preprocesses the received images, adjusting brightness and contrast, and removing noise to improve the accuracy of image recognition.

[1762] Step 4:

[1763] Server: The preprocessed images are input into a machine learning model (e.g., TensorFlow, PyTorch) and analyzed to identify the items in the refrigerator.

[1764] Step 5:

[1765] Server: Stores information about items identified through image analysis in a database and updates the current inventory information in the refrigerator.

[1766] Step 6:

[1767] Server: Based on the updated inventory information, the API of the recipe suggestion service is called, and the available ingredients are matched to obtain recipes that can be made.

[1768] Step 7:

[1769] Device: The camera and microphone on the user's device are used to collect facial and voice data.

[1770] Step 8:

[1771] Emotion engine: Analyzes collected data and determines the user's emotional state (e.g., joy, sadness, stress, etc.).

[1772] Step 9:

[1773] Server: Based on the emotional information determined by the emotion engine, the server optimizes and suggests recipes that suit the user's current emotional state.

[1774] Step 10:

[1775] Device: The recipe information sent from the server is displayed on the user's smartphone or tablet. The displayed recipe also includes a list of required ingredients.

[1776] Step 11:

[1777] On the device: Based on the ingredients list of the suggested recipe, identify any missing ingredients and notify the user.

[1778] Step 12:

[1779] Users: Access the system from their smartphones or tablets while on the go, such as at the supermarket, to check real-time inventory information and a list of materials that are in short supply.

[1780] Step 13:

[1781] User: If there are missing materials while at home, select the delivery service integration option.

[1782] Step 14:

[1783] Server: Calls the delivery service's API to automatically order missing ingredients.

[1784] Step 15:

[1785] Server: Notifies the user once the order is complete and tracks the delivery progress.

[1786] Step 16:

[1787] Terminal: Once the delivery is complete, the user is notified again and the refrigerator inventory information is updated.

[1788] Step 17:

[1789] Server: Updates the algorithm to improve the accuracy of future suggestions based on the user's purchase history and frequency of use.

[1790] Example 2

[1791] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1792] Conventional refrigerator management systems are limited in their ability to manage refrigerator inventory and suggest recipes, making it difficult to provide personalized services that reflect the user's emotional state and circumstances. Furthermore, there are few ways to check inventory information while on the go, making it difficult to take appropriate action when necessary ingredients are unavailable.

[1793] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for capturing images of the inside of the refrigerator, a means for uploading the captured images to the server, a means for preprocessing and analyzing the uploaded images to identify items, a means for saving data on the identified items in a database and updating inventory information, a means for suggesting possible recipes by calling an API of a recipe suggestion service based on the updated inventory information, a means for recognizing the user's emotional state and selecting an optimal recipe, and a means for displaying an ingredient list for the suggested recipe on a terminal. This allows the user to efficiently manage inventory in the refrigerator and receive personalized recipe suggestions based on their emotional state. Furthermore, by checking inventory information and receiving notifications of missing ingredients while on the go, the user can appropriately procure the necessary ingredients.

[1794] The "means for taking images inside the refrigerator" is a mechanism for taking images inside the refrigerator using a camera installed inside the refrigerator.

[1795] The "means for uploading captured images to a server" is a mechanism for transmitting captured image data to a server via a communication means.

[1796] The "means for preprocessing and analyzing uploaded images to identify items" refers to a mechanism that performs preprocessing such as adjusting the brightness and contrast of received image data and removing noise, and then identifies items using a machine learning model.

[1797] The "means for saving data on identified items in a database and updating inventory information" is a mechanism for saving information on items identified by analysis in a database and updating the latest inventory status.

[1798] "Means for calling the API of a recipe suggestion service based on updated inventory information and suggesting recipes that can be made" is a mechanism for calling the API of an external recipe suggestion service based on the latest inventory information and obtaining recipe information that can be made.

[1799] The "means for recognizing the user's emotional state and selecting the optimal recipe" is a mechanism for analyzing the user's facial expressions and voice data to determine their emotional state and select a recipe that is appropriate for that state.

[1800] The "means for displaying the ingredient list of the proposed recipe on the terminal" is a mechanism for displaying the recipe information and ingredient list sent from the server on the user's smartphone or tablet.

[1801] The present invention relates to a system that manages items in a refrigerator, suggests optimal recipes based on the user's emotional state, and notifies the user of any missing ingredients. The system includes a camera in the refrigerator, a server connected to the Internet, a user device (such as a smartphone or tablet), and an emotion engine.

[1802] System Configuration

[1803] 1. Camera in the refrigerator

[1804] Hardware: Camera device

[1805] Software: Image acquisition program

[1806] Function: Takes pictures of the inside of the refrigerator periodically or at the user's command.

[1807] 2. Server

[1808] Hardware: Server Device

[1809] Software: Image preprocessing program, machine learning model (e.g., TensorFlow, PyTorch), recipe suggestion service API, database management system

[1810] Function: Preprocesses received images to identify and specify items. Stores information about identified items in a database and updates inventory information in the refrigerator. Calls the API of a recipe suggestion service based on the updated inventory information and obtains information about recipes that can be created.

[1811] 3. User Device

[1812] Hardware: Smartphones and tablet devices

[1813] Software: Display program, notification program, emotion recognition function

[1814] Function: Displays recipe information sent from the server to the user and notifies them of missing ingredients. Collects the user's facial expressions and voice data and analyzes them with the emotion engine.

[1815] 4. Emotion Engine

[1816] Software: Emotional state analysis algorithm

[1817] Function: Analyzes collected user facial and voice data to determine the user's emotional state. Based on the determined emotional information, selects a recipe that is appropriate for the user's current emotional state.

[1818] Specific examples

[1819] Example 1:

[1820] User: There are carrots, milk, and eggs in the refrigerator. The user comes home from work and is stressed.

[1821] Server: These items are identified through image analysis and stored in a database.

[1822] Emotion engine: Determines the user's stress level from facial expressions and voice data, and prioritizes recipes with a relaxing effect.

[1823] Server: Recipes that contribute to relaxation, such as "carbonara," are obtained from a recipe suggestion service.

[1824] Device: The smartphone will display a recipe for "Carbonara" along with a list of ingredients needed.

[1825] User: Checks the supermarket for a missing ingredient (e.g., bacon) and adds it to a shopping list.

[1826] Example 2 (An example of a prompt using a generative AI model):

[1827] Prompt: "The ingredients in my fridge are carrots, potatoes, cheese, and eggs. I'm feeling relaxed today, so can you recommend a recipe that will fit that mood?"

[1828] Server: Performs image analysis and emotion recognition to suggest relaxing recipes such as "cheese omelet."

[1829] This allows the system to provide a more satisfying cooking experience by allowing users to efficiently manage inventory in the refrigerator, suggesting optimal recipes based on their emotional state at the time, and notifying them of any missing ingredients.

[1830] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1831] Step 1:

[1832] Device: A camera installed inside the refrigerator takes images of the inside of the refrigerator periodically or based on the user's instructions.

[1833] Input: User instruction or timer event.

[1834] Data processing: Acquisition of image data.

[1835] Output: A captured image of the inside of a refrigerator.

[1836] Step 2:

[1837] Terminal: Uploads captured images to a server via Wi-Fi or other communication means.

[1838] Input: Captured image data.

[1839] Data processing: Transfer of image data.

[1840] Output: Image data sent to the server.

[1841] Step 3:

[1842] Server: Preprocesses the received images, adjusting brightness and contrast and removing noise.

[1843] Input: Image data sent to the server.

[1844] Data processing: Image preprocessing (brightness, contrast adjustment, noise removal).

[1845] Output: Preprocessed image data.

[1846] Step 4:

[1847] Server: The preprocessed image data is input into a machine learning model (e.g., TensorFlow, PyTorch) and analyzed to identify the items in the refrigerator.

[1848] Input: Preprocessed image data.

[1849] Data processing: Identification and labeling of items.

[1850] Output: A list of identified items.

[1851] Step 5:

[1852] Server: Stores information about items identified through analysis in a database and updates the inventory information in the refrigerator.

[1853] Input: A list of identified items.

[1854] Data processing: storing information in the database and updating inventory information.

[1855] Output: Updated inventory information.

[1856] Step 6:

[1857] Server: Based on the updated inventory information, the API of the recipe suggestion service is called, and the available ingredients are matched to obtain recipes that can be made.

[1858] Input: Updated inventory information.

[1859] Data processing: Calling the recipe suggestion service API and obtaining recipes.

[1860] Output: A list of suggested recipes.

[1861] Step 7:

[1862] Terminal: Collects the user's facial expressions and voice data and sends them to the emotion engine to recognize the user's emotional state.

[1863] Input: User's facial and voice data.

[1864] Data processing: Collecting and transmitting emotion data.

[1865] Output: Data sent to the emotion engine.

[1866] Step 8:

[1867] Emotion engine: Analyzes collected data and determines the user's emotional state.

[1868] Input: User's facial and voice data.

[1869] Data processing: Analysis of emotional state (e.g., stress, joy).

[1870] Output: Determined emotional state.

[1871] Step 9:

[1872] Server: Based on the emotional information determined by the emotion engine, selects a recipe that is appropriate for the user's current emotional state.

[1873] Input: A list of identified emotional states and suggested recipes.

[1874] Data processing: Recipe selection based on emotional state.

[1875] Output: The best recipe for the user.

[1876] Step 10:

[1877] Terminal: The optimal recipe information sent from the server is displayed on the terminal, and the user is notified of the list of necessary ingredients.

[1878] Input: The selected optimal recipe.

[1879] Data processing: Display recipe information.

[1880] Output: Recipe and ingredients list displayed on the user's device.

[1881] Step 11:

[1882] On the device: Based on the ingredients list of a suggested recipe, identify any missing ingredients and notify the user.

[1883] Input: Materials list and current inventory information.

[1884] Data processing: Identifying missing materials.

[1885] Output: Notify user of missing material.

[1886] Step 12:

[1887] User: Accesses the system from a terminal while on the go, such as at a supermarket, to check refrigerator inventory information and a list of ingredients that are in short supply.

[1888] Input: The user's access request.

[1889] Data processing: Providing real-time inventory information.

[1890] Output: Inventory information and a list of missing materials displayed on the user's terminal.

[1891] Step 13:

[1892] User: When at home, select the delivery service integration option to automatically order missing ingredients.

[1893] Input: User's delivery service selection.

[1894] Data processing: Sending orders to delivery services.

[1895] Output: Delivery order confirmation.

[1896] Step 14:

[1897] Server: Notifies the user that the purchase or delivery is complete and tracks the delivery progress.

[1898] Input: Delivery completion information.

[1899] Data Processing: Delivery progress tracking and notification.

[1900] Output: Delivery completion notification to the user.

[1901] Step 15:

[1902] Terminal: Once the delivery is complete, the user is notified again and the refrigerator inventory is updated.

[1903] Input: Delivery completion information for snooze.

[1904] Data processing: Updating refrigerator inventory information.

[1905] Output: Notification of updated inventory information.

[1906] Step 16:

[1907] Server: Updates the algorithm to improve the accuracy of future suggestions based on the user's purchase history and frequency of use.

[1908] Input: Purchase history and usage frequency data.

[1909] Data processing: Algorithm updates and optimizations.

[1910] Output: Improved accuracy of suggestions for the next time.

[1911] (Application example 2)

[1912] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1913] The present invention aims to improve user satisfaction and convenience by providing a refrigerator inventory management and recipe suggestion system that not only efficiently manages items in a refrigerator, but also suggests optimal cooking instructions based on the user's emotional state and provides a series of processes including automatic ordering of missing ingredients. However, conventional refrigerator management systems have problems in that they do not take into account responses based on the user's emotions and do not have a function for automatically ordering missing ingredients.

[1914] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1915] In this invention, the server includes means for taking images of the inside of the refrigerator, means for uploading the taken images to the server, means for analyzing the uploaded images to identify items, means for saving data on the identified items in a database and updating inventory information, means for calling an API of a recipe suggestion service based on the updated inventory information to provide creatable cooking instructions, means for collecting facial expression and voice data using a camera and microphone of the terminal to acquire the user's emotional state, means for analyzing the collected data to determine the user's emotion, means for proposing cooking instructions according to the determined emotion, means for displaying an ingredient list for the proposed cooking instructions on the terminal, and means for identifying missing items from the suggested ingredient list and automatically ordering them. This allows the user to not only manage inventory in the refrigerator, but also to suggest cooking instructions according to the user's emotional state and automatically order missing ingredients.

[1916] The "means for taking images inside the refrigerator" is a device that uses a camera installed inside the refrigerator to take images of items inside the refrigerator periodically or in response to instructions.

[1917] The "means for uploading captured images to a server" is a communication device that transfers captured images to a server via the Internet.

[1918] The "means for analyzing uploaded images to identify items" is a device that analyzes the images received on the server and uses image recognition algorithms to identify the items in the refrigerator.

[1919] The "means for saving data on identified items in a database and updating inventory information" is a device that saves information on items identified through analysis in a database and updates the inventory status in the refrigerator to the latest status.

[1920] "Means for providing cooking instructions that can be created by calling the API of a recipe suggestion service based on updated inventory information" refers to a device that uses the latest inventory information to access the API of a recipe suggestion service and obtains cooking instructions based on that information.

[1921] "Means for collecting facial expression and voice data using a camera or microphone on a terminal to obtain the emotional state of a user" refers to a device that collects facial expression and voice data using a camera or microphone installed on a terminal in order to grasp the emotional state of a user.

[1922] The "means for analyzing collected data to determine the user's emotions" is a device that analyzes collected facial and voice data and uses emotion recognition algorithms to identify the user's emotional state.

[1923] The "means for proposing cooking instructions according to the determined emotion" is a device that selects and provides optimal cooking instructions based on the user's emotional state.

[1924] The "means for displaying an ingredient list for the proposed cooking instructions on a terminal" is a device that displays an ingredient list based on the proposed cooking instructions on a user's terminal.

[1925] The "means for identifying missing items in the proposed ingredient list and automatically ordering them" refers to a device that verifies the ingredient list based on the proposed cooking instructions, identifies missing items, and automatically orders them.

[1926] System Configuration

[1927] The present invention relates to a "refrigerator item management and recipe suggestion system based on emotional state" and uses the following hardware and software.

[1928] Hardware used:

[1929] Refrigerator camera (connectable Wi-Fi camera)

[1930] User devices (smartphones, tablets)

[1931] server

[1932] Software used:

[1933] Emotion recognition engine (e.g. Affectiva, Microsoft Emotion API)

[1934] Image analysis engine (e.g. TensorFlow, PyTorch)

[1935] Cloud databases (e.g. AWS DynamoDB, Google Cloud Firestore)

[1936] Delivery service APIs (e.g. UberEats, DoorDash)

[1937] Basic operations

[1938] Taking and uploading images of the inside of the refrigerator

[1939] The device sends instructions to a camera installed inside the refrigerator, taking pictures of the inside of the refrigerator periodically or based on the user's instructions. The captured image data is then uploaded to a server via Wi-Fi.

[1940] Image analysis and item identification

[1941] The server preprocesses the uploaded images and uses image recognition algorithms (such as TensorFlow or PyTorch) to identify the items in the refrigerator. The identified item data is stored in a cloud database and inventory information is updated.

[1942] Acquiring and analyzing emotional states

[1943] The device collects facial expression and voice data of the user to obtain the user's emotional state, and the collected data is analyzed using an emotion recognition engine to determine the user's emotional state.

[1944] Recipe suggestions and ingredient lists

[1945] The server then calls the API of the recipe suggestion service to obtain optimal cooking instructions based on the updated inventory information and emotional state. The obtained cooking instructions and ingredient list are then displayed on the user's device.

[1946] Automatic ordering of missing materials

[1947] The terminal identifies any missing items from the proposed materials list and automatically orders them through the delivery service API, with the order status tracked and notified to the user in real time.

[1948] Specific examples

[1949] After the user returns home from work, an image of the contents of the refrigerator is automatically uploaded to the app. The app then acquires the user's facial expression data and determines their emotional state as "tired." Based on this information, the server analyzes the inventory and determines that carbonara is ready to be made, but that there is a shortage of bacon. When the user selects the delivery option, the bacon is automatically ordered, and the progress is displayed in the app.

[1950] Prompt Sentence Examples

[1951] After the user returns home, an image of the inside of the refrigerator and the user's facial expression data are collected from the smartphone. Image analysis is used to check the inventory in the refrigerator, and if fatigue is determined from the facial expression data, what recipes should be suggested and how to order missing ingredients via a delivery service? Please explain this with specific steps.

[1952] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1953] Step 1:

[1954] The user issues a command to take a picture of the refrigerator from the terminal. The input is the user's command, and the output is image data of the inside of the refrigerator. Specifically, the terminal sends a command to take a picture to the camera installed inside the refrigerator, and the camera takes a picture of the current inside of the refrigerator.

[1955] Step 2:

[1956] The captured image data is uploaded from the device to the server. The input is the image data of the inside of the refrigerator, and the output is an image file saved on the server. Specifically, the device transfers the image data to the server via Wi-Fi.

[1957] Step 3:

[1958] The server preprocesses the uploaded image data and performs image analysis. The input is the image file stored on the server, and the output is the data of the identified items in the refrigerator. Specifically, the server adjusts the brightness and contrast of the image and uses a machine learning model to identify the items.

[1959] Step 4:

[1960] The identified item data is stored in a database, and inventory information is updated. The input is the identified item data in the refrigerator, and the output is the updated inventory information. Specifically, the server connects to the database, stores the latest item information, and updates the inventory information.

[1961] Step 5:

[1962] The user's device collects facial and voice data to acquire the user's emotional state. The input is the user's real-time facial and voice data, and the output is the collected data. Specifically, the device collects the data using a camera and microphone.

[1963] Step 6:

[1964] The collected data is sent to a server and analyzed by an emotion recognition engine. The input is the collected facial and voice data, and the output is the user's emotional state. Specifically, the server uses an emotion recognition algorithm to determine the user's emotions.

[1965] Step 7:

[1966] Based on the updated inventory information and emotional state, the server calls the API of the recipe suggestion service to obtain optimal cooking instructions. The input is the updated inventory information and the user's emotional state, and the output is cooking instructions. Specifically, the server calls the API of the recipe suggestion service to obtain cooking instructions.

[1967] Step 8:

[1968] The acquired cooking instructions and ingredient list are displayed on the user's terminal. The input is the cooking instructions and ingredient list, and the output is the information displayed on the terminal. Specifically, the server sends this information to the terminal, and the terminal displays it.

[1969] Step 9:

[1970] The user's device identifies missing items from the proposed materials list and automatically places an order. The input is the materials list and inventory information, and the output is order information for the delivery service. Specifically, the device checks the inventory information to identify missing items and calls the delivery service's API to place the order.

[1971] Step 10:

[1972] The user is notified of the order progress and the inventory information is updated after delivery is completed. The input is the progress of the delivery service, and the output is the updated inventory information and a notification to the user. Specifically, the server receives information from the delivery service, updates the inventory information, and notifies the user.

[1973] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1974] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1975] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1976] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1977] FIG. 9 illustrates an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and behaviors arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1978] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1979] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1980] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1981] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1982] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1983] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1984] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1985] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1986] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1987] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1988] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1989] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1990] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1991] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1992] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1993] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1994] The following is further disclosed regarding the above embodiment.

[1995] (Claim 1)

[1996] A means for taking an image of the inside of a refrigerator;

[1997] A means for uploading the captured image to a server;

[1998] means for analyzing the uploaded image to identify the item;

[1999] a means for storing data of the identified items in a database and updating inventory information;

[2000] A means for suggesting recipes that can be made by calling the API of a recipe suggestion service based on the updated inventory information;

[2001] a means for displaying on the terminal a list of ingredients for the suggested recipe;

[2002] A system including:

[2003] (Claim 2)

[2004] 10. The system of claim 1, further comprising means for pre-processing the uploaded image.

[2005] (Claim 3)

[2006] 10. The system of claim 1, further comprising means for identifying missing ingredients from the ingredient list of the suggested recipe and notifying the user of the missing ingredients.

[2007] "Example 1"

[2008] (Claim 1)

[2009] A means for taking an image of the inside of a refrigerator;

[2010] A means for uploading the captured ...

Claims

1. A means for taking an image of the inside of a refrigerator; A means for uploading the captured image to a server; means for analyzing the uploaded image to identify the item; a means for storing data of the identified items in a database and updating inventory information; A means for suggesting recipes that can be made by calling the API of a recipe suggestion service based on the updated inventory information; a means for displaying on the terminal a list of ingredients for the suggested recipe; A system including:

2. The system of claim 1 further comprising means for pre-processing uploaded images.

3. 10. The system of claim 1, further comprising means for identifying missing ingredients from the ingredient list of the proposed recipe and notifying the user of the missing ingredients.

Images