system

Abstract

We provide the system. [Solution] A means for acquiring image data from inside the refrigerator and recognizing the food items, A method for calculating the priority of use based on the expiration date of recognized ingredients, A means of generating alerts based on usage priority, A method for retrieving current ingredient data from a database and automatically generating available menus, A means of creating a shopping list based on the generated menu, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance as a response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In household and commercial refrigerators, due to insufficient management of food ingredients, food ingredients past their expiration dates are often discarded, causing food loss and wasteful expenses. This problem increases the economic and environmental burdens, especially from the perspective of food loss, which is a current social issue and price increase. Therefore, it is necessary to efficiently manage the food ingredients in the refrigerator and propose available menus to reduce food waste and solve the food loss problem and wasteful expenses.

Means for Solving the Problems

[0005] This invention is a system that recognizes food items in a refrigerator from image data, calculates usage priority based on expiration dates, and generates alerts based on that information. Furthermore, the system retrieves current food data from a database, automatically generates available menus, and creates necessary shopping lists. By linking with mobile devices, it allows users to check food status, alerts, menus, and shopping lists, thereby reducing food waste and unnecessary expenses. This streamlines food management in the refrigerator, reducing economic and environmental burdens.

[0006] "Image data" refers to visual information acquired using cameras or other imaging devices.

[0007] "Ingredients" refers to materials stored in a refrigerator and used for cooking or eating.

[0008] "Recognition" refers to the process of identifying specific food ingredients or their condition from image data.

[0009] "Best before date" indicates the period during which food can maintain its quality under storage conditions.

[0010] "Usage priority" is an indicator that shows the order in which ingredients should be used, based on their expiration dates and other evaluation criteria.

[0011] An "alert" refers to a notification or warning intended to draw attention to something, such as food items nearing their expiration date.

[0012] A "database" is a system that manages and stores data in a structured format.

[0013] A "menu" refers to the combination of meals or dishes offered, and includes the recipes and preparation procedures.

[0014] The "shopping list" refers to a list of supplementary food ingredients that need to be purchased based on the current food ingredients and the planned menu.

[0015] A "mobile device" refers to a portable and multifunctional electronic device such as a mobile phone or a tablet.

Brief Description of the Drawings

[0016] [Figure 1] It is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] It is a conceptual diagram showing an example of the main functions of a data processing device and a smart device according to the first embodiment. [Figure 3] It is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] It is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which a plurality of emotions are mapped. [Figure 10] It shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13]It is a sequence diagram showing the processing flow of the data processing system in Embodiment 2 when the emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when the emotion engine is combined.

Mode for Carrying Out the Invention

[0017] Hereinafter, an example of an embodiment of the system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be one arithmetic unit or a combination of a plurality of arithmetic units. Also, the processor may be one type of arithmetic unit or a combination of a plurality of types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0020] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0021] In the following embodiments, the numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, and the like.

[0022] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0024] [First Embodiment]

[0025] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0026] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0029] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0032] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0036] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0037] This invention is a system that uses generative AI to manage food in a refrigerator and ensure waste-free consumption. The system acquires image data from the refrigerator, recognizes food items, calculates usage priority based on their expiration dates, and generates alerts. Furthermore, it can retrieve current food data from a database, automatically generate available menus, and create necessary shopping lists. This system also integrates with mobile devices, allowing users to check food status, alerts, menus, and shopping lists.

[0038] Specific processing flow of the system

[0039] 1. Collection and management of food ingredient data

[0040] server:

[0041] The camera inside the refrigerator is activated to acquire the latest video data. This data is then sent to an image recognition engine. The image recognition engine identifies and labels the food items (e.g., "milk," "eggs," "spinach," etc.). Furthermore, it reads the expiration date information of the food items and stores this recognized food information in a database.

[0042] Specific example:

[0043] The refrigerator camera captures images, and the image recognition engine recognizes the milk as "milk." If the expiration date is "2023-10-12," that information is saved in the database in the format "Milk: Expiration Date 2023-10-12."

[0044] 2. Calculation of ingredient usage priority and generation of alerts

[0045] server:

[0046] Retrieve all ingredient data from the database. Compare the current date with the expiration date of each ingredient and calculate its usage priority (urgency of consumption). Determine whether ingredients with high usage priority should be alerted (e.g., within 3 days of expiration). Generate an alert message for ingredients that should be alerted.

[0047] Device (mobile app):

[0048] The app receives alert messages from the server and displays them on the app.

[0049] Specific example:

[0050] The server retrieves data for "Milk: Expiration Date 2023-10-12" and confirms that the current date is "2023-10-10". Since the expiration date is in two days, it is determined to be an alert item. It generates the message "Your milk is nearing its expiration date. Please consume it as soon as possible" and sends it to the mobile app.

[0051] 3. Automatic generation of culinary menus

[0052] server:

[0053] The system retrieves current ingredient data from the database and searches the recipe database for menus that can be created based on the available ingredients. It identifies recipes that prioritize the use of high-priority ingredients and generates a list.

[0054] Device (mobile app):

[0055] The system receives data from the server and displays a list of available recipes. If the user selects a recipe, it displays detailed instructions for preparing it.

[0056] Specific example:

[0057] The server recognizes from the database that it has "milk, eggs, and spinach," and searches the recipe database for a "cream pasta" recipe. Since milk is a high-priority ingredient, this recipe is suggested first, and detailed instructions are displayed in the mobile app.

[0058] 4. Automatic generation of necessary shopping lists

[0059] server:

[0060] Based on current ingredient data and the user's desired recipe, the system identifies missing ingredients. It then automatically generates a shopping list and sends it to the user.

[0061] User:

[0062] Check your shopping list on the mobile app. Select the ingredients you plan to buy and complete the list.

[0063] Specific example:

[0064] The server checks if there is no pasta in the refrigerator based on the user's requested "cream pasta" recipe. It automatically adds "pasta" to the shopping list and displays "Please purchase pasta (200g)" in the mobile app.

[0065] This system makes it easier for users to manage their ingredients and effectively reduce unnecessary expenses and food waste.

[0066] The following describes the processing flow.

[0067] Step 1:

[0068] Server: Activates the camera inside the refrigerator and acquires the latest video data. Sends the video data to the image recognition engine.

[0069] Step 2:

[0070] Server: The image recognition engine analyzes the video data and identifies the food ingredients. The identified food ingredients are then labeled (e.g., "milk," "egg," "spinach," etc.).

[0071] Step 3:

[0072] Server: Reads the expiration date information of the recognized food item. Saves the read food item information and expiration date to the database.

[0073] Step 4:

[0074] Server: Retrieves all ingredient data from the database. Compares the current date with the expiration date of each ingredient and calculates the usage priority.

[0075] Step 5:

[0076] Server: Determines whether high-priority ingredients should be subject to an alert. Generates an alert message for ingredients that should trigger an alert.

[0077] Step 6:

[0078] Device (mobile app): Receives alert messages from the server. Displays alert messages on the app.

[0079] Step 7:

[0080] Server: Retrieves current ingredient data from the database. Searches the recipe database for possible menus based on available ingredients.

[0081] Step 8:

[0082] Server: Identifies recipes that prioritize the use of high-priority ingredients and generates a list of them.

[0083] Step 9:

[0084] Terminal (mobile app): Displays a list of available recipes received from the server. If the user selects a recipe, it displays detailed instructions for preparing it.

[0085] Step 10:

[0086] Server: Based on the current ingredient data and the recipe selected by the user, it identifies any missing ingredients. It automatically generates a shopping list and sends it to the user.

[0087] Step 11:

[0088] User: Check the shopping list on the mobile app. Select the ingredients you plan to buy and complete the list.

[0089] Step 12:

[0090] User: Based on the shopping list, purchase the necessary ingredients and use them without waste.

[0091] (Example 1)

[0092] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0093] Traditional refrigerator food management systems rely on manual processes for managing expiration dates and suggesting menus, leading to food waste and loss. Furthermore, users face the challenge of needing to prioritize ingredients and create shopping lists, which requires considerable time and effort. Therefore, there was a need for an automated refrigerator food management system.

[0094] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0095] In this invention, the server includes means for acquiring image data from inside the refrigerator and recognizing food items, means for calculating usage priority based on the expiration dates of the recognized food items, means for generating alerts based on the usage priority, and means for acquiring current food item data from a database and automatically generating available menus. This enables detailed management and efficient use of food items inside the refrigerator.

[0096] "Image data from inside the refrigerator" refers to video information acquired by a camera inside the refrigerator, which is used for recognizing and managing food items.

[0097] "Food ingredients" refers to the food and beverages stored in the refrigerator, and specifically includes items such as milk, eggs, and vegetables.

[0098] "Means of recognition" refers to the process of identifying and labeling food ingredients using image recognition technology.

[0099] The "best before date" refers to the date by which food is expected to maintain its quality; after this date, the likelihood of quality deterioration increases.

[0100] A "means for calculating usage priority" is an algorithm that determines which ingredients should be used preferentially based on their expiration dates and other conditions.

[0101] "Means for generating alerts" refers to a function that creates notifications to inform users of the need to consume high-priority ingredients.

[0102] A "database" is a system that centrally manages digital data, including information on ingredients and recipes.

[0103] "Available menus" refers to a list of dishes that can be made using the ingredients currently in the refrigerator.

[0104] "Methods for automatically generating menus" refers to algorithms that use ingredient information in a database to search for and select the most suitable cooking recipes.

[0105] "A means of creating a shopping list" refers to a function that allows users to list the ingredients they need and prevent them from forgetting to buy them.

[0106] A "mobile device" refers to a communication terminal that a user can carry with them, and includes smartphones and tablets.

[0107] "Image recognition technology" is a technique that analyzes image data acquired by a camera to identify specific objects or features.

[0108] A "generative AI model" refers to an algorithm that uses AI to learn from data and perform a specific task.

[0109] This invention is a system that uses generative AI to manage food in a refrigerator and consume it efficiently without waste. This system mainly consists of the following elements:

[0110] 1. Collection and management of food ingredient data

[0111] The server periodically acquires image data from inside the refrigerator using a high-resolution camera. This video data is used to recognize and label food items using image recognition technologies such as TENSORFLOW® and OpenCV. The information on the recognized food items and their expiration dates are then stored in a database. For example, if the camera inside the refrigerator recognizes "milk" and its expiration date is "2023-10-12", this information is stored in the database as "Milk: Expiration Date 2023-10-12".

[0112] 2. Calculation of ingredient usage priority and generation of alerts.

[0113] The server retrieves all food data from the database and compares the current date with the expiration date of each food item. Based on this, it calculates the priority of use. For foods with a high priority, it generates an alert message. This alert message is used to notify the user of the need to consume food items that are nearing their expiration date. For example, if the data for "Milk: Expiration Date 2023-10-12" reveals that the expiration date is in two days, it generates an alert message saying, "The milk is about to expire. Please consume it as soon as possible," and sends it to the mobile device.

[0114] 3. Automatic generation of culinary menus

[0115] The server retrieves the current ingredient data and searches the recipe database for available menus. It identifies recipes that prioritize the use of high-priority ingredients and generates a list. This list is sent to the mobile device, allowing the user to view detailed preparation instructions. For example, if the current ingredient data is "milk, eggs, spinach," the server searches for a "cream pasta" recipe and suggests it to the user.

[0116] 4. Automatic generation of necessary shopping lists

[0117] The server identifies missing ingredients based on the current ingredient data and the user's desired recipe, and automatically generates a shopping list of the necessary items. For example, if the user wants to make "cream pasta," the server will confirm that pasta is missing and add "Please purchase pasta (200g)" to the shopping list.

[0118] This system allows users to easily understand the status of ingredients and manage them effectively based on usage priority, preventing waste. Furthermore, by integrating with mobile devices, ingredients can be managed from anywhere, contributing to a reduction in forgotten purchases and food waste.

[0119] Example of a prompt:

[0120] "Please share a recipe using the milk, spinach, and eggs I have in my refrigerator. Also, please let me know if there are any other ingredients I might need."

[0121] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0122] Step 1:

[0123] Obtain image data from inside the refrigerator.

[0124] The server periodically activates a camera installed inside the refrigerator to acquire image data from within. This time, the video is acquired daily at 8:00 AM.

[0125] Input: Trigger for video acquisition (regular schedule)

[0126] Data processing / data calculation: Activate the camera and capture the internal video data.

[0127] Output: High-resolution image data

[0128] Specific actions:

[0129] The server activates the refrigerator's camera based on a schedule and captures the current state of its interior as video.

[0130] Step 2:

[0131] Food identification using an image recognition engine

[0132] The server sends the acquired high-resolution image data to an image recognition engine to identify the food ingredients. For example, it might run a model using TensorFlow.

[0133] Input: High-resolution image data

[0134] Data processing / data calculation: Identify and label food ingredients using image recognition engines such as TensorFlow.

[0135] Output: Food labels and recognition data (e.g., milk, eggs, spinach)

[0136] Specific actions:

[0137] The server sends image data to TensorFlow, which runs the model to identify the ingredients as "milk," "eggs," and "spinach." It then generates information with labels for these ingredients.

[0138] Step 3:

[0139] Reading and saving the expiration date to the database.

[0140] The server reads the expiration date of the identified food item and saves that information to the database.

[0141] Input: Food labels and recognition data

[0142] Data processing / data calculation: Read the expiration date on food packaging using OCR technology.

[0143] Output: Food ingredients and their expiration date information (Example: Milk: Expiration date 2023-10-12)

[0144] Specific actions:

[0145] The server uses OCR technology to read the expiration date from the package image of the identified food item and saves this information to the database in the format "Milk: Expiration Date 2023-10-12".

[0146] Step 4:

[0147] Calculation of Usage Priority

[0148] The server retrieves all ingredient data from the database and calculates usage priority based on expiration dates.

[0149] Input: Food ingredients and expiration date information retrieved from the database.

[0150] Data processing / calculation: Compare the current date with the expiration date and calculate the usage priority.

[0151] Output: Ingredient information with set usage priority.

[0152] Specific actions:

[0153] The server retrieves data for "Milk: Expiration Date 2023-10-12" and compares it to the current date (e.g., 2023-10-10). Since the expiration date is 2 days away, it is given a high priority for use.

[0154] Step 5:

[0155] Alert generation

[0156] The server generates alert messages for ingredients with high usage priority.

[0157] Input: Ingredient information with set usage priority.

[0158] Data processing / data calculation: Generate alerts based on usage priority.

[0159] Output: Alert message

[0160] Specific actions:

[0161] The server generates an alert message saying, "The milk is nearing its expiration date. Please consume it as soon as possible," and sends it to the mobile device.

[0162] Step 6:

[0163] Automatic menu generation

[0164] The server retrieves current ingredient data and searches and generates available menus from the recipe database.

[0165] Input: Current ingredient data retrieved from the database

[0166] Data processing / data calculation: Search the recipe database for possible menus based on ingredient data and generate a list.

[0167] Output: List of culinary options

[0168] Specific actions:

[0169] The server retrieves data for "milk, eggs, and spinach," searches the recipe database for "cream pasta" recipes, generates them, and adds them to the list.

[0170] Step 7:

[0171] Displaying recipes and detailed instructions

[0172] The device (mobile app) receives data from the server and displays it so that the user can view detailed creation instructions.

[0173] Input: List of possible menu items

[0174] Data processing / data calculation: Format data from the server for display.

[0175] Output: Menu list and detailed instructions

[0176] Specific actions:

[0177] The mobile app displays detailed instructions for users who select a "cream pasta" recipe. These instructions include steps such as, "First, prepare the milk, then add the eggs and mix."

[0178] Step 8:

[0179] Automatically generate shopping lists

[0180] The server identifies any missing ingredients based on the recipe selected by the user and generates a shopping list.

[0181] Input: Selected recipe and current ingredient data

[0182] Data processing / data calculation: Identify missing ingredients and generate a shopping list.

[0183] Output: Shopping list

[0184] Specific actions:

[0185] The server compares the "cream pasta" recipe with the current ingredient data and confirms that pasta is missing. It automatically adds "Please purchase pasta (200g)" to the shopping list.

[0186] Step 9:

[0187] Review and edit your shopping list

[0188] Users can view their shopping list on the mobile app and add or edit items they plan to buy as needed.

[0189] Input: Shopping list

[0190] Data processing / data calculation: Operation data when a user edits a list.

[0191] Output: Updated shopping list

[0192] Specific actions:

[0193] The user opens the mobile app, sees the message "Please purchase pasta (200g)," and adds any other items they need to the list.

[0194] (Application Example 1)

[0195] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0196] There is a need for methods to effectively manage the ingredients in refrigerators, reduce food waste due to expired food, and enable users to utilize ingredients efficiently. Furthermore, an efficient system is needed to allow users to easily obtain recipes and quickly replenish missing ingredients. This is expected to save time and improve consumer behavior.

[0197] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0198] In this invention, the server includes means for acquiring image data from inside the refrigerator and recognizing ingredients; means for calculating usage priority based on the expiration date of the recognized ingredients; means for generating alerts based on the usage priority; means for acquiring current ingredient data from a database and automatically generating available menus; means for creating a shopping list based on the generated menus; means for suggesting recipes using available ingredients based on the user's refrigerator information; and means for coordinating with a food delivery service that automatically orders any missing ingredients based on the suggested recipes. This enables efficient use and rapid replenishment of ingredients.

[0199] "Image data" refers to data that represents visual information in a digital format.

[0200] "Ingredients" refer to food products intended for cooking or consumption.

[0201] "Recognition" means identifying and understanding a specific object.

[0202] The "best before date" indicates the period during which food can be eaten safely and deliciously.

[0203] "Usage priority" refers to the order in which multiple ingredients should be used first.

[0204] An "alert" is a warning message that notifies the user.

[0205] A "database" is a digital system that organizes, stores, and manages related data.

[0206] A "menu" refers to the composition and plan of the dishes that will be served as meals within a certain period of time.

[0207] A "shopping list" is a list of items that need to be purchased.

[0208] "Mobile devices" is a general term for portable electronic devices.

[0209] A "recipe" is a set of instructions that shows how to prepare food and the steps involved.

[0210] A "food delivery service" is a service that delivers ordered food to a specified location.

[0211] This invention is a system for efficiently managing food items in a refrigerator and consuming them without waste. This system is realized through the collaboration of a server and a mobile device. The server manages food data, while the mobile device functions as an interface with the user.

[0212] Collection and management of food ingredient data

[0213] Server: The server activates a camera installed inside the refrigerator and acquires the latest image data. The acquired image data is used to identify and label food items using an image recognition engine such as TensorFlow or OpenCV. In addition, the expiration date is automatically read, and this information is stored in the food item database.

[0214] Specific example: An image of milk is recognized, and along with labeling, the expiration date "2023-10-12" is added.

[0215] Calculation of ingredient usage priority and generation of alerts

[0216] Server: The server retrieves all ingredient data from the database and calculates usage priority by comparing the current date with the expiration date of each ingredient. It generates alerts for ingredients nearing their expiration date and notifies the user's mobile device.

[0217] Specific example: The server retrieves data for "Milk: Expiration Date 2023-10-12," verifies that the current date is "2023-10-10," and then generates a warning message: "The milk is nearing its expiration date."

[0218] Automatic generation of cookable menus

[0219] Server: The server retrieves current ingredient data from the database and automatically generates menus based on the available ingredients from the recipe database. Recipes using high-priority ingredients are also suggested.

[0220] Specific example: A recipe for "cream pasta" is suggested based on the ingredients in the refrigerator (milk, eggs, spinach).

[0221] Automatically generate a shopping list of necessary items.

[0222] Server: The server identifies missing ingredients based on the recipe selected by the user and the current ingredient data, and automatically generates a shopping list. Furthermore, it sends this list to the user's mobile device.

[0223] Specific example: A user selects a "cream pasta" recipe, and since there is no pasta in the refrigerator, "pasta (200g)" is automatically added to the shopping list.

[0224] Collaboration with food delivery

[0225] Server: The server also has a function to automatically coordinate with food delivery services and order any missing ingredients. This allows users to quickly obtain the ingredients they need.

[0226] Specific example: Based on a shopping list, automatically order "pasta (200g)" that is missing from the refrigerator using a food delivery service.

[0227] Example of a prompt:

[0228] "Create a program that retrieves information about the ingredients in the refrigerator, generates recipes to suggest in a food delivery application based on that information, and automatically orders any missing ingredients from a food delivery service."

[0229] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0230] Step 1:

[0231] The server activates the camera inside the refrigerator and retrieves the latest image data.

[0232] Input: Image data from inside the refrigerator.

[0233] Processing: Image data is acquired from a camera inside the refrigerator and sent to the system. The image recognition engine identifies the food items.

[0234] Output: A list of identified ingredients.

[0235] Step 2:

[0236] The server uses TensorFlow or OpenCV to label the recognized food items and read their expiration dates.

[0237] Input: Image data of food ingredients identified by an image recognition engine.

[0238] Processing: Label food items with labels such as "milk" and "eggs," and extract their expiration date information.

[0239] Output: A list of labeled ingredients and their corresponding expiration dates.

[0240] Step 3:

[0241] The server stores a list of ingredients and expiration date data in a database.

[0242] Input: A list of labeled ingredients and their expiration dates.

[0243] Processing: The process of saving ingredient information to a database.

[0244] Output: Ingredient data stored in the database.

[0245] Step 4:

[0246] The server retrieves all ingredient data from the database, calculates usage priority, and generates a warning message.

[0247] Input: A list of ingredients stored in the database and the expiration date of each ingredient.

[0248] Processing: Compare the current date with the expiration date of each ingredient and calculate the priority for use. Generate a warning message for ingredients that are nearing their expiration date.

[0249] Output: The generated warning message.

[0250] Step 5:

[0251] The server sends a warning message to the mobile device.

[0252] Input: The generated warning message.

[0253] Process: Send a warning message to a mobile device.

[0254] Output: Warning message displayed on the mobile device.

[0255] Step 6:

[0256] The server retrieves current ingredient data from the database and automatically generates usable menus from the recipe database.

[0257] Input: Food ingredient data stored in the database.

[0258] Processing: Based on ingredient data, the system searches for available recipes in the recipe database and automatically generates them.

[0259] Output: Suggested recipe list.

[0260] Step 7:

[0261] The server sends the suggested recipe list to the mobile device and displays it to the user.

[0262] Input: Suggested recipe list.

[0263] Processing: Send the recipe list to your mobile device.

[0264] Output: Recipe list displayed on mobile devices.

[0265] Step 8:

[0266] The user selects their desired recipe on their mobile device, and the server identifies any missing ingredients and automatically generates a shopping list.

[0267] Input: User-selected recipe, current ingredient data.

[0268] Processing: Identifies missing ingredients and automatically generates a shopping list.

[0269] Output: Auto-generated shopping list.

[0270] Step 9:

[0271] The server uses an automatically generated shopping list to place orders with a food delivery service.

[0272] Input: Auto-generated shopping list.

[0273] Processing: Send order data to the food delivery service.

[0274] Output: Order confirmation for food delivery service.

[0275] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0276] This invention is a system that manages food items in a refrigerator and provides suggestions that take the user's emotions into consideration, utilizing generative AI and an emotion engine. The system acquires image data from inside the refrigerator, recognizes food items, calculates usage priority based on their expiration dates, and generates alerts. It also retrieves current food data from a database, automatically generates available menus, and creates necessary shopping lists. This system works in conjunction with mobile devices, allowing users to check the status of food items, alerts, menus, and shopping lists. Furthermore, it recognizes the user's emotions and uses that information to improve food selection and menu suggestions.

[0277] Specific processing flow of the system

[0278] 1. Collection and management of food ingredient data

[0279] server:

[0280] The camera inside the refrigerator is activated to acquire the latest video data. This data is then sent to an image recognition engine. The image recognition engine identifies and labels the food items (e.g., "milk," "eggs," "spinach," etc.). Furthermore, it reads the expiration date information of the food items and stores this recognized food information in a database.

[0281] Specific example:

[0282] The refrigerator camera acquires an image, and the image recognition engine recognizes the milk as "milk". Also, if the expiration date is marked as "2023-10-12", save that information in the database in the format of "milk: expiration date 2023-10-12".

[0283] 2. Calculation of ingredient usage priority and alert generation

[0284] Server:

[0285] Retrieve all ingredient data from the database. Compare the current date with the expiration date of each ingredient to calculate the usage priority. Determine whether an ingredient is an alert target based on its usage priority. Generate an alert message for the alert target ingredients.

[0286] Terminal (mobile app):

[0287] Receive the alert message from the server and display the alert message on the app.

[0288] Specific example:

[0289] The server retrieves the data of "milk: expiration date 2023-10-12" and confirms that the current date is "2023-10-10". Since the expiration date is in 2 days, it is determined as an alert target. Generate a message "The expiration date of milk is approaching. Please consume it soon." and send it to the mobile app.

[0290] 3. Automatic generation of possible menus

[0291] Server:

[0292] Retrieve the current ingredient data from the database again. Search the recipe database for possible menus that can be created based on the available ingredients. Identify recipes that preferentially use ingredients with high usage priority and generate a list of them.

[0293] Terminal (mobile app):

[0294] Displays a list of available recipes received from the server. If the user selects a recipe, it displays detailed instructions for preparing it.

[0295] Specific example:

[0296] The server recognizes from the database that it has "milk, eggs, and spinach," and searches the recipe database for a "cream pasta" recipe. Since milk is a high-priority ingredient, this recipe is suggested first, and detailed instructions are displayed in the mobile app.

[0297] 4. Automatic generation of necessary shopping lists

[0298] server:

[0299] Based on current ingredient data and the user's selected recipe, the system identifies missing ingredients. It then automatically generates a shopping list and sends it to the user.

[0300] User:

[0301] Check your shopping list on the mobile app. Select the ingredients you plan to buy and complete the list.

[0302] Specific example:

[0303] The server checks if there is no pasta in the refrigerator based on the user's requested "cream pasta" recipe. It automatically adds "pasta" to the shopping list and displays "Please purchase pasta (200g)" in the mobile app.

[0304] Utilizing the Emotion Engine

[0305] 5. Collection of emotional data

[0306] Device (mobile device):

[0307] Analyze the user's facial expressions and voice tones through built-in cameras and sensors, and collect emotional data. For example, use the camera to perform user face recognition and determine signs of smiling or stress.

[0308] 6. Analysis and Reflection of Emotional Data

[0309] Server:

[0310] Analyze the emotional data sent from the terminal, and judge the user's current emotional state. Based on that information, adjust the selection of ingredients and recipe suggestions. For example, when the user is feeling stressed, prioritize suggesting recipes that are easy to make and favorite ingredients.

[0311] Specific Example:

[0312] The face recognition function of the mobile app captures the user's smile and sends "The user is relaxed" to the server. The server determines that "Since the user is relaxed, propose a new challenge recipe" and adjusts the proposal content.

[0313] 7. Adjustment of Alerts and Notifications

[0314] Server: <00Q0990>

[0315] Based on the emotional data, adjust the expression method and notification timing of alerts. For example, when the user is feeling stressed, change the notification to a softer expression or delay the notification timing.

[0316] Specific Example:

[0317] The server determines that "The user is feeling stressed" and changes the alert to a softer expression such as "The expiration date of the milk is approaching. Please consume it when you have time." and sends the alert.

[0318] Through the processing flow described above, this invention streamlines the management of ingredients in the refrigerator and reduces food waste and loss by providing suggestions that take user feelings into consideration.

[0319] The following describes the processing flow.

[0320] Step 1:

[0321] Server: Activates the camera inside the refrigerator and acquires the latest video data. Then, it sends the acquired video data to the image recognition engine.

[0322] Step 2:

[0323] Server: The image recognition engine analyzes the video data and identifies the food ingredients. The identified food ingredients are then labeled (e.g., "milk," "egg," "spinach," etc.).

[0324] Step 3:

[0325] Server: Reads the expiration date information of the recognized food item. Saves the read food item information and expiration date to the database.

[0326] Step 4:

[0327] Server: Retrieves all ingredient data from the database. Compares the current date with the expiration date of each ingredient and calculates the usage priority (urgency of consumption).

[0328] Step 5:

[0329] Server: Determines whether high-priority ingredients should be subject to an alert. Generates an alert message for ingredients that should trigger an alert.

[0330] Step 6:

[0331] Device (mobile app): Receives alert messages from the server. Displays alert messages on the app.

[0332] Step 7:

[0333] Server: Retrieves current ingredient data from the database. Searches the recipe database for possible menus based on available ingredients.

[0334] Step 8:

[0335] Server: Identifies recipes that prioritize the use of high-priority ingredients and generates a list of them.

[0336] Step 9:

[0337] Terminal (mobile app): Displays a list of available recipes received from the server. If the user selects a recipe, it displays detailed instructions for preparing it.

[0338] Step 10:

[0339] Server: Based on current ingredient data and the user's selected recipe, it identifies missing ingredients. It automatically generates a shopping list and sends it to the user.

[0340] Step 11:

[0341] User: Check the shopping list on the mobile app. Select the ingredients you plan to buy and complete the list.

[0342] Step 12:

[0343] Mobile device: Analyzes the user's facial expressions and voice tone through its built-in camera and sensors to collect emotional data.

[0344] Step 13:

[0345] Server: Analyzes emotional data sent from the terminal to determine the user's emotional state. It determines whether the user is relaxed or stressed.

[0346] Step 14:

[0347] Server: Based on emotional data, it adjusts ingredient selection and menu suggestions. For example, if a user is feeling stressed, it suggests easy-to-make recipes.

[0348] Step 15:

[0349] Server: Adjusts the way alerts are expressed and the timing of notifications based on sentiment data. If the user is feeling stressed, the notification's wording is softened.

[0350] Step 16:

[0351] Device (mobile app): Displays an alert message using gentle language. For example, it might say, "The milk is nearing its expiration date. Please consume it when you have time."

[0352] By following these steps, the management of ingredients in the refrigerator becomes more efficient, and suggestions that take user feelings into consideration are made, resulting in a reduction of food waste and loss.

[0353] (Example 2)

[0354] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0355] Traditional refrigerator food management systems are inefficient and lead to food waste because expiration dates and usage priorities are calculated manually. Furthermore, they fail to improve the user experience because they do not consider the user's emotional state when making suggestions. Additionally, the uniform timing and expression of alerts and notifications can increase user stress. To solve these problems, a system is needed that automates food management and provides suggestions based on the user's emotional state.

[0356] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0357] In this invention, the server includes means for acquiring video data from inside the refrigerator and recognizing ingredients, means for calculating usage priority based on the expiration date of the recognized ingredients, means for generating alerts based on usage priority, means for acquiring current ingredient data from a database and automatically generating available menus, means for creating a shopping list based on the generated menus, means for analyzing sentiment data collected from the terminal and adjusting ingredient selection and menu suggestions based on that information, and means for adjusting alert messages and notification timing based on sentiment data. This enables more efficient ingredient management and suggestions that take into account the user's emotions.

[0358] "Video data" refers to image information acquired from cameras or other imaging devices inside a refrigerator.

[0359] "Ingredients" refers to food and materials stored in the refrigerator.

[0360] "Means of recognition" refers to technologies and devices used to identify food ingredients by analyzing image data.

[0361] "Best before date" refers to the period during which food can maintain its appropriate quality.

[0362] "Usage priority" refers to an indicator used to calculate the priority of which ingredients should be used based on their expiration dates and current usage.

[0363] An "alert" refers to a warning message that is sent to the user based on usage priority.

[0364] A "database" refers to a storage system used to manage information such as ingredient details and recipe information.

[0365] A "menu" refers to a list of dishes that are automatically generated based on the available ingredients.

[0366] A "shopping list" refers to a list of missing ingredients identified based on the generated meal plan.

[0367] "Terminal" refers to a device, such as a mobile device, that a user uses to input and confirm information.

[0368] "Emotional data" refers to data that represents the user's emotional state, and includes information obtained from cameras, microphones, sensors, and other sources.

[0369] "Means of adjusting suggestions" refers to technologies and devices that suggest ingredients and menus suitable for the user based on emotional data.

[0370] "Notification timing" refers to the timing at which alerts and notifications are sent to users.

[0371] This invention is a system that uses generative AI and an emotion engine to manage food items in a refrigerator and provide suggestions that take the user's emotions into consideration. The system acquires video data from inside the refrigerator, recognizes food items, calculates usage priority based on their expiration dates, and generates alerts. It also retrieves current food data from a database, automatically generates available menus, and creates necessary shopping lists. This system works in conjunction with mobile devices, allowing users to check the status of food items, alerts, menus, and shopping lists. Furthermore, it recognizes the user's emotions and uses that information to improve food selection and menu suggestions.

[0372] As a concrete measure, the server activates the camera inside the refrigerator to acquire the latest video data. The acquired video data is sent to an image recognition engine such as Google Cloud Vision API to identify the food items. Based on this identification result, the food items are labeled, and the expiration date information is read, and all of this information is stored in a database (e.g., MySQL). For example, if the acquired video data recognizes milk as "milk" and the expiration date is "2023-10-12", it will be stored in the database in the format "Milk: Expiration Date 2023-10-12".

[0373] The server retrieves all ingredient data from the database, compares the current date with the expiration date of each ingredient, and calculates the usage priority. For ingredients with high usage priority, it generates an alert message and sends it to the mobile device. For example, if the current date is "2023-10-10" and the expiration date of the milk is "2023-10-12", it generates an alert message saying "The milk is nearing its expiration date. Please consume it as soon as possible," and notifies the user.

[0374] Furthermore, the server retrieves current ingredient data from the database and automatically generates menus based on the available ingredients. It identifies recipes that prioritize the use of high-priority ingredients and generates a list of them. For example, if the refrigerator contains "milk, eggs, and spinach," it will search for and preferentially suggest a "cream pasta" recipe.

[0375] The terminal displays detailed instructions for preparing a recipe once the user selects their desired recipe. The server also identifies any missing ingredients based on the selected recipe, automatically generates a shopping list, and sends it to the user. For example, if the user selects "cream pasta" and there is no pasta in the refrigerator, the server automatically adds "pasta" to the shopping list and notifies the user with the message, "Please purchase pasta (200g)."

[0376] As an emotion engine, the device uses its built-in camera and microphone to collect user emotion data. For example, it uses Microsoft® Azure® Emotion API to analyze the user's facial expressions and determine signs of smiling or stress. The results are sent to a server, and suggestions are adjusted based on the analyzed emotion data. For example, if the user is relaxed, it might suggest a new challenge recipe, while if they are stressed, it might suggest an easy recipe.

[0377] In this way, the management of ingredients in the refrigerator becomes more efficient, and suggestions that take user feelings into consideration are also made, resulting in a reduction of food waste and loss.

[0378] Examples of prompts to input into a generative AI model:

[0379] "Based on the data of the ingredients in the refrigerator, please suggest recipes that can be made using the ingredients with the highest priority for use. Also, if the user is relaxed, please suggest a new challenge recipe."

[0380] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0381] Step 1:

[0382] server:

[0383] The system activates the camera inside the refrigerator and acquires the latest video data. This acquired video data is sent to an image recognition engine to recognize the food items. For example, the Google Cloud Vision API is used to identify and label each food item in the video. The input to this process is video data from the camera, and the output is the label information of the recognized food items.

[0384] Specific actions:

[0385] Activate camera → Acquire video data → Send to image recognition engine → Acquire food label information

[0386] Step 2:

[0387] server:

[0388] The system reads the expiration date based on the label information of the recognized food items. For example, it extracts the expiration date by applying text recognition technology to each label. The read expiration date information and food item label information are stored in a database. The input to this process is the label information of the recognized food items, and the output is the food items and their expiration date information stored in the database.

[0389] Specific actions:

[0390] Check label information → Apply text recognition technology → Extract expiration date → Save food labels and expiration dates to a database

[0391] Step 3:

[0392] server:

[0393] All ingredient data is retrieved from the database, and the current date is compared with the expiration date of each ingredient to calculate its usage priority. For example, a Python script can be used to calculate the difference between the current date and each expiration date to determine the usage priority. The input to this process is the ingredient information from the database and the current date, and the output is the ingredient data with the calculated usage priority.

[0394] Specific actions:

[0395] Retrieve ingredient data from the database → Get the current date → Compare with the expiration date → Calculate usage priority

[0396] Step 4:

[0397] server:

[0398] This system determines which ingredients should be alerted based on their usage priority and generates an alert message. For example, for ingredients with a high usage priority, it might generate a message such as, "The milk is nearing its expiration date. Please consume it as soon as possible." The input to this process is ingredient data with calculated usage priority, and the output is the generated alert message.

[0399] Specific actions:

[0400] Identify ingredients with high usage priority → Determine which items require an alert → Generate an alert message

[0401] Step 5:

[0402] Device (mobile app):

[0403] This process receives alert messages sent from the server and displays them in the user interface. Users can then view these alert messages. The input to this process is the alert message from the server, and the output is the alert display to the user.

[0404] Specific actions:

[0405] Receive an alert message → Display the alert in the user interface

[0406] Step 6:

[0407] server:

[0408] The system retrieves current ingredient data from the database and automatically generates menus based on the available ingredients. For example, it uses a cloud-based recipe API to search for recipes that prioritize the use of high-priority ingredients. The input to this process is ingredient data from the database, and the output is a generated menu list.

[0409] Specific actions:

[0410] Retrieve ingredient data from the database → Send a request to the recipe API → Retrieve recipes → Generate a menu list

[0411] Step 7:

[0412] Device (mobile app):

[0413] The system displays a list of menus received from the server in the user interface. When the user selects a desired recipe, detailed instructions for its preparation are displayed. The input to this process is menu data from the server, and the output is the displayed detailed preparation instructions.

[0414] Specific actions:

[0415] Receive menu list → Display in user interface → Display detailed instructions

[0416] Step 8:

[0417] server:

[0418] Based on the current ingredient data and the user's selected recipe, the system identifies missing ingredients and automatically generates a shopping list. For example, a Python script can be used to list ingredients not present in the refrigerator and generate a shopping list of the necessary items. The input to this process is the current ingredient data and the selected recipe, and the output is a shopping list of the missing ingredients.

[0419] Specific actions:

[0420] Retrieve ingredient data and recipe information → Identify missing ingredients → Generate a shopping list

[0421] Step 9:

[0422] User:

[0423] The user checks their shopping list on a mobile app, selects the ingredients they plan to buy, and completes the list. The input for this process is a shopping list sent from the server, and the output is the final shopping list completed by the user.

[0424] Specific actions:

[0425] Review your shopping list → Select the ingredients you plan to buy → Complete the list

[0426] Step 10:

[0427] Device (mobile device):

[0428] The system uses its built-in camera and microphone to collect emotional data such as the user's facial expressions and voice tone. For example, it uses the Emotion API to analyze the user's facial expressions and determine signs of smiles or stress. The input to this process is data from the camera and microphone, and the output is the analyzed emotional data.

[0429] Specific actions:

[0430] Activate camera / microphone → Acquire video / audio data → Send to Emotion API → Analyze emotion data

[0431] Step 11:

[0432] server:

[0433] Based on analyzed emotional data, the system adjusts ingredient selection and menu suggestions. For example, if the user is relaxed, it suggests a new, challenging recipe; if they are stressed, it suggests an easy-to-make recipe. The input to this process is analyzed emotional data, and the output is the adjusted recipe suggestion.

[0434] Specific actions:

[0435] Acquire emotional data → Adjust ingredient selection and menu suggestions → Send suggestions

[0436] Step 12:

[0437] server:

[0438] Based on emotional data, alert messages and notification timing are adjusted. For example, if a user is stressed, notifications might be softened or delayed. The input to this process is emotional data, and the output is an adjusted alert message.

[0439] Specific actions:

[0440] Analyze emotional data → Adjust alert messages and notification timing → Send alert messages

[0441] (Application Example 2)

[0442] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0443] Traditional refrigerator food management systems only managed expiration dates and usage priorities, failing to consider user emotions or provide emotionally-based suggestions. This resulted in inefficient food utilization and decreased user satisfaction. In particular, in retail environments, it was difficult for staff to efficiently manage ingredients while maintaining a stress-free work environment.

[0444] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0445] In this invention, the server includes means for acquiring image data from inside the refrigerator and recognizing ingredients; means for calculating usage priority based on the expiration date of the recognized ingredients; means for generating alerts based on usage priority; means for acquiring current ingredient data from a database and automatically generating available menus; means for creating a shopping list based on the generated menus; means for collecting ingredient data from inside the store in cooperation with image recognition technology and storing it in a database; means for collecting user emotion data using emotion recognition technology and analyzing it with a data analysis engine; and means for adjusting ingredient usage priority and menu suggestions based on the analysis results. This enables efficient management of ingredients and suggestions that take into account the user's emotions.

[0446] "A means of acquiring image data from inside a refrigerator and recognizing food items" refers to a technology that uses a camera inside the refrigerator to acquire images and automatically recognizes the type of food item from those images.

[0447] A "method for calculating usage priority based on expiration dates" is an algorithm that calculates the priority of consumption based on the expiration date information of food ingredients and determines the optimal order of use.

[0448] "A means of generating alerts based on usage priority" refers to a system that notifies or warns users about ingredients with high usage priority.

[0449] "A means of retrieving current ingredient data from a database and automatically generating available menus" refers to a system that automatically generates currently available menus and recipes based on ingredient information stored in a database.

[0450] "A means of creating a shopping list based on a generated menu" refers to a technology that identifies missing ingredients from a generated menu or recipe and automatically creates a shopping list of the necessary items.

[0451] "A method for collecting food ingredient data inside a store in conjunction with image recognition technology and storing it in a database" refers to a system that acquires images of food ingredients through cameras installed inside the store, analyzes them, and stores the data as food ingredient information in a database.

[0452] "Methods for collecting user emotional data using emotion recognition technology and analyzing it with a data analysis engine" refer to technologies that use devices such as cameras and microphones to analyze the user's facial expressions and tone of voice and estimate their emotional state.

[0453] "A means of adjusting the priority of ingredient use and menu suggestions based on analysis results" refers to a system that adjusts the priority of ingredient use and menu suggestions according to the user's current emotional state, based on collected emotional data.

[0454] This invention relates to a refrigerator food management system that provides suggestions while considering the user's emotional state. The system acquires image data from inside the refrigerator, recognizes food items, calculates usage priority based on their expiration dates, and generates alerts. Furthermore, it retrieves current food data from a database, automatically generates available menus, and creates necessary shopping lists. It also uses emotion recognition technology to collect and analyze user emotional data, adjusting menu suggestions and alert notifications based on the results.

[0455] Hardware and software configuration

[0456] The server acquires image data from a camera inside the refrigerator (e.g., Logitech HD Pro Webcam). It uses an image recognition engine (e.g., OpenCV) to recognize the food items and stores the expiration date information of the recognized items in a database (e.g., MySQL). It retrieves all food item data from the database, calculates usage priority, and generates alerts for high-priority items. It also automatically generates menus from a recipe database based on the food item data and adds missing ingredients to the shopping list. For emotion recognition technology, it uses an engine that performs facial expression analysis and voice analysis (e.g., Microsoft Azure Emotion API) to analyze emotional data collected from the user. Based on the analysis results, the system adjusts the suggestions and notifications according to the user's emotional state.

[0457] example:

[0458] The server analyzes image data acquired by a camera inside the refrigerator and recognizes that the expiration date for the food item "milk" is two days away (October 12, 2023). Because the expiration date is approaching, it is determined that it has a high priority for use and generates an alert saying, "The milk is about to expire. Please consume it as soon as possible." Furthermore, it automatically generates a menu based on the current food data and suggests a "cream pasta" recipe to the user. If the user selects "cream pasta," the missing "pasta" is automatically added to the shopping list. In addition, if the emotion recognition engine detects that the user is experiencing stress, the notification wording is softened, for example, to "The milk is about to expire. Please consume it when you have time."

[0459] Specific example

[0460] Example of a prompt:

[0461] Design a system that manages the contents of a refrigerator and provides emotionally sensitive suggestions. The system will capture images of ingredients using a camera and identify them using an image recognition engine. It will then calculate usage priority based on expiration dates and generate alerts as needed. Additionally, it will offer menu suggestions that take user emotions into consideration.

[0462] Through these specific embodiments, the present invention can achieve both efficient food management and proposals that take into account the emotional state of the user, thereby reducing food waste and improving user satisfaction.

[0463] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0464] Step 1:

[0465] Acquisition of image data and food recognition inside the refrigerator

[0466] The server retrieves the latest image data from a camera installed inside the refrigerator.

[0467] Input: Image data from inside the refrigerator

[0468] Processing: Use an image recognition engine (e.g., OpenCV) to identify ingredients and label each ingredient.

[0469] Output: Data on recognized food items (e.g., milk, eggs, spinach)

[0470] Step 2:

[0471] Reading and saving expiration date information of food ingredients to a database.

[0472] The server reads the expiration date of the recognized food item from the image data and saves that information to the database.

[0473] Input: Data on recognized food ingredients and expiration date information

[0474] Processing: Store the expiration dates of ingredients in a database (e.g., MySQL).

[0475] Output: Food ingredient data stored in the database (Example: Milk: Expiration date 2023-10-12)

[0476] Step 3:

[0477] Calculation of usage priority and generation of alerts

[0478] The server retrieves all ingredient data from the database, compares the current date with the expiration date, and calculates the usage priority. It then generates alerts for ingredients with high usage priority.

[0479] Input: Ingredient data retrieved from the database and the current date

[0480] Processing: Prioritize food items nearing their expiration date and create alert messages for those items.

[0481] Output: Alert message (Example: The milk is nearing its expiration date. Please consume it as soon as possible.)

[0482] Step 4:

[0483] Automatic menu generation

[0484] The server retrieves current ingredient data from the database and automatically generates available menus from the recipe database.

[0485] Input: Food ingredient data retrieved from the database

[0486] Process: Search the recipe database and identify possible menus based on currently available ingredients.

[0487] Output: Automatically generated menu list (Example: Cream Pasta)

[0488] Step 5:

[0489] Automatically generate shopping lists

[0490] The server adds any missing ingredients to the shopping list based on the generated menu.

[0491] Input: Automatically generated menu and current ingredient data

[0492] Process: Identify missing ingredients and create a shopping list.

[0493] Output: Shopping list (Example: Please buy 200g of pasta)

[0494] Step 6:

[0495] Collection and analysis of emotional data

[0496] The device (mobile device) analyzes the user's facial expressions and voice tone through its built-in camera and sensors, and collects emotional data.

[0497] Input: User's facial expressions and tone of voice

[0498] Processing: Analyze emotion data using an emotion recognition engine (e.g., Microsoft Azure Emotion API).

[0499] Output: Analyzed sentiment data (e.g., User is relaxed)

[0500] Step 7:

[0501] Adjusting suggestions based on sentiment data

[0502] Based on the analysis results, the server adjusts the priority of ingredient usage, menu suggestions, and alert notifications.

[0503] Input: Analyzed sentiment data

[0504] Processing: Adjust notification wording and timing according to emotional state (e.g., change to a gentler expression).

[0505] Output: Customized notification message (e.g., "Your milk is nearing its expiration date. Please consume it when you have time.")

[0506] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0507] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include those described above. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions shown by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0508] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0509] [Second Embodiment]

[0510] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0511] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0512] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0513] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0514] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0515] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0516] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0517] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0518] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0519] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0520] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0521] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0522] This invention is a system that uses generative AI to manage food in a refrigerator and ensure waste-free consumption. The system acquires image data from the refrigerator, recognizes food items, calculates usage priority based on their expiration dates, and generates alerts. Furthermore, it can retrieve current food data from a database, automatically generate available menus, and create necessary shopping lists. This system also integrates with mobile devices, allowing users to check food status, alerts, menus, and shopping lists.

[0523] Specific processing flow of the system

[0524] 1. Collection and management of food ingredient data

[0525] server:

[0526] The camera inside the refrigerator is activated to acquire the latest video data. This data is then sent to an image recognition engine. The image recognition engine identifies and labels the food items (e.g., "milk," "eggs," "spinach," etc.). Furthermore, it reads the expiration date information of the food items and stores this recognized food information in a database.

[0527] Specific example:

[0528] The refrigerator camera captures images, and the image recognition engine recognizes the milk as "milk." If the expiration date is "2023-10-12," that information is saved in the database in the format "Milk: Expiration Date 2023-10-12."

[0529] 2. Calculation of ingredient usage priority and generation of alerts

[0530] server:

[0531] Retrieve all ingredient data from the database. Compare the current date with the expiration date of each ingredient and calculate its usage priority (urgency of consumption). Determine whether ingredients with high usage priority should be alerted (e.g., within 3 days of expiration). Generate an alert message for ingredients that should be alerted.

[0532] Device (mobile app):

[0533] The app receives alert messages from the server and displays them on the app.

[0534] Specific example:

[0535] The server retrieves data for "Milk: Expiration Date 2023-10-12" and confirms that the current date is "2023-10-10". Since the expiration date is in two days, it is determined to be an alert item. It generates the message "Your milk is nearing its expiration date. Please consume it as soon as possible" and sends it to the mobile app.

[0536] 3. Automatic generation of culinary menus

[0537] server:

[0538] The system retrieves current ingredient data from the database and searches the recipe database for menus that can be created based on the available ingredients. It identifies recipes that prioritize the use of high-priority ingredients and generates a list.

[0539] Device (mobile app):

[0540] The system receives data from the server and displays a list of available recipes. If the user selects a recipe, it displays detailed instructions for preparing it.

[0541] Specific example:

[0542] The server recognizes from the database that it has "milk, eggs, and spinach," and searches the recipe database for a "cream pasta" recipe. Since milk is a high-priority ingredient, this recipe is suggested first, and detailed instructions are displayed in the mobile app.

[0543] 4. Automatic generation of necessary shopping lists

[0544] server:

[0545] Based on current ingredient data and the user's desired recipe, the system identifies missing ingredients. It then automatically generates a shopping list and sends it to the user.

[0546] User:

[0547] Check your shopping list on the mobile app. Select the ingredients you plan to buy and complete the list.

[0548] Specific example:

[0549] The server checks if there is no pasta in the refrigerator based on the user's requested "cream pasta" recipe. It automatically adds "pasta" to the shopping list and displays "Please purchase pasta (200g)" in the mobile app.

[0550] This system makes it easier for users to manage their ingredients and effectively reduce unnecessary expenses and food waste.

[0551] The following describes the processing flow.

[0552] Step 1:

[0553] Server: Activates the camera inside the refrigerator and acquires the latest video data. Sends the video data to the image recognition engine.

[0554] Step 2:

[0555] Server: The image recognition engine analyzes the video data and identifies the food ingredients. The identified food ingredients are then labeled (e.g., "milk," "egg," "spinach," etc.).

[0556] Step 3:

[0557] Server: Reads the expiration date information of the recognized food item. Saves the read food item information and expiration date to the database.

[0558] Step 4:

[0559] Server: Retrieves all ingredient data from the database. Compares the current date with the expiration date of each ingredient and calculates the usage priority.

[0560] Step 5:

[0561] Server: Determines whether high-priority ingredients should be subject to an alert. Generates an alert message for ingredients that should trigger an alert.

[0562] Step 6:

[0563] Device (mobile app): Receives alert messages from the server. Displays alert messages on the app.

[0564] Step 7:

[0565] Server: Retrieves current ingredient data from the database. Searches the recipe database for possible menus based on available ingredients.

[0566] Step 8:

[0567] Server: Identifies recipes that prioritize the use of high-priority ingredients and generates a list of them.

[0568] Step 9:

[0569] Terminal (mobile app): Displays a list of available recipes received from the server. If the user selects a recipe, it displays detailed instructions for preparing it.

[0570] Step 10:

[0571] Server: Based on the current ingredient data and the recipe selected by the user, it identifies any missing ingredients. It automatically generates a shopping list and sends it to the user.

[0572] Step 11:

[0573] User: Check the shopping list on the mobile app. Select the ingredients you plan to buy and complete the list.

[0574] Step 12:

[0575] User: Based on the shopping list, purchase the necessary ingredients and use them without waste.

[0576] (Example 1)

[0577] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0578] Traditional refrigerator food management systems rely on manual processes for managing expiration dates and suggesting menus, leading to food waste and loss. Furthermore, users face the challenge of needing to prioritize ingredients and create shopping lists, which requires considerable time and effort. Therefore, there was a need for an automated refrigerator food management system.

[0579] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0580] In this invention, the server includes means for acquiring image data from inside the refrigerator and recognizing food items, means for calculating usage priority based on the expiration dates of the recognized food items, means for generating alerts based on the usage priority, and means for acquiring current food item data from a database and automatically generating available menus. This enables detailed management and efficient use of food items inside the refrigerator.

[0581] "Image data from inside the refrigerator" refers to video information acquired by a camera inside the refrigerator, which is used for recognizing and managing food items.

[0582] "Food ingredients" refers to the food and beverages stored in the refrigerator, and specifically includes items such as milk, eggs, and vegetables.

[0583] "Means of recognition" refers to the process of identifying and labeling food ingredients using image recognition technology.

[0584] The "best before date" refers to the date by which food is expected to maintain its quality; after this date, the likelihood of quality deterioration increases.

[0585] A "means for calculating usage priority" is an algorithm that determines which ingredients should be used preferentially based on their expiration dates and other conditions.

[0586] "Means for generating alerts" refers to a function that creates notifications to inform users of the need to consume high-priority ingredients.

[0587] A "database" is a system that centrally manages digital data, including information on ingredients and recipes.

[0588] "Available menus" refers to a list of dishes that can be made using the ingredients currently in the refrigerator.

[0589] "Methods for automatically generating menus" refer to algorithms that use ingredient information in a database to search for and select the most suitable cooking recipes.

[0590] "A means of creating a shopping list" refers to a function that allows users to list the ingredients they need and prevent them from forgetting to buy them.

[0591] A "mobile device" refers to a communication terminal that a user can carry with them, and includes smartphones and tablets.

[0592] "Image recognition technology" is a technique that analyzes image data acquired by a camera to identify specific objects or features.

[0593] A "generative AI model" refers to an algorithm that uses AI to learn from data and perform a specific task.

[0594] This invention is a system that uses generative AI to manage food in a refrigerator and consume it efficiently without waste. This system mainly consists of the following elements:

[0595] 1. Collection and management of food ingredient data

[0596] The server periodically acquires image data from inside the refrigerator using a high-resolution camera. This video data is then used with image recognition technologies such as TensorFlow and OpenCV to recognize and label food items. The information on the recognized food items and their expiration dates are then stored in a database. For example, if the camera inside the refrigerator recognizes "milk" and its expiration date is "2023-10-12", this information is saved in the database as "Milk: Expiration Date 2023-10-12".

[0597] 2. Calculation of ingredient usage priority and generation of alerts.

[0598] The server retrieves all food data from the database and compares the current date with the expiration date of each food item. Based on this, it calculates the priority of use. For foods with a high priority, it generates an alert message. This alert message is used to notify the user of the need to consume food items that are nearing their expiration date. For example, if the data for "Milk: Expiration Date 2023-10-12" reveals that the expiration date is in two days, it generates an alert message saying, "The milk is about to expire. Please consume it as soon as possible," and sends it to the mobile device.

[0599] 3. Automatic generation of culinary menus

[0600] The server retrieves the current ingredient data and searches the recipe database for available menus. It identifies recipes that prioritize the use of high-priority ingredients and generates a list. This list is sent to the mobile device, allowing the user to view detailed preparation instructions. For example, if the current ingredient data is "milk, eggs, spinach," the server searches for a "cream pasta" recipe and suggests it to the user.

[0601] 4. Automatic generation of necessary shopping lists

[0602] The server identifies missing ingredients based on the current ingredient data and the user's desired recipe, and automatically generates a shopping list of the necessary items. For example, if the user wants to make "cream pasta," the server will confirm that pasta is missing and add "Please purchase pasta (200g)" to the shopping list.

[0603] This system allows users to easily understand the status of ingredients and manage them effectively based on usage priority, preventing waste. Furthermore, by integrating with mobile devices, ingredients can be managed from anywhere, contributing to a reduction in forgotten purchases and food waste.

[0604] Example of a prompt:

[0605] "Please share a recipe using the milk, spinach, and eggs I have in my refrigerator. Also, please let me know if there are any other ingredients I might need."

[0606] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0607] Step 1:

[0608] Obtain image data from inside the refrigerator.

[0609] The server periodically activates a camera installed inside the refrigerator to acquire image data from within. This time, the video is acquired daily at 8:00 AM.

[0610] Input: Trigger for video acquisition (regular schedule)

[0611] Data processing / data calculation: Activate the camera and capture the internal video data.

[0612] Output: High-resolution image data

[0613] Specific actions:

[0614] The server activates the refrigerator's camera based on a schedule and captures the current state of its interior as video.

[0615] Step 2:

[0616] Food identification using an image recognition engine

[0617] The server sends the acquired high-resolution image data to an image recognition engine to identify the food ingredients. For example, it might run a model using TensorFlow.

[0618] Input: High-resolution image data

[0619] Data processing / data calculation: Identify and label food ingredients using image recognition engines such as TensorFlow.

[0620] Output: Food labels and recognition data (e.g., milk, eggs, spinach)

[0621] Specific actions:

[0622] The server sends image data to TensorFlow, which runs the model to identify the ingredients as "milk," "eggs," and "spinach." It then generates information with labels for these ingredients.

[0623] Step 3:

[0624] Reading and saving the expiration date to the database.

[0625] The server reads the expiration date of the identified food item and saves that information to the database.

[0626] Input: Food labels and recognition data

[0627] Data processing / data calculation: Read the expiration date on food packaging using OCR technology.

[0628] Output: Food ingredients and their expiration date information (Example: Milk: Expiration date 2023-10-12)

[0629] Specific actions:

[0630] The server uses OCR technology to read the expiration date from the package image of the identified food item and saves this information to the database in the format "Milk: Expiration Date 2023-10-12".

[0631] Step 4:

[0632] Calculation of Usage Priority

[0633] The server retrieves all ingredient data from the database and calculates usage priority based on expiration dates.

[0634] Input: Food ingredients and expiration date information retrieved from the database.

[0635] Data processing / calculation: Compare the current date with the expiration date and calculate the usage priority.

[0636] Output: Ingredient information with set usage priority.

[0637] Specific actions:

[0638] The server retrieves data for "Milk: Expiration Date 2023-10-12" and compares it to the current date (e.g., 2023-10-10). Since the expiration date is 2 days away, it is given a high priority for use.

[0639] Step 5:

[0640] Alert generation

[0641] The server generates alert messages for ingredients with high usage priority.

[0642] Input: Ingredient information with set usage priority.

[0643] Data processing / data calculation: Generate alerts based on usage priority.

[0644] Output: Alert message

[0645] Specific actions:

[0646] The server generates an alert message saying, "The milk is nearing its expiration date. Please consume it as soon as possible," and sends it to the mobile device.

[0647] Step 6:

[0648] Automatic menu generation

[0649] The server retrieves current ingredient data and searches and generates available menus from the recipe database.

[0650] Input: Current ingredient data retrieved from the database

[0651] Data processing / data calculation: Search the recipe database for possible menus based on ingredient data and generate a list.

[0652] Output: List of culinary options

[0653] Specific actions:

[0654] The server retrieves data for "milk, eggs, and spinach," searches the recipe database for "cream pasta" recipes, generates them, and adds them to the list.

[0655] Step 7:

[0656] Displaying recipes and detailed instructions

[0657] The device (mobile app) receives data from the server and displays it so that the user can view detailed creation instructions.

[0658] Input: List of possible menu items

[0659] Data processing / data calculation: Format data from the server for display.

[0660] Output: Menu list and detailed instructions

[0661] Specific actions:

[0662] The mobile app displays detailed instructions for users who select a "cream pasta" recipe. These instructions include steps such as, "First, prepare the milk, then add the eggs and mix."

[0663] Step 8:

[0664] Automatically generate shopping lists

[0665] The server identifies any missing ingredients based on the recipe selected by the user and generates a shopping list.

[0666] Input: Selected recipe and current ingredient data

[0667] Data processing / data calculation: Identify missing ingredients and generate a shopping list.

[0668] Output: Shopping list

[0669] Specific actions:

[0670] The server compares the "cream pasta" recipe with the current ingredient data and confirms that pasta is missing. It automatically adds "Please purchase pasta (200g)" to the shopping list.

[0671] Step 9:

[0672] Review and edit your shopping list

[0673] Users can view their shopping list on the mobile app and add or edit items they plan to buy as needed.

[0674] Input: Shopping list

[0675] Data processing / data calculation: Operation data when a user edits a list.

[0676] Output: Updated shopping list

[0677] Specific actions:

[0678] The user opens the mobile app, sees the message "Please purchase pasta (200g)," and adds any other items they need to the list.

[0679] (Application Example 1)

[0680] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0681] There is a need for methods to effectively manage the ingredients in refrigerators, reduce food waste due to expired food, and enable users to utilize ingredients efficiently. Furthermore, an efficient system is needed to allow users to easily obtain recipes and quickly replenish missing ingredients. This is expected to save time and improve consumer behavior.

[0682] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0683] In this invention, the server includes means for acquiring image data from inside the refrigerator and recognizing ingredients; means for calculating usage priority based on the expiration date of the recognized ingredients; means for generating alerts based on the usage priority; means for acquiring current ingredient data from a database and automatically generating available menus; means for creating a shopping list based on the generated menus; means for suggesting recipes using available ingredients based on the user's refrigerator information; and means for coordinating with a food delivery service that automatically orders any missing ingredients based on the suggested recipes. This enables efficient use and rapid replenishment of ingredients.

[0684] "Image data" refers to data that represents visual information in a digital format.

[0685] "Ingredients" refer to food products intended for cooking or consumption.

[0686] "Recognition" means identifying and understanding a specific object.

[0687] The "best before date" indicates the period during which food can be eaten safely and deliciously.

[0688] "Usage priority" refers to the order in which multiple ingredients should be used first.

[0689] An "alert" is a warning message that notifies the user.

[0690] A "database" is a digital system that organizes, stores, and manages related data.

[0691] A "menu" refers to the composition and plan of the dishes that will be served as meals within a certain period of time.

[0692] A "shopping list" is a list of items that need to be purchased.

[0693] "Mobile devices" is a general term for portable electronic devices.

[0694] A "recipe" is a set of instructions that shows how to prepare food and the steps involved.

[0695] A "food delivery service" is a service that delivers ordered food to a specified location.

[0696] This invention is a system for efficiently managing food items in a refrigerator and consuming them without waste. This system is realized through the collaboration of a server and a mobile device. The server manages food data, while the mobile device functions as an interface with the user.

[0697] Collection and management of food ingredient data

[0698] Server: The server activates a camera installed inside the refrigerator and acquires the latest image data. The acquired image data is used to identify and label food items using an image recognition engine such as TensorFlow or OpenCV. In addition, the expiration date is automatically read, and this information is stored in the food database.

[0699] Specific example: An image of milk is recognized, and along with labeling, the expiration date "2023-10-12" is added.

[0700] Calculation of ingredient usage priority and generation of alerts

[0701] Server: The server retrieves all ingredient data from the database and calculates usage priority by comparing the current date with the expiration date of each ingredient. It generates alerts for ingredients nearing their expiration date and notifies the user's mobile device.

[0702] Specific example: The server retrieves data for "Milk: Expiration Date 2023-10-12," verifies that the current date is "2023-10-10," and then generates a warning message: "The milk is nearing its expiration date."

[0703] Automatic generation of cookable menus

[0704] Server: The server retrieves current ingredient data from the database and automatically generates menus based on the available ingredients from the recipe database. Recipes using high-priority ingredients are also suggested.

[0705] Specific example: A recipe for "cream pasta" is suggested based on the ingredients in the refrigerator (milk, eggs, spinach).

[0706] Automatically generate a shopping list of necessary items.

[0707] Server: The server identifies missing ingredients based on the recipe selected by the user and the current ingredient data, and automatically generates a shopping list. Furthermore, it sends this list to the user's mobile device.

[0708] Specific example: A user selects a "cream pasta" recipe, and since there is no pasta in the refrigerator, "pasta (200g)" is automatically added to the shopping list.

[0709] Collaboration with food delivery

[0710] Server: The server also has a function to automatically coordinate with food delivery services and order any missing ingredients. This allows users to quickly obtain the ingredients they need.

[0711] Specific example: Based on a shopping list, automatically order "pasta (200g)" that is missing from the refrigerator using a food delivery service.

[0712] Example of a prompt:

[0713] "Create a program that retrieves information about the ingredients in the refrigerator, generates recipes to suggest in a food delivery application based on that information, and automatically orders any missing ingredients from a food delivery service."

[0714] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0715] Step 1:

[0716] The server activates the camera inside the refrigerator and retrieves the latest image data.

[0717] Input: Image data from inside the refrigerator.

[0718] Processing: Image data is acquired from a camera inside the refrigerator and sent to the system. The image recognition engine identifies the food items.

[0719] Output: A list of identified ingredients.

[0720] Step 2:

[0721] The server uses TensorFlow or OpenCV to label the recognized food items and read their expiration dates.

[0722] Input: Image data of food ingredients identified by an image recognition engine.

[0723] Processing: Label food items with labels such as "milk" and "eggs," and extract their expiration date information.

[0724] Output: A list of labeled ingredients and their corresponding expiration dates.

[0725] Step 3:

[0726] The server stores a list of ingredients and expiration date data in a database.

[0727] Input: A list of labeled ingredients and their expiration dates.

[0728] Processing: The process of saving ingredient information to a database.

[0729] Output: Ingredient data stored in the database.

[0730] Step 4:

[0731] The server retrieves all ingredient data from the database, calculates usage priority, and generates a warning message.

[0732] Input: A list of ingredients stored in the database and the expiration date of each ingredient.

[0733] Processing: Compare the current date with the expiration date of each ingredient and calculate the priority for use. Generate a warning message for ingredients that are nearing their expiration date.

[0734] Output: The generated warning message.

[0735] Step 5:

[0736] The server sends a warning message to the mobile device.

[0737] Input: The generated warning message.

[0738] Process: Send a warning message to a mobile device.

[0739] Output: Warning message displayed on the mobile device.

[0740] Step 6:

[0741] The server retrieves current ingredient data from the database and automatically generates usable menus from the recipe database.

[0742] Input: Food ingredient data stored in the database.

[0743] Processing: Based on ingredient data, the system searches for available recipes in the recipe database and automatically generates them.

[0744] Output: Suggested recipe list.

[0745] Step 7:

[0746] The server sends the suggested recipe list to the mobile device and displays it to the user.

[0747] Input: Suggested recipe list.

[0748] Processing: Send the recipe list to your mobile device.

[0749] Output: Recipe list displayed on mobile devices.

[0750] Step 8:

[0751] The user selects their desired recipe on their mobile device, and the server identifies any missing ingredients and automatically generates a shopping list.

[0752] Input: User-selected recipe, current ingredient data.

[0753] Processing: Identifies missing ingredients and automatically generates a shopping list.

[0754] Output: Auto-generated shopping list.

[0755] Step 9:

[0756] The server uses an automatically generated shopping list to place orders with a food delivery service.

[0757] Input: Auto-generated shopping list.

[0758] Processing: Send order data to the food delivery service.

[0759] Output: Order confirmation for food delivery service.

[0760] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0761] This invention is a system that manages food items in a refrigerator and provides suggestions that take the user's emotions into consideration, utilizing generative AI and an emotion engine. The system acquires image data from inside the refrigerator, recognizes food items, calculates usage priority based on their expiration dates, and generates alerts. It also retrieves current food data from a database, automatically generates available menus, and creates necessary shopping lists. This system works in conjunction with mobile devices, allowing users to check the status of food items, alerts, menus, and shopping lists. Furthermore, it recognizes the user's emotions and uses that information to improve food selection and menu suggestions.

[0762] Specific processing flow of the system

[0763] 1. Collection and management of food ingredient data

[0764] server:

[0765] The camera inside the refrigerator is activated to acquire the latest video data. This data is then sent to an image recognition engine. The image recognition engine identifies and labels the food items (e.g., "milk," "eggs," "spinach," etc.). Furthermore, it reads the expiration date information of the food items and stores this recognized food information in a database.

[0766] Specific example:

[0767] The refrigerator camera captures images, and the image recognition engine recognizes the milk as "milk." If the expiration date is "2023-10-12," that information is saved in the database in the format "Milk: Expiration Date 2023-10-12."

[0768] 2. Calculation of ingredient usage priority and generation of alerts

[0769] server:

[0770] Retrieve all ingredient data from the database. Compare the current date with the expiration date of each ingredient and calculate the usage priority. Determine whether ingredients with high usage priority should be subject to an alert. Generate an alert message for ingredients that should trigger an alert.

[0771] Device (mobile app):

[0772] The app receives alert messages from the server and displays them on the app.

[0773] Specific example:

[0774] The server retrieves data for "Milk: Expiration Date 2023-10-12" and confirms that the current date is "2023-10-10". Since the expiration date is in two days, it is determined to be an alert item. It generates the message "Your milk is nearing its expiration date. Please consume it as soon as possible" and sends it to the mobile app.

[0775] 3. Automatic generation of culinary menus

[0776] server:

[0777] Retrieve the current ingredient data from the database. Search the recipe database for menus that can be created based on the available ingredients. Identify recipes that prioritize the use of high-priority ingredients and generate a list of them.

[0778] Device (mobile app):

[0779] Displays a list of available recipes received from the server. If the user selects a recipe, it displays detailed instructions for preparing it.

[0780] Specific example:

[0781] The server recognizes from the database that it has "milk, eggs, and spinach," and searches the recipe database for a "cream pasta" recipe. Since milk is a high-priority ingredient, this recipe is suggested first, and detailed instructions are displayed in the mobile app.

[0782] 4. Automatic generation of necessary shopping lists

[0783] server:

[0784] Based on current ingredient data and the user's selected recipe, the system identifies missing ingredients. It then automatically generates a shopping list and sends it to the user.

[0785] User:

[0786] Check your shopping list on the mobile app. Select the ingredients you plan to buy and complete the list.

[0787] Specific example:

[0788] The server checks if there is no pasta in the refrigerator based on the user's requested "cream pasta" recipe. It automatically adds "pasta" to the shopping list and displays "Please purchase pasta (200g)" in the mobile app.

[0789] Utilizing the Emotion Engine

[0790] 5. Collection of emotional data

[0791] Device (mobile device):

[0792] The device analyzes the user's facial expressions and voice tone through its built-in camera and sensors, collecting emotional data. For example, it uses the camera to recognize the user's face and determine signs of smiling or stress.

[0793] 6. Analysis and reflection of emotional data

[0794] server:

[0795] The system analyzes emotional data transmitted from the device to determine the user's current emotional state. Based on this information, it adjusts ingredient selection and menu suggestions. For example, if the user is feeling stressed, it prioritizes suggesting easy-to-make recipes and ingredients the user likes.

[0796] Specific example:

[0797] The mobile app's facial recognition feature captures the user's smile and sends a message to the server indicating that "the user is relaxed." The server then determines that "the user is relaxed, so we should suggest a new challenge recipe" and adjusts the suggested recipe accordingly.

[0798] 7. Adjusting alerts and notifications

[0799] server:

[0800] Based on emotional data, the system adjusts how alerts are phrased and when notifications are sent. For example, if a user is stressed, the notification might be softened or delayed.

[0801] Specific example:

[0802] The server determines that "the user is feeling stressed" and sends an alert with softer wording, such as "The milk is nearing its expiration date. Please consume it when you have time."

[0803] Through the processing flow described above, this invention streamlines the management of ingredients in the refrigerator and, by making suggestions that take user feelings into consideration, reduces food waste and loss.

[0804] The following describes the processing flow.

[0805] Step 1:

[0806] Server: Activates the camera inside the refrigerator and acquires the latest video data. Then, it sends the acquired video data to the image recognition engine.

[0807] Step 2:

[0808] Server: The image recognition engine analyzes the video data and identifies the food ingredients. The identified food ingredients are then labeled (e.g., "milk," "egg," "spinach," etc.).

[0809] Step 3:

[0810] Server: Reads the expiration date information of the recognized food item. Saves the read food item information and expiration date to the database.

[0811] Step 4:

[0812] Server: Retrieves all ingredient data from the database. Compares the current date with the expiration date of each ingredient and calculates the usage priority (urgency of consumption).

[0813] Step 5:

[0814] Server: Determines whether high-priority ingredients should be subject to an alert. Generates an alert message for ingredients that should trigger an alert.

[0815] Step 6:

[0816] Device (mobile app): Receives alert messages from the server. Displays alert messages on the app.

[0817] Step 7:

[0818] Server: Retrieves current ingredient data from the database. Searches the recipe database for possible menus based on available ingredients.

[0819] Step 8:

[0820] Server: Identifies recipes that prioritize the use of high-priority ingredients and generates a list of them.

[0821] Step 9:

[0822] Terminal (mobile app): Displays a list of available recipes received from the server. If the user selects a recipe, it displays detailed instructions for preparing it.

[0823] Step 10:

[0824] Server: Based on current ingredient data and the user's selected recipe, it identifies missing ingredients. It automatically generates a shopping list and sends it to the user.

[0825] Step 11:

[0826] User: Check the shopping list on the mobile app. Select the ingredients you plan to buy and complete the list.

[0827] Step 12:

[0828] Mobile device: Analyzes the user's facial expressions and voice tone through its built-in camera and sensors to collect emotional data.

[0829] Step 13:

[0830] Server: Analyzes emotional data sent from the terminal to determine the user's emotional state. It determines whether the user is relaxed or stressed.

[0831] Step 14:

[0832] Server: Based on emotional data, it adjusts ingredient selection and menu suggestions. For example, if a user is feeling stressed, it suggests easy-to-make recipes.

[0833] Step 15:

[0834] Server: Adjusts the way alerts are expressed and the timing of notifications based on sentiment data. If the user is feeling stressed, the notification's wording is softened.

[0835] Step 16:

[0836] Device (mobile app): Displays an alert message using gentle language. For example, it might say, "The milk is nearing its expiration date. Please consume it when you have time."

[0837] By following these steps, the management of ingredients in the refrigerator becomes more efficient, and suggestions that take user feelings into consideration are made, resulting in a reduction of food waste and loss.

[0838] (Example 2)

[0839] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0840] Traditional refrigerator food management systems are inefficient and lead to food waste because expiration dates and usage priorities are calculated manually. Furthermore, they fail to improve the user experience because they do not consider the user's emotional state when making suggestions. Additionally, the uniform timing and expression of alerts and notifications can increase user stress. To solve these problems, a system is needed that automates food management and provides suggestions based on the user's emotional state.

[0841] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0842] In this invention, the server includes means for acquiring video data from inside the refrigerator and recognizing ingredients, means for calculating usage priority based on the expiration date of the recognized ingredients, means for generating alerts based on usage priority, means for acquiring current ingredient data from a database and automatically generating available menus, means for creating a shopping list based on the generated menus, means for analyzing sentiment data collected from the terminal and adjusting ingredient selection and menu suggestions based on that information, and means for adjusting alert messages and notification timing based on sentiment data. This enables more efficient ingredient management and suggestions that take into account the user's emotions.

[0843] "Video data" refers to image information acquired from cameras or other imaging devices inside a refrigerator.

[0844] "Ingredients" refers to food and materials stored in the refrigerator.

[0845] "Means of recognition" refers to technologies and devices used to identify food ingredients by analyzing image data.

[0846] "Best before date" refers to the period during which food can maintain its appropriate quality.

[0847] "Usage priority" refers to an indicator used to calculate the priority of which ingredients should be used based on their expiration dates and current usage.

[0848] An "alert" refers to a warning message that is sent to the user based on usage priority.

[0849] A "database" refers to a storage system used to manage information such as ingredient details and recipe information.

[0850] A "menu" refers to a list of dishes that are automatically generated based on the available ingredients.

[0851] A "shopping list" refers to a list of missing ingredients identified based on the generated meal plan.

[0852] "Terminal" refers to a device, such as a mobile device, that a user uses to input and confirm information.

[0853] "Emotional data" refers to data that represents the user's emotional state, and includes information obtained from cameras, microphones, sensors, and other sources.

[0854] "Means of adjusting suggestions" refers to technologies and devices that suggest ingredients and menus suitable for the user based on emotional data.

[0855] "Notification timing" refers to the timing at which alerts and notifications are sent to users.

[0856] This invention is a system that uses generative AI and an emotion engine to manage food items in a refrigerator and provide suggestions that take the user's emotions into consideration. The system acquires video data from inside the refrigerator, recognizes food items, calculates usage priority based on their expiration dates, and generates alerts. It also retrieves current food data from a database, automatically generates available menus, and creates necessary shopping lists. This system works in conjunction with mobile devices, allowing users to check the status of food items, alerts, menus, and shopping lists. Furthermore, it recognizes the user's emotions and uses that information to improve food selection and menu suggestions.

[0857] As a concrete step, the server activates the camera inside the refrigerator to acquire the latest video data. The acquired video data is sent to an image recognition engine such as the Google Cloud Vision API to identify the food items. Based on this identification result, the food items are labeled, and the expiration date information is read, and all of this information is stored in a database (e.g., MySQL). For example, if the acquired video data recognizes milk as "milk" and the expiration date is "2023-10-12", it will be stored in the database in the format "Milk: Expiration Date 2023-10-12".

[0858] The server retrieves all ingredient data from the database, compares the current date with the expiration date of each ingredient, and calculates the usage priority. For ingredients with high usage priority, it generates an alert message and sends it to the mobile device. For example, if the current date is "2023-10-10" and the expiration date of the milk is "2023-10-12", it generates an alert message saying "The milk is nearing its expiration date. Please consume it as soon as possible," and notifies the user.

[0859] Furthermore, the server retrieves current ingredient data from the database and automatically generates menus based on the available ingredients. It identifies recipes that prioritize the use of high-priority ingredients and generates a list of them. For example, if the refrigerator contains "milk, eggs, and spinach," it will search for and preferentially suggest a "cream pasta" recipe.

[0860] The terminal displays detailed instructions for preparing a recipe once the user selects their desired recipe. The server also identifies any missing ingredients based on the selected recipe, automatically generates a shopping list, and sends it to the user. For example, if the user selects "cream pasta" and there is no pasta in the refrigerator, the server automatically adds "pasta" to the shopping list and notifies the user with the message, "Please purchase pasta (200g)."

[0861] As an emotion engine, the device uses its built-in camera and microphone to collect user emotion data. For example, it uses Microsoft Azure's Emotion API to analyze the user's facial expressions and determine signs of smiling or stress. The results are sent to a server, and suggestions are adjusted based on the analyzed emotion data. For instance, if the user is relaxed, it might suggest a new challenge recipe, while if they are stressed, it might suggest an easy recipe.

[0862] In this way, the management of ingredients in the refrigerator becomes more efficient, and suggestions that take user feelings into consideration are also made, resulting in a reduction of food waste and loss.

[0863] Examples of prompts to input into a generative AI model:

[0864] "Based on the data of the ingredients in the refrigerator, please suggest recipes that can be made using the ingredients with the highest priority for use. Also, if the user is relaxed, please suggest a new challenge recipe."

[0865] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0866] Step 1:

[0867] server:

[0868] The system activates the camera inside the refrigerator and acquires the latest video data. This acquired video data is sent to an image recognition engine to recognize the food items. For example, the Google Cloud Vision API is used to identify and label each food item in the video. The input to this process is video data from the camera, and the output is the label information of the recognized food items.

[0869] Specific actions:

[0870] Activate camera → Acquire video data → Send to image recognition engine → Acquire food label information

[0871] Step 2:

[0872] server:

[0873] The system reads the expiration date based on the label information of the recognized food items. For example, it extracts the expiration date by applying text recognition technology to each label. The read expiration date information and food item label information are stored in a database. The input to this process is the label information of the recognized food items, and the output is the food items and their expiration date information stored in the database.

[0874] Specific actions:

[0875] Check label information → Apply text recognition technology → Extract expiration date → Save food labels and expiration dates to a database

[0876] Step 3:

[0877] server:

[0878] All ingredient data is retrieved from the database, and the current date is compared with the expiration date of each ingredient to calculate its usage priority. For example, a Python script can be used to calculate the difference between the current date and each expiration date to determine the usage priority. The input to this process is the ingredient information from the database and the current date, and the output is the ingredient data with the calculated usage priority.

[0879] Specific actions:

[0880] Retrieve ingredient data from the database → Get the current date → Compare with the expiration date → Calculate usage priority

[0881] Step 4:

[0882] server:

[0883] This system determines which ingredients should be alerted based on their usage priority and generates an alert message. For example, for ingredients with a high usage priority, it might generate a message such as, "The milk is nearing its expiration date. Please consume it as soon as possible." The input to this process is ingredient data with calculated usage priority, and the output is the generated alert message.

[0884] Specific actions:

[0885] Identify ingredients with high usage priority → Determine which items require an alert → Generate an alert message

[0886] Step 5:

[0887] Device (mobile app):

[0888] This process receives alert messages sent from the server and displays them in the user interface. Users can then view these alert messages. The input to this process is the alert message from the server, and the output is the alert display to the user.

[0889] Specific actions:

[0890] Receive an alert message → Display the alert in the user interface

[0891] Step 6:

[0892] server:

[0893] The system retrieves current ingredient data from the database and automatically generates menus based on the available ingredients. For example, it uses a cloud-based recipe API to search for recipes that prioritize the use of high-priority ingredients. The input to this process is ingredient data from the database, and the output is a generated menu list.

[0894] Specific actions:

[0895] Retrieve ingredient data from the database → Send a request to the recipe API → Retrieve recipes → Generate a menu list

[0896] Step 7:

[0897] Device (mobile app):

[0898] The system displays a list of menus received from the server in the user interface. When the user selects a desired recipe, detailed instructions for its preparation are displayed. The input to this process is menu data from the server, and the output is the displayed detailed preparation instructions.

[0899] Specific actions:

[0900] Receive menu list → Display in user interface → Display detailed instructions

[0901] Step 8:

[0902] server:

[0903] Based on the current ingredient data and the user's selected recipe, the system identifies missing ingredients and automatically generates a shopping list. For example, a Python script can be used to list ingredients not present in the refrigerator and generate a shopping list of the necessary items. The input to this process is the current ingredient data and the selected recipe, and the output is a shopping list of the missing ingredients.

[0904] Specific actions:

[0905] Retrieve ingredient data and recipe information → Identify missing ingredients → Generate a shopping list

[0906] Step 9:

[0907] User:

[0908] The user checks their shopping list on a mobile app, selects the ingredients they plan to buy, and completes the list. The input for this process is a shopping list sent from the server, and the output is the final shopping list completed by the user.

[0909] Specific actions:

[0910] Review your shopping list → Select the ingredients you plan to buy → Complete the list

[0911] Step 10:

[0912] Device (mobile device):

[0913] The system uses its built-in camera and microphone to collect emotional data such as the user's facial expressions and voice tone. For example, it uses the Emotion API to analyze the user's facial expressions and determine signs of smiles or stress. The input to this process is data from the camera and microphone, and the output is the analyzed emotional data.

[0914] Specific actions:

[0915] Activate camera / microphone → Acquire video / audio data → Send to Emotion API → Analyze emotion data

[0916] Step 11:

[0917] server:

[0918] Based on analyzed emotional data, the system adjusts ingredient selection and menu suggestions. For example, if the user is relaxed, it suggests a new, challenging recipe; if they are stressed, it suggests an easy-to-make recipe. The input to this process is analyzed emotional data, and the output is the adjusted recipe suggestion.

[0919] Specific actions:

[0920] Acquire emotional data → Adjust ingredient selection and menu suggestions → Send suggestions

[0921] Step 12:

[0922] server:

[0923] Based on emotional data, alert messages and notification timing are adjusted. For example, if a user is stressed, the notification's wording might be softened or the notification timing delayed. The input to this process is emotional data, and the output is the adjusted alert message.

[0924] Specific actions:

[0925] Analyze emotional data → Adjust alert messages and notification timing → Send alert messages

[0926] (Application Example 2)

[0927] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0928] Traditional refrigerator food management systems only managed expiration dates and usage priorities, failing to consider user emotions or provide emotionally-based suggestions. This resulted in inefficient food utilization and decreased user satisfaction. In particular, in retail environments, it was difficult for staff to efficiently manage ingredients while maintaining a stress-free work environment.

[0929] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0930] In this invention, the server includes means for acquiring image data from inside the refrigerator and recognizing ingredients; means for calculating usage priority based on the expiration date of the recognized ingredients; means for generating alerts based on usage priority; means for acquiring current ingredient data from a database and automatically generating available menus; means for creating a shopping list based on the generated menus; means for collecting ingredient data from inside the store in cooperation with image recognition technology and storing it in a database; means for collecting user emotion data using emotion recognition technology and analyzing it with a data analysis engine; and means for adjusting ingredient usage priority and menu suggestions based on the analysis results. This enables efficient management of ingredients and suggestions that take into account the user's emotions.

[0931] "A means of acquiring image data from inside a refrigerator and recognizing food items" refers to a technology that uses a camera inside the refrigerator to acquire images and automatically recognizes the type of food item from those images.

[0932] "A method for calculating usage priority based on expiration dates" refers to an algorithm that calculates the priority of consumption based on the expiration date information of food ingredients and determines the optimal order of use.

[0933] "A means of generating alerts based on usage priority" refers to a system that notifies or warns users about ingredients with high usage priority.

[0934] "A means of retrieving current ingredient data from a database and automatically generating available menus" refers to a system that automatically generates currently available menus and recipes based on ingredient information stored in a database.

[0935] "A means of creating a shopping list based on a generated menu" refers to a technology that identifies missing ingredients from a generated menu or recipe and automatically creates a shopping list of the necessary items.

[0936] "A method for collecting food ingredient data inside a store in conjunction with image recognition technology and storing it in a database" refers to a system that acquires images of food ingredients through cameras installed inside the store, analyzes them, and stores them in a database as food ingredient data.

[0937] "Methods for collecting user emotional data using emotion recognition technology and analyzing it with a data analysis engine" refer to technologies that use devices such as cameras and microphones to analyze the user's facial expressions and tone of voice and estimate their emotional state.

[0938] "A means of adjusting the priority of ingredient use and menu suggestions based on analysis results" refers to a system that adjusts the priority of ingredient use and menu suggestions according to the user's current emotional state, based on collected emotional data.

[0939] This invention relates to a refrigerator food management system that provides suggestions while considering the user's emotional state. The system acquires image data from inside the refrigerator, recognizes food items, calculates usage priority based on their expiration dates, and generates alerts. Furthermore, it retrieves current food data from a database, automatically generates available menus, and creates necessary shopping lists. It also uses emotion recognition technology to collect and analyze user emotional data, adjusting menu suggestions and alert notifications based on the results.

[0940] Hardware and software configuration

[0941] The server acquires image data from a camera inside the refrigerator (e.g., Logitech HD Pro Webcam). It uses an image recognition engine (e.g., OpenCV) to recognize the food items and stores the expiration date information of the recognized items in a database (e.g., MySQL). It retrieves all food item data from the database, calculates usage priority, and generates alerts for high-priority items. It also automatically generates menus from a recipe database based on the food item data and adds missing ingredients to the shopping list. For emotion recognition technology, it uses an engine that performs facial expression analysis and voice analysis (e.g., Microsoft Azure Emotion API) to analyze emotional data collected from the user. Based on the analysis results, the system adjusts the suggestions and notifications according to the user's emotional state.

[0942] example:

[0943] The server analyzes image data acquired by a camera inside the refrigerator and recognizes that the expiration date for the food item "milk" is two days away (October 12, 2023). Because the expiration date is approaching, it is determined that it has a high priority for use and generates an alert saying, "The milk is about to expire. Please consume it as soon as possible." Furthermore, it automatically generates a menu based on the current food data and suggests a "cream pasta" recipe to the user. If the user selects "cream pasta," the missing "pasta" is automatically added to the shopping list. In addition, if the emotion recognition engine detects that the user is experiencing stress, the notification wording is softened, for example, to "The milk is about to expire. Please consume it when you have time."

[0944] Specific example

[0945] Example of a prompt:

[0946] Design a system that manages the contents of a refrigerator and provides emotionally sensitive suggestions. The system will capture images of ingredients using a camera and identify them using an image recognition engine. It will then calculate usage priority based on expiration dates and generate alerts as needed. Additionally, it will offer menu suggestions that take user emotions into consideration.

[0947] Through these specific embodiments, the present invention can achieve both efficient food management and proposals that take into account the emotional state of the user, thereby reducing food waste and improving user satisfaction.

[0948] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0949] Step 1:

[0950] Acquisition of image data and food recognition inside the refrigerator

[0951] The server retrieves the latest image data from a camera installed inside the refrigerator.

[0952] Input: Image data from inside the refrigerator

[0953] Processing: Use an image recognition engine (e.g., OpenCV) to identify ingredients and label each ingredient.

[0954] Output: Data on recognized food items (e.g., milk, eggs, spinach)

[0955] Step 2:

[0956] Reading and saving expiration date information of food ingredients to a database.

[0957] The server reads the expiration date of the recognized food item from the image data and saves that information to the database.

[0958] Input: Data on recognized food ingredients and expiration date information

[0959] Processing: Store the expiration dates of ingredients in a database (e.g., MySQL).

[0960] Output: Food ingredient data stored in the database (Example: Milk: Expiration date 2023-10-12)

[0961] Step 3:

[0962] Calculation of usage priority and generation of alerts

[0963] The server retrieves all ingredient data from the database, compares the current date with the expiration date, and calculates the usage priority. It then generates alerts for ingredients with high usage priority.

[0964] Input: Ingredient data retrieved from the database and the current date

[0965] Processing: Prioritize food items nearing their expiration date and create alert messages for those items.

[0966] Output: Alert message (Example: The milk is nearing its expiration date. Please consume it as soon as possible.)

[0967] Step 4:

[0968] Automatic menu generation

[0969] The server retrieves current ingredient data from the database and automatically generates available menus from the recipe database.

[0970] Input: Food ingredient data retrieved from the database

[0971] Process: Search the recipe database and identify possible menus based on currently available ingredients.

[0972] Output: Automatically generated menu list (Example: Cream Pasta)

[0973] Step 5:

[0974] Automatically generate shopping lists

[0975] The server adds any missing ingredients to the shopping list based on the generated menu.

[0976] Input: Automatically generated menu and current ingredient data

[0977] Process: Identify missing ingredients and create a shopping list.

[0978] Output: Shopping list (Example: Please buy 200g of pasta)

[0979] Step 6:

[0980] Collection and analysis of emotional data

[0981] The device (mobile device) analyzes the user's facial expressions and voice tone through its built-in camera and sensors, and collects emotional data.

[0982] Input: User's facial expressions and tone of voice

[0983] Processing: Analyze emotion data using an emotion recognition engine (e.g., Microsoft Azure Emotion API).

[0984] Output: Analyzed sentiment data (e.g., User is relaxed)

[0985] Step 7:

[0986] Adjusting suggestions based on sentiment data

[0987] Based on the analysis results, the server adjusts the priority of ingredient usage, menu suggestions, and alert notifications.

[0988] Input: Analyzed sentiment data

[0989] Processing: Adjust notification wording and timing according to emotional state (e.g., change to a gentler expression).

[0990] Output: Customized notification message (e.g., "Your milk is nearing its expiration date. Please consume it when you have time.")

[0991] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0992] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include those described above. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions shown by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0993] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0994] [Third Embodiment]

[0995] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0996] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0997] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0998] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0999] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1000] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1001] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1002] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1003] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1004] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1005] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1006] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[1007] This invention is a system that uses generative AI to manage food in a refrigerator and ensure waste-free consumption. The system acquires image data from the refrigerator, recognizes food items, calculates usage priority based on their expiration dates, and generates alerts. Furthermore, it can retrieve current food data from a database, automatically generate available menus, and create necessary shopping lists. This system also integrates with mobile devices, allowing users to check food status, alerts, menus, and shopping lists.

[1008] Specific processing flow of the system

[1009] 1. Collection and management of food ingredient data

[1010] server:

[1011] The camera inside the refrigerator is activated to acquire the latest video data. This data is then sent to an image recognition engine. The image recognition engine identifies and labels the food items (e.g., "milk," "eggs," "spinach," etc.). Furthermore, it reads the expiration date information of the food items and stores this recognized food information in a database.

[1012] Specific example:

[1013] The refrigerator camera captures images, and the image recognition engine recognizes the milk as "milk." If the expiration date is "2023-10-12," that information is saved in the database in the format "Milk: Expiration Date 2023-10-12."

[1014] 2. Calculation of ingredient usage priority and generation of alerts

[1015] server:

[1016] Retrieve all ingredient data from the database. Compare the current date with the expiration date of each ingredient and calculate its usage priority (urgency of consumption). Determine whether ingredients with high usage priority should be alerted (e.g., within 3 days of expiration). Generate an alert message for ingredients that should be alerted.

[1017] Device (mobile app):

[1018] The app receives alert messages from the server and displays them on the app.

[1019] Specific example:

[1020] The server retrieves data for "Milk: Expiration Date 2023-10-12" and confirms that the current date is "2023-10-10". Since the expiration date is in two days, it is determined to be an alert item. It generates the message "Your milk is nearing its expiration date. Please consume it as soon as possible" and sends it to the mobile app.

[1021] 3. Automatic generation of culinary menus

[1022] server:

[1023] The system retrieves current ingredient data from the database and searches the recipe database for menus that can be created based on the available ingredients. It identifies recipes that prioritize the use of high-priority ingredients and generates a list.

[1024] Device (mobile app):

[1025] The system receives data from the server and displays a list of available recipes. If the user selects a recipe, it displays detailed instructions for preparing it.

[1026] Specific example:

[1027] The server recognizes from the database that it has "milk, eggs, and spinach," and searches the recipe database for a "cream pasta" recipe. Since milk is a high-priority ingredient, this recipe is suggested first, and detailed instructions are displayed in the mobile app.

[1028] 4. Automatic generation of necessary shopping lists

[1029] server:

[1030] Based on current ingredient data and the user's desired recipe, the system identifies missing ingredients. It then automatically generates a shopping list and sends it to the user.

[1031] User:

[1032] Check your shopping list on the mobile app. Select the ingredients you plan to buy and complete the list.

[1033] Specific example:

[1034] The server checks if there is no pasta in the refrigerator based on the user's requested "cream pasta" recipe. It automatically adds "pasta" to the shopping list and displays "Please purchase pasta (200g)" in the mobile app.

[1035] This system makes it easier for users to manage their ingredients and effectively reduce unnecessary expenses and food waste.

[1036] The following describes the processing flow.

[1037] Step 1:

[1038] Server: Activates the camera inside the refrigerator and acquires the latest video data. Sends the video data to the image recognition engine.

[1039] Step 2:

[1040] Server: The image recognition engine analyzes the video data and identifies the food ingredients. The identified food ingredients are then labeled (e.g., "milk," "egg," "spinach," etc.).

[1041] Step 3:

[1042] Server: Reads the expiration date information of the recognized food item. Saves the read food item information and expiration date to the database.

[1043] Step 4:

[1044] Server: Retrieves all ingredient data from the database. Compares the current date with the expiration date of each ingredient and calculates the usage priority.

[1045] Step 5:

[1046] Server: Determines whether high-priority ingredients should be subject to an alert. Generates an alert message for ingredients that should trigger an alert.

[1047] Step 6:

[1048] Device (mobile app): Receives alert messages from the server. Displays alert messages on the app.

[1049] Step 7:

[1050] Server: Retrieves current ingredient data from the database. Searches the recipe database for possible menus based on available ingredients.

[1051] Step 8:

[1052] Server: Identifies recipes that prioritize the use of high-priority ingredients and generates a list of them.

[1053] Step 9:

[1054] Terminal (mobile app): Displays a list of available recipes received from the server. If the user selects a recipe, it displays detailed instructions for preparing it.

[1055] Step 10:

[1056] Server: Based on the current ingredient data and the recipe selected by the user, it identifies any missing ingredients. It automatically generates a shopping list and sends it to the user.

[1057] Step 11:

[1058] User: Check the shopping list on the mobile app. Select the ingredients you plan to buy and complete the list.

[1059] Step 12:

[1060] User: Based on the shopping list, purchase the necessary ingredients and use them without waste.

[1061] (Example 1)

[1062] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1063] Traditional refrigerator food management systems rely on manual processes for managing expiration dates and suggesting menus, leading to food waste and loss. Furthermore, users face the challenge of needing to prioritize ingredients and create shopping lists, which requires considerable time and effort. Therefore, there was a need for an automated refrigerator food management system.

[1064] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1065] In this invention, the server includes means for acquiring image data from inside the refrigerator and recognizing food items, means for calculating usage priority based on the expiration dates of the recognized food items, means for generating alerts based on the usage priority, and means for acquiring current food item data from a database and automatically generating available menus. This enables detailed management and efficient use of food items inside the refrigerator.

[1066] "Image data from inside the refrigerator" refers to video information acquired by a camera inside the refrigerator, which is used for recognizing and managing food items.

[1067] "Food ingredients" refers to the food and beverages stored in the refrigerator, and specifically includes items such as milk, eggs, and vegetables.

[1068] "Means of recognition" refers to the process of identifying and labeling food ingredients using image recognition technology.

[1069] The "best before date" refers to the date by which food is expected to maintain its quality; after this date, the likelihood of quality deterioration increases.

[1070] A "means for calculating usage priority" is an algorithm that determines which ingredients should be used preferentially based on their expiration dates and other conditions.

[1071] "Means for generating alerts" refers to a function that creates notifications to inform users of the need to consume high-priority ingredients.

[1072] A "database" is a system that centrally manages digital data, including information on ingredients and recipes.

[1073] "Available menus" refers to a list of dishes that can be made using the ingredients currently in the refrigerator.

[1074] "Methods for automatically generating menus" refer to algorithms that use ingredient information in a database to search for and select the most suitable cooking recipes.

[1075] "A means of creating a shopping list" refers to a function that allows users to list the ingredients they need and prevent them from forgetting to buy them.

[1076] A "mobile device" refers to a communication terminal that a user can carry with them, and includes smartphones and tablets.

[1077] "Image recognition technology" is a technique that analyzes image data acquired by a camera to identify specific objects or features.

[1078] A "generative AI model" refers to an algorithm that uses AI to learn from data and perform a specific task.

[1079] This invention is a system that uses generative AI to manage food in a refrigerator and consume it efficiently without waste. This system mainly consists of the following elements:

[1080] 1. Collection and management of food ingredient data

[1081] The server periodically acquires image data from inside the refrigerator using a high-resolution camera. This video data is then used with image recognition technologies such as TensorFlow and OpenCV to recognize and label food items. The information on the recognized food items and their expiration dates are then stored in a database. For example, if the camera inside the refrigerator recognizes "milk" and its expiration date is "2023-10-12", this information is saved in the database as "Milk: Expiration Date 2023-10-12".

[1082] 2. Calculation of ingredient usage priority and generation of alerts.

[1083] The server retrieves all food data from the database and compares the current date with the expiration date of each food item. Based on this, it calculates the priority of use. For foods with a high priority, it generates an alert message. This alert message is used to notify the user of the need to consume food items that are nearing their expiration date. For example, if the data for "Milk: Expiration Date 2023-10-12" reveals that the expiration date is in two days, it generates an alert message saying, "The milk is about to expire. Please consume it as soon as possible," and sends it to the mobile device.

[1084] 3. Automatic generation of culinary menus

[1085] The server retrieves the current ingredient data and searches the recipe database for available menus. It identifies recipes that prioritize the use of high-priority ingredients and generates a list. This list is sent to the mobile device, allowing the user to view detailed preparation instructions. For example, if the current ingredient data is "milk, eggs, spinach," the server searches for a "cream pasta" recipe and suggests it to the user.

[1086] 4. Automatic generation of necessary shopping lists

[1087] The server identifies missing ingredients based on the current ingredient data and the user's desired recipe, and automatically generates a shopping list of the necessary items. For example, if the user wants to make "cream pasta," the server will confirm that pasta is missing and add "Please purchase pasta (200g)" to the shopping list.

[1088] This system allows users to easily understand the status of ingredients and manage them effectively based on usage priority, preventing waste. Furthermore, by integrating with mobile devices, ingredients can be managed from anywhere, contributing to a reduction in forgotten purchases and food waste.

[1089] Example of a prompt:

[1090] "Please share a recipe using the milk, spinach, and eggs I have in my refrigerator. Also, please let me know if there are any other ingredients I might need."

[1091] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1092] Step 1:

[1093] Obtain image data from inside the refrigerator.

[1094] The server periodically activates a camera installed inside the refrigerator to acquire image data from within. This time, the video is acquired daily at 8:00 AM.

[1095] Input: Trigger for video acquisition (regular schedule)

[1096] Data processing / data calculation: Activate the camera and capture the internal video data.

[1097] Output: High-resolution image data

[1098] Specific actions:

[1099] The server activates the refrigerator's camera based on a schedule and captures the current state of its interior as video.

[1100] Step 2:

[1101] Food identification using an image recognition engine

[1102] The server sends the acquired high-resolution image data to an image recognition engine to identify the food ingredients. For example, it might run a model using TensorFlow.

[1103] Input: High-resolution image data

[1104] Data processing / data calculation: Identify and label food ingredients using image recognition engines such as TensorFlow.

[1105] Output: Food labels and recognition data (e.g., milk, eggs, spinach)

[1106] Specific actions:

[1107] The server sends image data to TensorFlow, which runs the model to identify the ingredients as "milk," "eggs," and "spinach." It then generates information with labels for these ingredients.

[1108] Step 3:

[1109] Reading and saving the expiration date to the database.

[1110] The server reads the expiration date of the identified food item and saves that information to the database.

[1111] Input: Food labels and recognition data

[1112] Data processing / data calculation: Read the expiration date on food packaging using OCR technology.

[1113] Output: Food ingredients and their expiration date information (Example: Milk: Expiration date 2023-10-12)

[1114] Specific actions:

[1115] The server uses OCR technology to read the expiration date from the package image of the identified food item and saves this information to the database in the format "Milk: Expiration Date 2023-10-12".

[1116] Step 4:

[1117] Calculation of Usage Priority

[1118] The server retrieves all ingredient data from the database and calculates usage priority based on expiration dates.

[1119] Input: Food ingredients and expiration date information retrieved from the database.

[1120] Data processing / calculation: Compare the current date with the expiration date and calculate the usage priority.

[1121] Output: Ingredient information with set usage priority.

[1122] Specific actions:

[1123] The server retrieves data for "Milk: Expiration Date 2023-10-12" and compares it to the current date (e.g., 2023-10-10). Since the expiration date is 2 days away, it is given a high priority for use.

[1124] Step 5:

[1125] Alert generation

[1126] The server generates alert messages for ingredients with high usage priority.

[1127] Input: Ingredient information with set usage priority.

[1128] Data processing / data calculation: Generate alerts based on usage priority.

[1129] Output: Alert message

[1130] Specific actions:

[1131] The server generates an alert message saying, "The milk is nearing its expiration date. Please consume it as soon as possible," and sends it to the mobile device.

[1132] Step 6:

[1133] Automatic menu generation

[1134] The server retrieves current ingredient data and searches and generates available menus from the recipe database.

[1135] Input: Current ingredient data retrieved from the database

[1136] Data processing / data calculation: Search the recipe database for possible menus based on ingredient data and generate a list.

[1137] Output: List of culinary options

[1138] Specific actions:

[1139] The server retrieves data for "milk, eggs, and spinach," searches the recipe database for "cream pasta" recipes, generates them, and adds them to the list.

[1140] Step 7:

[1141] Displaying recipes and detailed instructions

[1142] The device (mobile app) receives data from the server and displays it so that the user can view detailed creation instructions.

[1143] Input: List of possible menu items

[1144] Data processing / data calculation: Format data from the server for display.

[1145] Output: Menu list and detailed instructions

[1146] Specific actions:

[1147] The mobile app displays detailed instructions for users who select a "cream pasta" recipe. These instructions include steps such as, "First, prepare the milk, then add the eggs and mix."

[1148] Step 8:

[1149] Automatically generate shopping lists

[1150] The server identifies any missing ingredients based on the recipe selected by the user and generates a shopping list.

[1151] Input: Selected recipe and current ingredient data

[1152] Data processing / data calculation: Identify missing ingredients and generate a shopping list.

[1153] Output: Shopping list

[1154] Specific actions:

[1155] The server compares the "cream pasta" recipe with the current ingredient data and confirms that pasta is missing. It automatically adds "Please purchase pasta (200g)" to the shopping list.

[1156] Step 9:

[1157] Review and edit your shopping list

[1158] Users can view their shopping list on the mobile app and add or edit items they plan to buy as needed.

[1159] Input: Shopping list

[1160] Data processing / data calculation: Operation data when a user edits a list.

[1161] Output: Updated shopping list

[1162] Specific actions:

[1163] The user opens the mobile app, sees the message "Please purchase pasta (200g)," and adds any other items they need to the list.

[1164] (Application Example 1)

[1165] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1166] There is a need for methods to effectively manage the ingredients in refrigerators, reduce food waste due to expired food, and enable users to utilize ingredients efficiently. Furthermore, an efficient system is needed to allow users to easily obtain recipes and quickly replenish missing ingredients. This is expected to save time and improve consumer behavior.

[1167] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1168] In this invention, the server includes means for acquiring image data from inside the refrigerator and recognizing ingredients; means for calculating usage priority based on the expiration date of the recognized ingredients; means for generating alerts based on the usage priority; means for acquiring current ingredient data from a database and automatically generating available menus; means for creating a shopping list based on the generated menus; means for suggesting recipes using available ingredients based on the user's refrigerator information; and means for coordinating with a food delivery service that automatically orders any missing ingredients based on the suggested recipes. This enables efficient use and rapid replenishment of ingredients.

[1169] "Image data" refers to data that represents visual information in a digital format.

[1170] "Ingredients" refer to food products intended for cooking or consumption.

[1171] "Recognition" means identifying and understanding a specific object.

[1172] The "best before date" indicates the period during which food can be eaten safely and deliciously.

[1173] "Usage priority" refers to the order in which multiple ingredients should be used first.

[1174] An "alert" is a warning message that notifies the user.

[1175] A "database" is a digital system that organizes, stores, and manages related data.

[1176] A "menu" refers to the composition and plan of the dishes that will be served as meals within a certain period of time.

[1177] A "shopping list" is a list of items that need to be purchased.

[1178] "Mobile devices" is a general term for portable electronic devices.

[1179] A "recipe" is a set of instructions that shows how to prepare food and the steps involved.

[1180] A "food delivery service" is a service that delivers ordered food to a specified location.

[1181] This invention is a system for efficiently managing food items in a refrigerator and consuming them without waste. This system is realized through the collaboration of a server and a mobile device. The server manages food data, while the mobile device functions as an interface with the user.

[1182] Collection and management of food ingredient data

[1183] Server: The server activates a camera installed inside the refrigerator and acquires the latest image data. The acquired image data is used to identify and label food items using an image recognition engine such as TensorFlow or OpenCV. In addition, the expiration date is automatically read, and this information is stored in the food database.

[1184] Specific example: An image of milk is recognized, and along with labeling, the expiration date "2023-10-12" is added.

[1185] Calculation of ingredient usage priority and generation of alerts

[1186] Server: The server retrieves all ingredient data from the database and calculates usage priority by comparing the current date with the expiration date of each ingredient. It generates alerts for ingredients nearing their expiration date and notifies the user's mobile device.

[1187] Specific example: The server retrieves data for "Milk: Expiration Date 2023-10-12," verifies that the current date is "2023-10-10," and then generates a warning message: "The milk is nearing its expiration date."

[1188] Automatic generation of cookable menus

[1189] Server: The server retrieves current ingredient data from the database and automatically generates menus based on the available ingredients from the recipe database. Recipes using high-priority ingredients are also suggested.

[1190] Specific example: A recipe for "cream pasta" is suggested based on the ingredients in the refrigerator (milk, eggs, spinach).

[1191] Automatically generate a shopping list of necessary items.

[1192] Server: The server identifies missing ingredients based on the recipe selected by the user and the current ingredient data, and automatically generates a shopping list. Furthermore, it sends this list to the user's mobile device.

[1193] Specific example: A user selects a "cream pasta" recipe, and since there is no pasta in the refrigerator, "pasta (200g)" is automatically added to the shopping list.

[1194] Collaboration with food delivery

[1195] Server: The server also has a function to automatically coordinate with food delivery services and order any missing ingredients. This allows users to quickly obtain the ingredients they need.

[1196] Specific example: Based on a shopping list, automatically order "pasta (200g)" that is missing from the refrigerator using a food delivery service.

[1197] Example of a prompt:

[1198] "Create a program that retrieves information about the ingredients in the refrigerator, generates recipes to suggest in a food delivery application based on that information, and automatically orders any missing ingredients from a food delivery service."

[1199] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1200] Step 1:

[1201] The server activates the camera inside the refrigerator and retrieves the latest image data.

[1202] Input: Image data from inside the refrigerator.

[1203] Processing: Image data is acquired from a camera inside the refrigerator and sent to the system. The image recognition engine identifies the food items.

[1204] Output: A list of identified ingredients.

[1205] Step 2:

[1206] The server uses TensorFlow or OpenCV to label the recognized food items and read their expiration dates.

[1207] Input: Image data of food ingredients identified by an image recognition engine.

[1208] Processing: Label food items with labels such as "milk" and "eggs," and extract their expiration date information.

[1209] Output: A list of labeled ingredients and their corresponding expiration dates.

[1210] Step 3:

[1211] The server stores a list of ingredients and expiration date data in a database.

[1212] Input: A list of labeled ingredients and their expiration dates.

[1213] Processing: The process of saving ingredient information to a database.

[1214] Output: Ingredient data stored in the database.

[1215] Step 4:

[1216] The server retrieves all ingredient data from the database, calculates usage priority, and generates a warning message.

[1217] Input: A list of ingredients stored in the database and the expiration date of each ingredient.

[1218] Processing: Compare the current date with the expiration date of each ingredient and calculate the priority for use. Generate a warning message for ingredients that are nearing their expiration date.

[1219] Output: The generated warning message.

[1220] Step 5:

[1221] The server sends a warning message to the mobile device.

[1222] Input: The generated warning message.

[1223] Process: Send a warning message to a mobile device.

[1224] Output: Warning message displayed on the mobile device.

[1225] Step 6:

[1226] The server retrieves current ingredient data from the database and automatically generates usable menus from the recipe database.

[1227] Input: Food ingredient data stored in the database.

[1228] Processing: Based on ingredient data, the system searches for available recipes in the recipe database and automatically generates them.

[1229] Output: Suggested recipe list.

[1230] Step 7:

[1231] The server sends the suggested recipe list to the mobile device and displays it to the user.

[1232] Input: Suggested recipe list.

[1233] Processing: Send the recipe list to your mobile device.

[1234] Output: Recipe list displayed on mobile devices.

[1235] Step 8:

[1236] The user selects their desired recipe on their mobile device, and the server identifies any missing ingredients and automatically generates a shopping list.

[1237] Input: User-selected recipe, current ingredient data.

[1238] Processing: Identifies missing ingredients and automatically generates a shopping list.

[1239] Output: Auto-generated shopping list.

[1240] Step 9:

[1241] The server uses an automatically generated shopping list to place orders with a food delivery service.

[1242] Input: Auto-generated shopping list.

[1243] Processing: Send order data to the food delivery service.

[1244] Output: Order confirmation for food delivery service.

[1245] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1246] This invention is a system that manages food items in a refrigerator and provides suggestions that take the user's emotions into consideration, utilizing generative AI and an emotion engine. The system acquires image data from inside the refrigerator, recognizes food items, calculates usage priority based on their expiration dates, and generates alerts. It also retrieves current food data from a database, automatically generates available menus, and creates necessary shopping lists. This system works in conjunction with mobile devices, allowing users to check the status of food items, alerts, menus, and shopping lists. Furthermore, it recognizes the user's emotions and uses that information to improve food selection and menu suggestions.

[1247] Specific processing flow of the system

[1248] 1. Collection and management of food ingredient data

[1249] server:

[1250] The camera inside the refrigerator is activated to acquire the latest video data. This data is then sent to an image recognition engine. The image recognition engine identifies and labels the food items (e.g., "milk," "eggs," "spinach," etc.). Furthermore, it reads the expiration date information of the food items and stores this recognized food information in a database.

[1251] Specific example:

[1252] The refrigerator camera captures images, and the image recognition engine recognizes the milk as "milk." If the expiration date is "2023-10-12," that information is saved in the database in the format "Milk: Expiration Date 2023-10-12."

[1253] 2. Calculation of ingredient usage priority and generation of alerts

[1254] server:

[1255] Retrieve all ingredient data from the database. Compare the current date with the expiration date of each ingredient and calculate the usage priority. Determine whether ingredients with high usage priority should be subject to an alert. Generate an alert message for ingredients that should trigger an alert.

[1256] Device (mobile app):

[1257] The app receives alert messages from the server and displays them on the app.

[1258] Specific example:

[1259] The server retrieves data for "Milk: Expiration Date 2023-10-12" and confirms that the current date is "2023-10-10". Since the expiration date is in two days, it is determined to be an alert item. It generates the message "Your milk is nearing its expiration date. Please consume it as soon as possible" and sends it to the mobile app.

[1260] 3. Automatic generation of culinary menus

[1261] server:

[1262] Retrieve the current ingredient data from the database. Search the recipe database for menus that can be created based on the available ingredients. Identify recipes that prioritize the use of high-priority ingredients and generate a list of them.

[1263] Device (mobile app):

[1264] Displays a list of available recipes received from the server. If the user selects a recipe, it displays detailed instructions for preparing it.

[1265] Specific example:

[1266] The server recognizes from the database that it has "milk, eggs, and spinach," and searches the recipe database for a "cream pasta" recipe. Since milk is a high-priority ingredient, this recipe is suggested first, and detailed instructions are displayed in the mobile app.

[1267] 4. Automatic generation of necessary shopping lists

[1268] server:

[1269] Based on current ingredient data and the user's selected recipe, the system identifies missing ingredients. It then automatically generates a shopping list and sends it to the user.

[1270] User:

[1271] Check your shopping list on the mobile app. Select the ingredients you plan to buy and complete the list.

[1272] Specific example:

[1273] The server checks if there is no pasta in the refrigerator based on the user's requested "cream pasta" recipe. It automatically adds "pasta" to the shopping list and displays "Please purchase pasta (200g)" in the mobile app.

[1274] Utilizing the Emotion Engine

[1275] 5. Collection of emotional data

[1276] Device (mobile device):

[1277] The device analyzes the user's facial expressions and voice tone through its built-in camera and sensors, collecting emotional data. For example, it uses the camera to recognize the user's face and determine signs of smiling or stress.

[1278] 6. Analysis and reflection of emotional data

[1279] server:

[1280] The system analyzes emotional data transmitted from the device to determine the user's current emotional state. Based on this information, it adjusts ingredient selection and menu suggestions. For example, if the user is feeling stressed, it prioritizes suggesting easy-to-make recipes and ingredients the user likes.

[1281] Specific example:

[1282] The mobile app's facial recognition feature captures the user's smile and sends a message to the server indicating that "the user is relaxed." The server then determines that "the user is relaxed, so we should suggest a new challenge recipe" and adjusts the suggested recipe accordingly.

[1283] 7. Adjusting alerts and notifications

[1284] server:

[1285] Based on emotional data, the system adjusts how alerts are phrased and when notifications are sent. For example, if a user is stressed, the notification might be softened or delayed.

[1286] Specific example:

[1287] The server determines that "the user is feeling stressed" and sends an alert with softer wording, such as "The milk is nearing its expiration date. Please consume it when you have time."

[1288] Through the processing flow described above, this invention streamlines the management of ingredients in the refrigerator and, by making suggestions that take user feelings into consideration, reduces food waste and loss.

[1289] The following describes the processing flow.

[1290] Step 1:

[1291] Server: Activates the camera inside the refrigerator and acquires the latest video data. Then, it sends the acquired video data to the image recognition engine.

[1292] Step 2:

[1293] Server: The image recognition engine analyzes the video data and identifies the food ingredients. The identified food ingredients are then labeled (e.g., "milk," "egg," "spinach," etc.).

[1294] Step 3:

[1295] Server: Reads the expiration date information of the recognized food item. Saves the read food item information and expiration date to the database.

[1296] Step 4:

[1297] Server: Retrieves all ingredient data from the database. Compares the current date with the expiration date of each ingredient and calculates the usage priority (urgency of consumption).

[1298] Step 5:

[1299] Server: Determines whether high-priority ingredients should be subject to an alert. Generates an alert message for ingredients that should trigger an alert.

[1300] Step 6:

[1301] Device (mobile app): Receives alert messages from the server. Displays alert messages on the app.

[1302] Step 7:

[1303] Server: Retrieves current ingredient data from the database. Searches the recipe database for possible menus based on available ingredients.

[1304] Step 8:

[1305] Server: Identifies recipes that prioritize the use of high-priority ingredients and generates a list of them.

[1306] Step 9:

[1307] Terminal (mobile app): Displays a list of available recipes received from the server. If the user selects a recipe, it displays detailed instructions for preparing it.

[1308] Step 10:

[1309] Server: Based on current ingredient data and the user's selected recipe, it identifies missing ingredients. It automatically generates a shopping list and sends it to the user.

[1310] Step 11:

[1311] User: Check the shopping list on the mobile app. Select the ingredients you plan to buy and complete the list.

[1312] Step 12:

[1313] Mobile device: Analyzes the user's facial expressions and voice tone through its built-in camera and sensors to collect emotional data.

[1314] Step 13:

[1315] Server: Analyzes emotional data sent from the terminal to determine the user's emotional state. It determines whether the user is relaxed or stressed.

[1316] Step 14:

[1317] Server: Based on emotional data, it adjusts ingredient selection and menu suggestions. For example, if a user is feeling stressed, it suggests easy-to-make recipes.

[1318] Step 15:

[1319] Server: Adjusts the way alerts are expressed and the timing of notifications based on sentiment data. If the user is feeling stressed, the notification's wording is softened.

[1320] Step 16:

[1321] Device (mobile app): Displays an alert message using gentle language. For example, it might say, "The milk is nearing its expiration date. Please consume it when you have time."

[1322] By following these steps, the management of ingredients in the refrigerator becomes more efficient, and suggestions that take user feelings into consideration are made, resulting in a reduction of food waste and loss.

[1323] (Example 2)

[1324] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1325] Traditional refrigerator food management systems are inefficient and lead to food waste because expiration dates and usage priorities are calculated manually. Furthermore, they fail to improve the user experience because they do not consider the user's emotional state when making suggestions. Additionally, the uniform timing and expression of alerts and notifications can increase user stress. To solve these problems, a system is needed that automates food management and provides suggestions based on the user's emotional state.

[1326] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1327] In this invention, the server includes means for acquiring video data from inside the refrigerator and recognizing ingredients, means for calculating usage priority based on the expiration date of the recognized ingredients, means for generating alerts based on usage priority, means for acquiring current ingredient data from a database and automatically generating available menus, means for creating a shopping list based on the generated menus, means for analyzing sentiment data collected from the terminal and adjusting ingredient selection and menu suggestions based on that information, and means for adjusting alert messages and notification timing based on sentiment data. This enables more efficient ingredient management and suggestions that take into account the user's emotions.

[1328] "Video data" refers to image information acquired from cameras or other imaging devices inside a refrigerator.

[1329] "Ingredients" refers to food and materials stored in the refrigerator.

[1330] "Means of recognition" refers to technologies and devices used to identify food ingredients by analyzing image data.

[1331] "Best before date" refers to the period during which food can maintain its appropriate quality.

[1332] "Usage priority" refers to an indicator used to calculate the priority of which ingredients should be used based on their expiration dates and current usage.

[1333] An "alert" refers to a warning message that is sent to the user based on usage priority.

[1334] A "database" refers to a storage system used to manage information such as ingredient details and recipe information.

[1335] A "menu" refers to a list of dishes that are automatically generated based on the available ingredients.

[1336] A "shopping list" refers to a list of missing ingredients identified based on the generated meal plan.

[1337] "Terminal" refers to a device, such as a mobile device, that a user uses to input and confirm information.

[1338] "Emotional data" refers to data that represents the user's emotional state, and includes information obtained from cameras, microphones, sensors, and other sources.

[1339] "Means of adjusting suggestions" refers to technologies and devices that suggest ingredients and menus suitable for the user based on emotional data.

[1340] "Notification timing" refers to the timing at which alerts and notifications are sent to users.

[1341] This invention is a system that uses generative AI and an emotion engine to manage food items in a refrigerator and provide suggestions that take the user's emotions into consideration. The system acquires video data from inside the refrigerator, recognizes food items, calculates usage priority based on their expiration dates, and generates alerts. It also retrieves current food data from a database, automatically generates available menus, and creates necessary shopping lists. This system works in conjunction with mobile devices, allowing users to check the status of food items, alerts, menus, and shopping lists. Furthermore, it recognizes the user's emotions and uses that information to improve food selection and menu suggestions.

[1342] As a concrete step, the server activates the camera inside the refrigerator to acquire the latest video data. The acquired video data is sent to an image recognition engine such as the Google Cloud Vision API to identify the food items. Based on this identification result, the food items are labeled, and the expiration date information is read, and all of this information is stored in a database (e.g., MySQL). For example, if the acquired video data recognizes milk as "milk" and the expiration date is "2023-10-12", it will be stored in the database in the format "Milk: Expiration Date 2023-10-12".

[1343] The server retrieves all ingredient data from the database, compares the current date with the expiration date of each ingredient, and calculates the usage priority. For ingredients with high usage priority, it generates an alert message and sends it to the mobile device. For example, if the current date is "2023-10-10" and the expiration date of the milk is "2023-10-12", it generates an alert message saying "The milk is nearing its expiration date. Please consume it as soon as possible," and notifies the user.

[1344] Furthermore, the server retrieves current ingredient data from the database and automatically generates menus based on the available ingredients. It identifies recipes that prioritize the use of high-priority ingredients and generates a list of them. For example, if the refrigerator contains "milk, eggs, and spinach," it will search for and preferentially suggest a "cream pasta" recipe.

[1345] The terminal displays detailed instructions for preparing a recipe once the user selects their desired recipe. The server also identifies any missing ingredients based on the selected recipe, automatically generates a shopping list, and sends it to the user. For example, if the user selects "cream pasta" and there is no pasta in the refrigerator, the server automatically adds "pasta" to the shopping list and notifies the user with the message, "Please purchase pasta (200g)."

[1346] As an emotion engine, the device uses its built-in camera and microphone to collect user emotion data. For example, it uses Microsoft Azure's Emotion API to analyze the user's facial expressions and determine signs of smiling or stress. The results are sent to a server, and suggestions are adjusted based on the analyzed emotion data. For instance, if the user is relaxed, it might suggest a new challenge recipe, while if they are stressed, it might suggest an easy recipe.

[1347] In this way, the management of ingredients in the refrigerator becomes more efficient, and suggestions that take user feelings into consideration are also made, resulting in a reduction of food waste and loss.

[1348] Examples of prompts to input into a generative AI model:

[1349] "Based on the data of the ingredients in the refrigerator, please suggest recipes that can be made using the ingredients with the highest priority for use. Also, if the user is relaxed, please suggest a new challenge recipe."

[1350] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1351] Step 1:

[1352] server:

[1353] The system activates the camera inside the refrigerator and acquires the latest video data. This acquired video data is sent to an image recognition engine to recognize the food items. For example, the Google Cloud Vision API is used to identify and label each food item in the video. The input to this process is video data from the camera, and the output is the label information of the recognized food items.

[1354] Specific actions:

[1355] Activate camera → Acquire video data → Send to image recognition engine → Acquire food label information

[1356] Step 2:

[1357] server:

[1358] The system reads the expiration date based on the label information of the recognized food items. For example, it extracts the expiration date by applying text recognition technology to each label. The read expiration date information and food item label information are stored in a database. The input to this process is the label information of the recognized food items, and the output is the food items and their expiration date information stored in the database.

[1359] Specific actions:

[1360] Check label information → Apply text recognition technology → Extract expiration date → Save food labels and expiration dates to a database

[1361] Step 3:

[1362] server:

[1363] All ingredient data is retrieved from the database, and the current date is compared with the expiration date of each ingredient to calculate its usage priority. For example, a Python script can be used to calculate the difference between the current date and each expiration date to determine the usage priority. The input to this process is the ingredient information from the database and the current date, and the output is the ingredient data with the calculated usage priority.

[1364] Specific actions:

[1365] Retrieve ingredient data from the database → Get the current date → Compare with the expiration date → Calculate usage priority

[1366] Step 4:

[1367] server:

[1368] This system determines which ingredients should be alerted based on their usage priority and generates an alert message. For example, for ingredients with a high usage priority, it might generate a message such as, "The milk is nearing its expiration date. Please consume it as soon as possible." The input to this process is ingredient data with calculated usage priority, and the output is the generated alert message.

[1369] Specific actions:

[1370] Identify ingredients with high usage priority → Determine which items require an alert → Generate an alert message

[1371] Step 5:

[1372] Device (mobile app):

[1373] This process receives alert messages sent from the server and displays them in the user interface. Users can then view these alert messages. The input to this process is the alert message from the server, and the output is the alert display to the user.

[1374] Specific actions:

[1375] Receive an alert message → Display the alert in the user interface

[1376] Step 6:

[1377] server:

[1378] The system retrieves current ingredient data from the database and automatically generates menus based on the available ingredients. For example, it uses a cloud-based recipe API to search for recipes that prioritize the use of high-priority ingredients. The input to this process is ingredient data from the database, and the output is a generated menu list.

[1379] Specific actions:

[1380] Retrieve ingredient data from the database → Send a request to the recipe API → Retrieve recipes → Generate a menu list

[1381] Step 7:

[1382] Device (mobile app):

[1383] The system displays a list of menus received from the server in the user interface. When the user selects a desired recipe, detailed instructions for its preparation are displayed. The input to this process is menu data from the server, and the output is the displayed detailed preparation instructions.

[1384] Specific actions:

[1385] Receive menu list → Display in user interface → Display detailed instructions

[1386] Step 8:

[1387] server:

[1388] Based on the current ingredient data and the user's selected recipe, the system identifies missing ingredients and automatically generates a shopping list. For example, a Python script can be used to list ingredients not present in the refrigerator and generate a shopping list of the necessary items. The input to this process is the current ingredient data and the selected recipe, and the output is a shopping list of the missing ingredients.

[1389] Specific actions:

[1390] Retrieve ingredient data and recipe information → Identify missing ingredients → Generate a shopping list

[1391] Step 9:

[1392] User:

[1393] The user checks their shopping list on a mobile app, selects the ingredients they plan to buy, and completes the list. The input for this process is a shopping list sent from the server, and the output is the final shopping list completed by the user.

[1394] Specific actions:

[1395] Review your shopping list → Select the ingredients you plan to buy → Complete the list

[1396] Step 10:

[1397] Device (mobile device):

[1398] The system uses its built-in camera and microphone to collect emotional data such as the user's facial expressions and voice tone. For example, it uses the Emotion API to analyze the user's facial expressions and determine signs of smiles or stress. The input to this process is data from the camera and microphone, and the output is the analyzed emotional data.

[1399] Specific actions:

[1400] Activate camera / microphone → Acquire video / audio data → Send to Emotion API → Analyze emotion data

[1401] Step 11:

[1402] server:

[1403] Based on analyzed emotional data, the system adjusts ingredient selection and menu suggestions. For example, if the user is relaxed, it suggests a new, challenging recipe; if they are stressed, it suggests an easy-to-make recipe. The input to this process is analyzed emotional data, and the output is the adjusted recipe suggestion.

[1404] Specific actions:

[1405] Acquire emotional data → Adjust ingredient selection and menu suggestions → Send suggestions

[1406] Step 12:

[1407] server:

[1408] Based on emotional data, alert messages and notification timing are adjusted. For example, if a user is stressed, the notification's wording might be softened or the notification timing delayed. The input to this process is emotional data, and the output is the adjusted alert message.

[1409] Specific actions:

[1410] Analyze emotional data → Adjust alert messages and notification timing → Send alert messages

[1411] (Application Example 2)

[1412] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1413] Traditional refrigerator food management systems only managed expiration dates and usage priorities, failing to consider user emotions or provide emotionally-based suggestions. This resulted in inefficient food utilization and decreased user satisfaction. In particular, in retail environments, it was difficult for staff to efficiently manage ingredients while maintaining a stress-free work environment.

[1414] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[1415] In this invention, the server includes means for acquiring image data from inside the refrigerator and recognizing ingredients; means for calculating usage priority based on the expiration date of the recognized ingredients; means for generating alerts based on usage priority; means for acquiring current ingredient data from a database and automatically generating available menus; means for creating a shopping list based on the generated menus; means for collecting ingredient data from inside the store in cooperation with image recognition technology and storing it in a database; means for collecting user emotion data using emotion recognition technology and analyzing it with a data analysis engine; and means for adjusting ingredient usage priority and menu suggestions based on the analysis results. This enables efficient management of ingredients and suggestions that take into account the user's emotions.

[1416] "A means of acquiring image data from inside a refrigerator and recognizing food items" refers to a technology that uses a camera inside the refrigerator to acquire images and automatically recognizes the type of food item from those images.

[1417] "A method for calculating usage priority based on expiration dates" refers to an algorithm that calculates the priority of consumption based on the expiration date information of food ingredients and determines the optimal order of use.

[1418] "A means of generating alerts based on usage priority" refers to a system that notifies or warns users about ingredients with high usage priority.

[1419] "A means of retrieving current ingredient data from a database and automatically generating available menus" refers to a system that automatically generates currently available menus and recipes based on ingredient information stored in a database.

[1420] "A means of creating a shopping list based on a generated menu" refers to a technology that identifies missing ingredients from a generated menu or recipe and automatically creates a shopping list of the necessary items.

[1421] "A method for collecting food ingredient data inside a store in conjunction with image recognition technology and storing it in a database" refers to a system that acquires images of food ingredients through cameras installed inside the store, analyzes them, and stores them in a database as food ingredient data.

[1422] "Methods for collecting user emotional data using emotion recognition technology and analyzing it with a data analysis engine" refer to technologies that use devices such as cameras and microphones to analyze the user's facial expressions and tone of voice and estimate their emotional state.

[1423] "A means of adjusting the priority of ingredient use and menu suggestions based on analysis results" refers to a system that adjusts the priority of ingredient use and menu suggestions according to the user's current emotional state, based on collected emotional data.

[1424] This invention relates to a refrigerator food management system that provides suggestions while considering the user's emotional state. The system acquires image data from inside the refrigerator, recognizes food items, calculates usage priority based on their expiration dates, and generates alerts. Furthermore, it retrieves current food data from a database, automatically generates available menus, and creates necessary shopping lists. It also uses emotion recognition technology to collect and analyze user emotional data, adjusting menu suggestions and alert notifications based on the results.

[1425] Hardware and software configuration

[1426] The server acquires image data from a camera inside the refrigerator (e.g., Logitech HD Pro Webcam). It uses an image recognition engine (e.g., OpenCV) to recognize the food items and stores the expiration date information of the recognized items in a database (e.g., MySQL). It retrieves all food item data from the database, calculates usage priority, and generates alerts for high-priority items. It also automatically generates menus from a recipe database based on the food item data and adds missing ingredients to the shopping list. For emotion recognition technology, it uses an engine that performs facial expression analysis and voice analysis (e.g., Microsoft Azure Emotion API) to analyze emotional data collected from the user. Based on the analysis results, the system adjusts the suggestions and notifications according to the user's emotional state.

[1427] example:

[1428] The server analyzes image data acquired by a camera inside the refrigerator and recognizes that the expiration date for the food item "milk" is two days away (October 12, 2023). Because the expiration date is approaching, it is determined that it has a high priority for use and generates an alert saying, "The milk is about to expire. Please consume it as soon as possible." Furthermore, it automatically generates a menu based on the current food data and suggests a "cream pasta" recipe to the user. If the user selects "cream pasta," the missing "pasta" is automatically added to the shopping list. In addition, if the emotion recognition engine detects that the user is experiencing stress, the notification wording is softened, for example, to "The milk is about to expire. Please consume it when you have time."

[1429] Specific example

[1430] Example of a prompt:

[1431] Design a system that manages the contents of a refrigerator and provides emotionally sensitive suggestions. The system will capture images of ingredients using a camera and identify them using an image recognition engine. It will then calculate usage priority based on expiration dates and generate alerts as needed. Additionally, it will offer menu suggestions that take user emotions into consideration.

[1432] Through these specific embodiments, the present invention can achieve both efficient food management and proposals that take into account the emotional state of the user, thereby reducing food waste and improving user satisfaction.

[1433] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1434] Step 1:

[1435] Acquisition of image data and food recognition inside the refrigerator

[1436] The server retrieves the latest image data from a camera installed inside the refrigerator.

[1437] Input: Image data from inside the refrigerator

[1438] Processing: Use an image recognition engine (e.g., OpenCV) to identify ingredients and label each ingredient.

[1439] Output: Data on recognized food items (e.g., milk, eggs, spinach)

[1440] Step 2:

[1441] Reading and saving expiration date information of food ingredients to a database.

[1442] The server reads the expiration date of the recognized food item from the image data and saves that information to the database.

[1443] Input: Data on recognized food ingredients and expiration date information

[1444] Processing: Store the expiration dates of ingredients in a database (e.g., MySQL).

[1445] Output: Food ingredient data stored in the database (Example: Milk: Expiration date 2023-10-12)

[1446] Step 3:

[1447] Calculation of usage priority and generation of alerts

[1448] The server retrieves all ingredient data from the database, compares the current date with the expiration date, and calculates the usage priority. It then generates alerts for ingredients with high usage priority.

[1449] Input: Ingredient data retrieved from the database and the current date

[1450] Processing: Prioritize food items nearing their expiration date and create alert messages for those items.

[1451] Output: Alert message (Example: The milk is nearing its expiration date. Please consume it as soon as possible.)

[1452] Step 4:

[1453] Automatic menu generation

[1454] The server retrieves current ingredient data from the database and automatically generates available menus from the recipe database.

[1455] Input: Food ingredient data retrieved from the database

[1456] Process: Search the recipe database and identify possible menus based on currently available ingredients.

[1457] Output: Automatically generated menu list (Example: Cream Pasta)

[1458] Step 5:

[1459] Automatically generate shopping lists

[1460] The server adds any missing ingredients to the shopping list based on the generated menu.

[1461] Input: Automatically generated menu and current ingredient data

[1462] Process: Identify missing ingredients and create a shopping list.

[1463] Output: Shopping list (Example: Please buy 200g of pasta)

[1464] Step 6:

[1465] Collection and analysis of emotional data

[1466] The device (mobile device) analyzes the user's facial expressions and voice tone through its built-in camera and sensors, and collects emotional data.

[1467] Input: User's facial expressions and tone of voice

[1468] Processing: Analyze emotion data using an emotion recognition engine (e.g., Microsoft Azure Emotion API).

[1469] Output: Analyzed sentiment data (e.g., User is relaxed)

[1470] Step 7:

[1471] Adjusting suggestions based on sentiment data

[1472] Based on the analysis results, the server adjusts the priority of ingredient usage, menu suggestions, and alert notifications.

[1473] Input: Analyzed sentiment data

[1474] Processing: Adjust notification wording and timing according to emotional state (e.g., change to a gentler expression).

[1475] Output: Customized notification message (e.g., "Your milk is nearing its expiration date. Please consume it when you have time.")

[1476] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1477] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include those described above. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions shown by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1478] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[1479] [Fourth Embodiment]

[1480] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[1481] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1482] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1483] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[1484] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1485] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1486] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1487] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[1488] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1489] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1490] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1491] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1492] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1493] This invention is a system that uses generative AI to manage food in a refrigerator and ensure waste-free consumption. The system acquires image data from the refrigerator, recognizes food items, calculates usage priority based on their expiration dates, and generates alerts. Furthermore, it can retrieve current food data from a database, automatically generate available menus, and create necessary shopping lists. This system also integrates with mobile devices, allowing users to check food status, alerts, menus, and shopping lists.

[1494] Specific processing flow of the system

[1495] 1. Collection and management of food ingredient data

[1496] server:

[1497] The camera inside the refrigerator is activated to acquire the latest video data. This data is then sent to an image recognition engine. The image recognition engine identifies and labels the food items (e.g., "milk," "eggs," "spinach," etc.). Furthermore, it reads the expiration date information of the food items and stores this recognized food information in a database.

[1498] Specific example:

[1499] The refrigerator camera captures images, and the image recognition engine recognizes the milk as "milk." If the expiration date is "2023-10-12," that information is saved in the database in the format "Milk: Expiration Date 2023-10-12."

[1500] 2. Calculation of ingredient usage priority and generation of alerts

[1501] server:

[1502] Retrieve all ingredient data from the database. Compare the current date with the expiration date of each ingredient and calculate its usage priority (urgency of consumption). Determine whether ingredients with high usage priority should be alerted (e.g., within 3 days of expiration). Generate an alert message for ingredients that should be alerted.

[1503] Device (mobile app):

[1504] The app receives alert messages from the server and displays them on the app.

[1505] Specific example:

[1506] The server retrieves data for "Milk: Expiration Date 2023-10-12" and confirms that the current date is "2023-10-10". Since the expiration date is in two days, it is determined to be an alert item. It generates the message "Your milk is nearing its expiration date. Please consume it as soon as possible" and sends it to the mobile app.

[1507] 3. Automatic generation of culinary menus

[1508] server:

[1509] The system retrieves current ingredient data from the database and searches the recipe database for menus that can be created based on the available ingredients. It identifies recipes that prioritize the use of high-priority ingredients and generates a list.

[1510] Device (mobile app):

[1511] The system receives data from the server and displays a list of available recipes. If the user selects a recipe, it displays detailed instructions for preparing it.

[1512] Specific example:

[1513] The server recognizes from the database that it has "milk, eggs, and spinach," and searches the recipe database for a "cream pasta" recipe. Since milk is a high-priority ingredient, this recipe is suggested first, and detailed instructions are displayed in the mobile app.

[1514] 4. Automatic generation of necessary shopping lists

[1515] server:

[1516] Based on current ingredient data and the user's desired recipe, the system identifies missing ingredients. It then automatically generates a shopping list and sends it to the user.

[1517] User:

[1518] Check your shopping list on the mobile app. Select the ingredients you plan to buy and complete the list.

[1519] Specific example:

[1520] The server checks if there is no pasta in the refrigerator based on the user's requested "cream pasta" recipe. It automatically adds "pasta" to the shopping list and displays "Please purchase pasta (200g)" in the mobile app.

[1521] This system makes it easier for users to manage their ingredients and effectively reduce unnecessary expenses and food waste.

[1522] The following describes the processing flow.

[1523] Step 1:

[1524] Server: Activates the camera inside the refrigerator and acquires the latest video data. Sends the video data to the image recognition engine.

[1525] Step 2:

[1526] Server: The image recognition engine analyzes the video data and identifies the food ingredients. The identified food ingredients are then labeled (e.g., "milk," "egg," "spinach," etc.).

[1527] Step 3:

[1528] Server: Reads the expiration date information of the recognized food item. Saves the read food item information and expiration date to the database.

[1529] Step 4:

[1530] Server: Retrieves all ingredient data from the database. Compares the current date with the expiration date of each ingredient and calculates the usage priority.

[1531] Step 5:

[1532] Server: Determines whether high-priority ingredients should be subject to an alert. Generates an alert message for ingredients that should trigger an alert.

[1533] Step 6:

[1534] Device (mobile app): Receives alert messages from the server. Displays alert messages on the app.

[1535] Step 7:

[1536] Server: Retrieves current ingredient data from the database. Searches the recipe database for possible menus based on available ingredients.

[1537] Step 8:

[1538] Server: Identifies recipes that prioritize the use of high-priority ingredients and generates a list of them.

[1539] Step 9:

[1540] Terminal (mobile app): Displays a list of available recipes received from the server. If the user selects a recipe, it displays detailed instructions for preparing it.

[1541] Step 10:

[1542] Server: Based on the current ingredient data and the recipe selected by the user, it identifies any missing ingredients. It automatically generates a shopping list and sends it to the user.

[1543] Step 11:

[1544] User: Check the shopping list on the mobile app. Select the ingredients you plan to buy and complete the list.

[1545] Step 12:

[1546] User: Based on the shopping list, purchase the necessary ingredients and use them without waste.

[1547] (Example 1)

[1548] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1549] Traditional refrigerator food management systems rely on manual processes for managing expiration dates and suggesting menus, leading to food waste and loss. Furthermore, users face the challenge of needing to prioritize ingredients and create shopping lists, which requires considerable time and effort. Therefore, there was a need for an automated refrigerator food management system.

[1550] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1551] In this invention, the server includes means for acquiring image data from inside the refrigerator and recognizing food items, means for calculating usage priority based on the expiration dates of the recognized food items, means for generating alerts based on the usage priority, and means for acquiring current food item data from a database and automatically generating available menus. This enables detailed management and efficient use of food items inside the refrigerator.

[1552] "Image data from inside the refrigerator" refers to video information acquired by a camera inside the refrigerator, which is used for recognizing and managing food items.

[1553] "Food ingredients" refers to the food and beverages stored in the refrigerator, and specifically includes items such as milk, eggs, and vegetables.

[1554] "Means of recognition" refers to the process of identifying and labeling food ingredients using image recognition technology.

[1555] The "best before date" refers to the date by which food is expected to maintain its quality; after this date, the likelihood of quality deterioration increases.

[1556] A "means for calculating usage priority" is an algorithm that determines which ingredients should be used preferentially based on their expiration dates and other conditions.

[1557] "Means for generating alerts" refers to a function that creates notifications to inform users of the need to consume high-priority ingredients.

[1558] A "database" is a system that centrally manages digital data, including information on ingredients and recipes.

[1559] "Available menus" refers to a list of dishes that can be made using the ingredients currently in the refrigerator.

[1560] "Methods for automatically generating menus" refer to algorithms that use ingredient information in a database to search for and select the most suitable cooking recipes.

[1561] "A means of creating a shopping list" refers to a function that allows users to list the ingredients they need and prevent them from forgetting to buy them.

[1562] A "mobile device" refers to a communication terminal that a user can carry with them, and includes smartphones and tablets.

[1563] "Image recognition technology" is a technique that analyzes image data acquired by a camera to identify specific objects or features.

[1564] A "generative AI model" refers to an algorithm that uses AI to learn from data and perform a specific task.

[1565] This invention is a system that uses generative AI to manage food in a refrigerator and consume it efficiently without waste. This system mainly consists of the following elements:

[1566] 1. Collection and management of food ingredient data

[1567] The server periodically acquires image data from inside the refrigerator using a high-resolution camera. This video data is then used with image recognition technologies such as TensorFlow and OpenCV to recognize and label food items. The information on the recognized food items and their expiration dates are then stored in a database. For example, if the camera inside the refrigerator recognizes "milk" and its expiration date is "2023-10-12", this information is saved in the database as "Milk: Expiration Date 2023-10-12".

[1568] 2. Calculation of ingredient usage priority and generation of alerts.

[1569] The server retrieves all food data from the database and compares the current date with the expiration date of each food item. Based on this, it calculates the priority of use. For foods with a high priority, it generates an alert message. This alert message is used to notify the user of the need to consume food items that are nearing their expiration date. For example, if the data for "Milk: Expiration Date 2023-10-12" reveals that the expiration date is in two days, it generates an alert message saying, "The milk is about to expire. Please consume it as soon as possible," and sends it to the mobile device.

[1570] 3. Automatic generation of culinary menus

[1571] The server retrieves the current ingredient data and searches the recipe database for available menus. It identifies recipes that prioritize the use of high-priority ingredients and generates a list. This list is sent to the mobile device, allowing the user to view detailed preparation instructions. For example, if the current ingredient data is "milk, eggs, spinach," the server searches for a "cream pasta" recipe and suggests it to the user.

[1572] 4. Automatic generation of necessary shopping lists

[1573] The server identifies missing ingredients based on the current ingredient data and the user's desired recipe, and automatically generates a shopping list of the necessary items. For example, if the user wants to make "cream pasta," the server will confirm that pasta is missing and add "Please purchase pasta (200g)" to the shopping list.

[1574] This system allows users to easily understand the status of ingredients and manage them effectively based on usage priority, preventing waste. Furthermore, by integrating with mobile devices, ingredients can be managed from anywhere, contributing to a reduction in forgotten purchases and food waste.

[1575] Example of a prompt:

[1576] "Please share a recipe using the milk, spinach, and eggs I have in my refrigerator. Also, please let me know if there are any other ingredients I might need."

[1577] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1578] Step 1:

[1579] Obtain image data from inside the refrigerator.

[1580] The server periodically activates a camera installed inside the refrigerator to acquire image data from within. This time, the video is acquired daily at 8:00 AM.

[1581] Input: Trigger for video acquisition (regular schedule)

[1582] Data processing / data calculation: Activate the camera and capture the internal video data.

[1583] Output: High-resolution image data

[1584] Specific actions:

[1585] The server activates the refrigerator's camera based on a schedule and captures the current state of its interior as video.

[1586] Step 2:

[1587] Food identification using an image recognition engine

[1588] The server sends the acquired high-resolution image data to an image recognition engine to identify the food ingredients. For example, it might run a model using TensorFlow.

[1589] Input: High-resolution image data

[1590] Data processing / data calculation: Identify and label food ingredients using image recognition engines such as TensorFlow.

[1591] Output: Food labels and recognition data (e.g., milk, eggs, spinach)

[1592] Specific actions:

[1593] The server sends image data to TensorFlow, which runs the model to identify the ingredients as "milk," "eggs," and "spinach." It then generates information with labels for these ingredients.

[1594] Step 3:

[1595] Reading and saving the expiration date to the database.

[1596] The server reads the expiration date of the identified food item and saves that information to the database.

[1597] Input: Food labels and recognition data

[1598] Data processing / data calculation: Read the expiration date on food packaging using OCR technology.

[1599] Output: Food ingredients and their expiration date information (Example: Milk: Expiration date 2023-10-12)

[1600] Specific actions:

[1601] The server uses OCR technology to read the expiration date from the package image of the identified food item and saves this information to the database in the format "Milk: Expiration Date 2023-10-12".

[1602] Step 4:

[1603] Calculation of Usage Priority

[1604] The server retrieves all ingredient data from the database and calculates usage priority based on expiration dates.

[1605] Input: Food ingredients and expiration date information retrieved from the database.

[1606] Data processing / calculation: Compare the current date with the expiration date and calculate the usage priority.

[1607] Output: Ingredient information with set usage priority.

[1608] Specific actions:

[1609] The server retrieves data for "Milk: Expiration Date 2023-10-12" and compares it to the current date (e.g., 2023-10-10). Since the expiration date is 2 days away, it is given a high priority for use.

[1610] Step 5:

[1611] Alert generation

[1612] The server generates alert messages for ingredients with high usage priority.

[1613] Input: Ingredient information with set usage priority.

[1614] Data processing / data calculation: Generate alerts based on usage priority.

[1615] Output: Alert message

[1616] Specific actions:

[1617] The server generates an alert message saying, "The milk is nearing its expiration date. Please consume it as soon as possible," and sends it to the mobile device.

[1618] Step 6:

[1619] Automatic menu generation

[1620] The server retrieves current ingredient data and searches and generates available menus from the recipe database.

[1621] Input: Current ingredient data retrieved from the database

[1622] Data processing / data calculation: Search the recipe database for possible menus based on ingredient data and generate a list.

[1623] Output: List of culinary options

[1624] Specific actions:

[1625] The server retrieves data for "milk, eggs, and spinach," searches the recipe database for "cream pasta" recipes, generates them, and adds them to the list.

[1626] Step 7:

[1627] Displaying recipes and detailed instructions

[1628] The device (mobile app) receives data from the server and displays it so that the user can view detailed creation instructions.

[1629] Input: List of possible menu items

[1630] Data processing / data calculation: Format data from the server for display.

[1631] Output: Menu list and detailed instructions

[1632] Specific actions:

[1633] The mobile app displays detailed instructions for users who select a "cream pasta" recipe. These instructions include steps such as, "First, prepare the milk, then add the eggs and mix."

[1634] Step 8:

[1635] Automatically generate shopping lists

[1636] The server identifies any missing ingredients based on the recipe selected by the user and generates a shopping list.

[1637] Input: Selected recipe and current ingredient data

[1638] Data processing / data calculation: Identify missing ingredients and generate a shopping list.

[1639] Output: Shopping list

[1640] Specific actions:

[1641] The server compares the "cream pasta" recipe with the current ingredient data and confirms that pasta is missing. It automatically adds "Please purchase pasta (200g)" to the shopping list.

[1642] Step 9:

[1643] Review and edit your shopping list

[1644] Users can view their shopping list on the mobile app and add or edit items they plan to buy as needed.

[1645] Input: Shopping list

[1646] Data processing / data calculation: Operation data when a user edits a list.

[1647] Output: Updated shopping list

[1648] Specific actions:

[1649] The user opens the mobile app, sees the message "Please purchase pasta (200g)," and adds any other items they need to the list.

[1650] (Application Example 1)

[1651] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1652] There is a need for methods to effectively manage the ingredients in refrigerators, reduce food waste due to expired food, and enable users to utilize ingredients efficiently. Furthermore, an efficient system is needed to allow users to easily obtain recipes and quickly replenish missing ingredients. This is expected to save time and improve consumer behavior.

[1653] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1654] In this invention, the server includes means for acquiring image data from inside the refrigerator and recognizing ingredients; means for calculating usage priority based on the expiration date of the recognized ingredients; means for generating alerts based on the usage priority; means for acquiring current ingredient data from a database and automatically generating available menus; means for creating a shopping list based on the generated menus; means for suggesting recipes using available ingredients based on the user's refrigerator information; and means for coordinating with a food delivery service that automatically orders any missing ingredients based on the suggested recipes. This enables efficient use and rapid replenishment of ingredients.

[1655] "Image data" refers to data that represents visual information in a digital format.

[1656] "Ingredients" refer to food products intended for cooking or consumption.

[1657] "Recognition" means identifying and understanding a specific object.

[1658] The "best before date" indicates the period during which food can be eaten safely and deliciously.

[1659] "Usage priority" refers to the order in which multiple ingredients should be used first.

[1660] An "alert" is a warning message that notifies the user.

[1661] A "database" is a digital system that organizes, stores, and manages related data.

[1662] A "menu" refers to the composition and plan of the dishes that will be served as meals within a certain period of time.

[1663] A "shopping list" is a list of items that need to be purchased.

[1664] "Mobile devices" is a general term for portable electronic devices.

[1665] A "recipe" is a set of instructions that shows how to prepare food and the steps involved.

[1666] A "food delivery service" is a service that delivers ordered food to a specified location.

[1667] This invention is a system for efficiently managing food items in a refrigerator and consuming them without waste. This system is realized through the collaboration of a server and a mobile device. The server manages food data, while the mobile device functions as an interface with the user.

[1668] Collection and management of food ingredient data

[1669] Server: The server activates a camera installed inside the refrigerator and acquires the latest image data. The acquired image data is used to identify and label food items using an image recognition engine such as TensorFlow or OpenCV. In addition, the expiration date is automatically read, and this information is stored in the food database.

[1670] Specific example: An image of milk is recognized, and along with labeling, the expiration date "2023-10-12" is added.

[1671] Calculation of ingredient usage priority and generation of alerts

[1672] Server: The server retrieves all ingredient data from the database and calculates usage priority by comparing the current date with the expiration date of each ingredient. It generates alerts for ingredients nearing their expiration date and notifies the user's mobile device.

[1673] Specific example: The server retrieves data for "Milk: Expiration Date 2023-10-12," verifies that the current date is "2023-10-10," and then generates a warning message: "The milk is nearing its expiration date."

[1674] Automatic generation of cookable menus

[1675] Server: The server retrieves current ingredient data from the database and automatically generates menus based on the available ingredients from the recipe database. Recipes using high-priority ingredients are also suggested.

[1676] Specific example: A recipe for "cream pasta" is suggested based on the ingredients in the refrigerator (milk, eggs, spinach).

[1677] Automatically generate a shopping list of necessary items.

[1678] Server: The server identifies missing ingredients based on the recipe selected by the user and the current ingredient data, and automatically generates a shopping list. Furthermore, it sends this list to the user's mobile device.

[1679] Specific example: A user selects a "cream pasta" recipe, and since there is no pasta in the refrigerator, "pasta (200g)" is automatically added to the shopping list.

[1680] Collaboration with food delivery

[1681] Server: The server also has a function to automatically coordinate with food delivery services and order any missing ingredients. This allows users to quickly obtain the ingredients they need.

[1682] Specific example: Based on a shopping list, automatically order "pasta (200g)" that is missing from the refrigerator using a food delivery service.

[1683] Example of a prompt:

[1684] "Create a program that retrieves information about the ingredients in the refrigerator, generates recipes to suggest in a food delivery application based on that information, and automatically orders any missing ingredients from a food delivery service."

[1685] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1686] Step 1:

[1687] The server activates the camera inside the refrigerator and retrieves the latest image data.

[1688] Input: Image data from inside the refrigerator.

[1689] Processing: Image data is acquired from a camera inside the refrigerator and sent to the system. The image recognition engine identifies the food items.

[1690] Output: A list of identified ingredients.

[1691] Step 2:

[1692] The server uses TensorFlow or OpenCV to label the recognized food items and read their expiration dates.

[1693] Input: Image data of food ingredients identified by an image recognition engine.

[1694] Processing: Label food items with labels such as "milk" and "eggs," and extract their expiration date information.

[1695] Output: A list of labeled ingredients and their corresponding expiration dates.

[1696] Step 3:

[1697] The server stores a list of ingredients and expiration date data in a database.

[1698] Input: A list of labeled ingredients and their expiration dates.

[1699] Processing: The process of saving ingredient information to a database.

[1700] Output: Ingredient data stored in the database.

[1701] Step 4:

[1702] The server retrieves all ingredient data from the database, calculates usage priority, and generates a warning message.

[1703] Input: A list of ingredients stored in the database and the expiration date of each ingredient.

[1704] Processing: Compare the current date with the expiration date of each ingredient and calculate the priority for use. Generate a warning message for ingredients that are nearing their expiration date.

[1705] Output: The generated warning message.

[1706] Step 5:

[1707] The server sends a warning message to the mobile device.

[1708] Input: The generated warning message.

[1709] Process: Send a warning message to a mobile device.

[1710] Output: Warning message displayed on the mobile device.

[1711] Step 6:

[1712] The server retrieves current ingredient data from the database and automatically generates usable menus from the recipe database.

[1713] Input: Food ingredient data stored in the database.

[1714] Processing: Based on ingredient data, the system searches for available recipes in the recipe database and automatically generates them.

[1715] Output: Suggested recipe list.

[1716] Step 7:

[1717] The server sends the suggested recipe list to the mobile device and displays it to the user.

[1718] Input: Suggested recipe list.

[1719] Processing: Send the recipe list to your mobile device.

[1720] Output: Recipe list displayed on mobile devices.

[1721] Step 8:

[1722] The user selects their desired recipe on their mobile device, and the server identifies any missing ingredients and automatically generates a shopping list.

[1723] Input: User-selected recipe, current ingredient data.

[1724] Processing: Identifies missing ingredients and automatically generates a shopping list.

[1725] Output: Auto-generated shopping list.

[1726] Step 9:

[1727] The server uses an automatically generated shopping list to place orders with a food delivery service.

[1728] Input: Auto-generated shopping list.

[1729] Processing: Send order data to the food delivery service.

[1730] Output: Order confirmation for food delivery service.

[1731] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1732] This invention is a system that manages food items in a refrigerator and provides suggestions that take the user's emotions into consideration, utilizing generative AI and an emotion engine. The system acquires image data from inside the refrigerator, recognizes food items, calculates usage priority based on their expiration dates, and generates alerts. It also retrieves current food data from a database, automatically generates available menus, and creates necessary shopping lists. This system works in conjunction with mobile devices, allowing users to check the status of food items, alerts, menus, and shopping lists. Furthermore, it recognizes the user's emotions and uses that information to improve food selection and menu suggestions.

[1733] Specific processing flow of the system

[1734] 1. Collection and management of food ingredient data

[1735] server:

[1736] The camera inside the refrigerator is activated to acquire the latest video data. This data is then sent to an image recognition engine. The image recognition engine identifies and labels the food items (e.g., "milk," "eggs," "spinach," etc.). Furthermore, it reads the expiration date information of the food items and stores this recognized food information in a database.

[1737] Specific example:

[1738] The refrigerator camera captures images, and the image recognition engine recognizes the milk as "milk." If the expiration date is "2023-10-12," that information is saved in the database in the format "Milk: Expiration Date 2023-10-12."

[1739] 2. Calculation of ingredient usage priority and generation of alerts

[1740] server:

[1741] Retrieve all ingredient data from the database. Compare the current date with the expiration date of each ingredient and calculate the usage priority. Determine whether ingredients with high usage priority should be subject to an alert. Generate an alert message for ingredients that should trigger an alert.

[1742] Device (mobile app):

[1743] The app receives alert messages from the server and displays them on the app.

[1744] Specific example:

[1745] The server retrieves data for "Milk: Expiration Date 2023-10-12" and confirms that the current date is "2023-10-10". Since the expiration date is in two days, it is determined to be an alert item. It generates the message "Your milk is nearing its expiration date. Please consume it as soon as possible" and sends it to the mobile app.

[1746] 3. Automatic generation of culinary menus

[1747] server:

[1748] Retrieve the current ingredient data from the database. Search the recipe database for menus that can be created based on the available ingredients. Identify recipes that prioritize the use of high-priority ingredients and generate a list of them.

[1749] Device (mobile app):

[1750] Displays a list of available recipes received from the server. If the user selects a recipe, it displays detailed instructions for preparing it.

[1751] Specific example:

[1752] The server recognizes from the database that it has "milk, eggs, and spinach," and searches the recipe database for a "cream pasta" recipe. Since milk is a high-priority ingredient, this recipe is suggested first, and detailed instructions are displayed in the mobile app.

[1753] 4. Automatic generation of necessary shopping lists

[1754] server:

[1755] Based on current ingredient data and the user's selected recipe, the system identifies missing ingredients. It then automatically generates a shopping list and sends it to the user.

[1756] User:

[1757] Check your shopping list on the mobile app. Select the ingredients you plan to buy and complete the list.

[1758] Specific example:

[1759] The server checks if there is no pasta in the refrigerator based on the user's requested "cream pasta" recipe. It automatically adds "pasta" to the shopping list and displays "Please purchase pasta (200g)" in the mobile app.

[1760] Utilizing the Emotion Engine

[1761] 5. Collection of emotional data

[1762] Device (mobile device):

[1763] The device analyzes the user's facial expressions and voice tone through its built-in camera and sensors, collecting emotional data. For example, it uses the camera to recognize the user's face and determine signs of smiling or stress.

[1764] 6. Analysis and reflection of emotional data

[1765] server:

[1766] The system analyzes emotional data transmitted from the device to determine the user's current emotional state. Based on this information, it adjusts ingredient selection and menu suggestions. For example, if the user is feeling stressed, it prioritizes suggesting easy-to-make recipes and ingredients the user likes.

[1767] Specific example:

[1768] The mobile app's facial recognition feature captures the user's smile and sends a message to the server indicating that "the user is relaxed." The server then determines that "the user is relaxed, so we should suggest a new challenge recipe" and adjusts the suggested recipe accordingly.

[1769] 7. Adjusting alerts and notifications

[1770] server:

[1771] Based on emotional data, the system adjusts how alerts are phrased and when notifications are sent. For example, if a user is stressed, the notification might be softened or delayed.

[1772] Specific example:

[1773] The server determines that "the user is feeling stressed" and sends an alert with softer wording, such as "The milk is nearing its expiration date. Please consume it when you have time."

[1774] Through the processing flow described above, this invention streamlines the management of ingredients in the refrigerator and, by making suggestions that take user feelings into consideration, reduces food waste and loss.

[1775] The following describes the processing flow.

[1776] Step 1:

[1777] Server: Activates the camera inside the refrigerator and acquires the latest video data. Then, it sends the acquired video data to the image recognition engine.

[1778] Step 2:

[1779] Server: The image recognition engine analyzes the video data and identifies the food ingredients. The identified food ingredients are then labeled (e.g., "milk," "egg," "spinach," etc.).

[1780] Step 3:

[1781] Server: Reads the expiration date information of the recognized food item. Saves the read food item information and expiration date to the database.

[1782] Step 4:

[1783] Server: Retrieves all ingredient data from the database. Compares the current date with the expiration date of each ingredient and calculates the usage priority (urgency of consumption).

[1784] Step 5:

[1785] Server: Determines whether high-priority ingredients should be subject to an alert. Generates an alert message for ingredients that should trigger an alert.

[1786] Step 6:

[1787] Device (mobile app): Receives alert messages from the server. Displays alert messages on the app.

[1788] Step 7:

[1789] Server: Retrieves current ingredient data from the database. Searches the recipe database for possible menus based on available ingredients.

[1790] Step 8:

[1791] Server: Identifies recipes that prioritize the use of high-priority ingredients and generates a list of them.

[1792] Step 9:

[1793] Terminal (mobile app): Displays a list of available recipes received from the server. If the user selects a recipe, it displays detailed instructions for preparing it.

[1794] Step 10:

[1795] Server: Based on current ingredient data and the user's selected recipe, it identifies missing ingredients. It automatically generates a shopping list and sends it to the user.

[1796] Step 11:

[1797] User: Check the shopping list on the mobile app. Select the ingredients you plan to buy and complete the list.

[1798] Step 12:

[1799] Mobile device: Analyzes the user's facial expressions and voice tone through its built-in camera and sensors to collect emotional data.

[1800] Step 13:

[1801] Server: Analyzes emotional data sent from the terminal to determine the user's emotional state. It determines whether the user is relaxed or stressed.

[1802] Step 14:

[1803] Server: Based on emotional data, it adjusts ingredient selection and menu suggestions. For example, if a user is feeling stressed, it suggests easy-to-make recipes.

[1804] Step 15:

[1805] Server: Adjusts the way alerts are expressed and the timing of notifications based on sentiment data. If the user is feeling stressed, the notification's wording is softened.

[1806] Step 16:

[1807] Device (mobile app): Displays an alert message using gentle language. For example, it might say, "The milk is nearing its expiration date. Please consume it when you have time."

[1808] By following these steps, the management of ingredients in the refrigerator becomes more efficient, and suggestions that take user feelings into consideration are made, resulting in a reduction of food waste and loss.

[1809] (Example 2)

[1810] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1811] Traditional refrigerator food management systems are inefficient and lead to food waste because expiration dates and usage priorities are calculated manually. Furthermore, they fail to improve the user experience because they do not consider the user's emotional state when making suggestions. Additionally, the uniform timing and expression of alerts and notifications can increase user stress. To solve these problems, a system is needed that automates food management and provides suggestions based on the user's emotional state.

[1812] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1813] In this invention, the server includes means for acquiring video data from inside the refrigerator and recognizing ingredients, means for calculating usage priority based on the expiration date of the recognized ingredients, means for generating alerts based on usage priority, means for acquiring current ingredient data from a database and automatically generating available menus, means for creating a shopping list based on the generated menus, means for analyzing sentiment data collected from the terminal and adjusting ingredient selection and menu suggestions based on that information, and means for adjusting alert messages and notification timing based on sentiment data. This enables more efficient ingredient management and suggestions that take into account the user's emotions.

[1814] "Video data" refers to image information acquired from cameras or other imaging devices inside a refrigerator.

[1815] "Ingredients" refers to food and materials stored in the refrigerator.

[1816] "Means of recognition" refers to technologies and devices used to identify food ingredients by analyzing image data.

[1817] "Best before date" refers to the period during which food can maintain its appropriate quality.

[1818] "Usage priority" refers to an indicator used to calculate the priority of which ingredients should be used based on their expiration dates and current usage.

[1819] An "alert" refers to a warning message that is sent to the user based on usage priority.

[1820] A "database" refers to a storage system used to manage information such as ingredient details and recipe information.

[1821] A "menu" refers to a list of dishes that are automatically generated based on the available ingredients.

[1822] A "shopping list" refers to a list of missing ingredients identified based on the generated meal plan.

[1823] "Terminal" refers to a device, such as a mobile device, that a user uses to input and confirm information.

[1824] "Emotional data" refers to data that represents the user's emotional state, and includes information obtained from cameras, microphones, sensors, and other sources.

[1825] "Means of adjusting suggestions" refers to technologies and devices that suggest ingredients and menus suitable for the user based on emotional data.

[1826] "Notification timing" refers to the timing at which alerts and notifications are sent to users.

[1827] This invention is a system that uses generative AI and an emotion engine to manage food items in a refrigerator and provide suggestions that take the user's emotions into consideration. The system acquires video data from inside the refrigerator, recognizes food items, calculates usage priority based on their expiration dates, and generates alerts. It also retrieves current food data from a database, automatically generates available menus, and creates necessary shopping lists. This system works in conjunction with mobile devices, allowing users to check the status of food items, alerts, menus, and shopping lists. Furthermore, it recognizes the user's emotions and uses that information to improve food selection and menu suggestions.

[1828] As a concrete step, the server activates the camera inside the refrigerator to acquire the latest video data. The acquired video data is sent to an image recognition engine such as the Google Cloud Vision API to identify the food items. Based on this identification result, the food items are labeled, and the expiration date information is read, and all of this information is stored in a database (e.g., MySQL). For example, if the acquired video data recognizes milk as "milk" and the expiration date is "2023-10-12", it will be stored in the database in the format "Milk: Expiration Date 2023-10-12".

[1829] The server retrieves all ingredient data from the database, compares the current date with the expiration date of each ingredient, and calculates the usage priority. For ingredients with high usage priority, it generates an alert message and sends it to the mobile device. For example, if the current date is "2023-10-10" and the expiration date of the milk is "2023-10-12", it generates an alert message saying "The milk is nearing its expiration date. Please consume it as soon as possible," and notifies the user.

[1830] Furthermore, the server retrieves current ingredient data from the database and automatically generates menus based on the available ingredients. It identifies recipes that prioritize the use of high-priority ingredients and generates a list of them. For example, if the refrigerator contains "milk, eggs, and spinach," it will search for and preferentially suggest a "cream pasta" recipe.

[1831] The terminal displays detailed instructions for preparing a recipe once the user selects their desired recipe. The server also identifies any missing ingredients based on the selected recipe, automatically generates a shopping list, and sends it to the user. For example, if the user selects "cream pasta" and there is no pasta in the refrigerator, the server automatically adds "pasta" to the shopping list and notifies the user with the message, "Please purchase pasta (200g)."

[1832] As an emotion engine, the device uses its built-in camera and microphone to collect user emotion data. For example, it uses Microsoft Azure's Emotion API to analyze the user's facial expressions and determine signs of smiling or stress. The results are sent to a server, and suggestions are adjusted based on the analyzed emotion data. For instance, if the user is relaxed, it might suggest a new challenge recipe, while if they are stressed, it might suggest an easy recipe.

[1833] In this way, the management of ingredients in the refrigerator becomes more efficient, and suggestions that take user feelings into consideration are also made, resulting in a reduction of food waste and loss.

[1834] Examples of prompts to input into a generative AI model:

[1835] "Based on the data of the ingredients in the refrigerator, please suggest recipes that can be made using the ingredients with the highest priority for use. Also, if the user is relaxed, please suggest a new challenge recipe."

[1836] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1837] Step 1:

[1838] server:

[1839] The system activates the camera inside the refrigerator and acquires the latest video data. This acquired video data is sent to an image recognition engine to recognize the food items. For example, the Google Cloud Vision API is used to identify and label each food item in the video. The input to this process is video data from the camera, and the output is the label information of the recognized food items.

[1840] Specific actions:

[1841] Activate camera → Acquire video data → Send to image recognition engine → Acquire food label information

[1842] Step 2:

[1843] server:

[1844] The system reads the expiration date based on the label information of the recognized food items. For example, it extracts the expiration date by applying text recognition technology to each label. The read expiration date information and food item label information are stored in a database. The input to this process is the label information of the recognized food items, and the output is the food items and their expiration date information stored in the database.

[1845] Specific actions:

[1846] Check label information → Apply text recognition technology → Extract expiration date → Save food labels and expiration dates to a database

[1847] Step 3:

[1848] server:

[1849] All ingredient data is retrieved from the database, and the current date is compared with the expiration date of each ingredient to calculate its usage priority. For example, a Python script can be used to calculate the difference between the current date and each expiration date to determine the usage priority. The input to this process is the ingredient information from the database and the current date, and the output is the ingredient data with the calculated usage priority.

[1850] Specific actions:

[1851] Retrieve ingredient data from the database → Get the current date → Compare with the expiration date → Calculate usage priority

[1852] Step 4:

[1853] server:

[1854] This system determines which ingredients should be alerted based on their usage priority and generates an alert message. For example, for ingredients with a high usage priority, it might generate a message such as, "The milk is nearing its expiration date. Please consume it as soon as possible." The input to this process is ingredient data with calculated usage priority, and the output is the generated alert message.

[1855] Specific actions:

[1856] Identify ingredients with high usage priority → Determine which items require an alert → Generate an alert message

[1857] Step 5:

[1858] Device (mobile app):

[1859] This process receives alert messages sent from the server and displays them in the user interface. Users can then view these alert messages. The input to this process is the alert message from the server, and the output is the alert display to the user.

[1860] Specific actions:

[1861] Receive an alert message → Display the alert in the user interface

[1862] Step 6:

[1863] server:

[1864] The system retrieves current ingredient data from the database and automatically generates menus based on the available ingredients. For example, it uses a cloud-based recipe API to search for recipes that prioritize the use of high-priority ingredients. The input to this process is ingredient data from the database, and the output is a generated menu list.

[1865] Specific actions:

[1866] Retrieve ingredient data from the database → Send a request to the recipe API → Retrieve recipes → Generate a menu list

[1867] Step 7:

[1868] Device (mobile app):

[1869] The system displays a list of menus received from the server in the user interface. When the user selects a desired recipe, detailed instructions for its preparation are displayed. The input to this process is menu data from the server, and the output is the displayed detailed preparation instructions.

[1870] Specific actions:

[1871] Receive menu list → Display in user interface → Display detailed instructions

[1872] Step 8:

[1873] server:

[1874] Based on the current ingredient data and the user's selected recipe, the system identifies missing ingredients and automatically generates a shopping list. For example, a Python script can be used to list ingredients not present in the refrigerator and generate a shopping list of the necessary items. The input to this process is the current ingredient data and the selected recipe, and the output is a shopping list of the missing ingredients.

[1875] Specific actions:

[1876] Retrieve ingredient data and recipe information → Identify missing ingredients → Generate a shopping list

[1877] Step 9:

[1878] User:

[1879] The user checks their shopping list on a mobile app, selects the ingredients they plan to buy, and completes the list. The input for this process is a shopping list sent from the server, and the output is the final shopping list completed by the user.

[1880] Specific actions:

[1881] Review your shopping list → Select the ingredients you plan to buy → Complete the list

[1882] Step 10:

[1883] Device (mobile device):

[1884] The system uses its built-in camera and microphone to collect emotional data such as the user's facial expressions and voice tone. For example, it uses the Emotion API to analyze the user's facial expressions and determine signs of smiles or stress. The input to this process is data from the camera and microphone, and the output is the analyzed emotional data.

[1885] Specific actions:

[1886] Activate camera / microphone → Acquire video / audio data → Send to Emotion API → Analyze emotion data

[1887] Step 11:

[1888] server:

[1889] Based on analyzed emotional data, the system adjusts ingredient selection and menu suggestions. For example, if the user is relaxed, it suggests a new, challenging recipe; if they are stressed, it suggests an easy-to-make recipe. The input to this process is analyzed emotional data, and the output is the adjusted recipe suggestion.

[1890] Specific actions:

[1891] Acquire emotional data → Adjust ingredient selection and menu suggestions → Send suggestions

[1892] Step 12:

[1893] server:

[1894] Based on emotional data, alert messages and notification timing are adjusted. For example, if a user is stressed, the notification's wording might be softened or the notification timing delayed. The input to this process is emotional data, and the output is the adjusted alert message.

[1895] Specific actions:

[1896] Analyze emotional data → Adjust alert messages and notification timing → Send alert messages

[1897] (Application Example 2)

[1898] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1899] Traditional refrigerator food management systems only managed expiration dates and usage priorities, failing to consider user emotions or provide emotionally-based suggestions. This resulted in inefficient food utilization and decreased user satisfaction. In particular, in retail environments, it was difficult for staff to efficiently manage ingredients while maintaining a stress-free work environment.

[1900] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[1901] In this invention, the server includes means for acquiring image data from inside the refrigerator and recognizing ingredients; means for calculating usage priority based on the expiration date of the recognized ingredients; means for generating alerts based on usage priority; means for acquiring current ingredient data from a database and automatically generating available menus; means for creating a shopping list based on the generated menus; means for collecting ingredient data from inside the store in cooperation with image recognition technology and storing it in a database; means for collecting user emotion data using emotion recognition technology and analyzing it with a data analysis engine; and means for adjusting ingredient usage priority and menu suggestions based on the analysis results. This enables efficient management of ingredients and suggestions that take into account the user's emotions.

[1902] "A means of acquiring image data from inside a refrigerator and recognizing food items" refers to a technology that uses a camera inside the refrigerator to acquire images and automatically recognizes the type of food item from those images.

[1903] "A method for calculating usage priority based on expiration dates" refers to an algorithm that calculates the priority of consumption based on the expiration date information of food ingredients and determines the optimal order of use.

[1904] "A means of generating alerts based on usage priority" refers to a system that notifies or warns users about ingredients with high usage priority.

[1905] "A means of retrieving current ingredient data from a database and automatically generating available menus" refers to a system that automatically generates currently available menus and recipes based on ingredient information stored in a database.

[1906] "A means of creating a shopping list based on a generated menu" refers to a technology that identifies missing ingredients from a generated menu or recipe and automatically creates a shopping list of the necessary items.

[1907] "A method for collecting food ingredient data inside a store in conjunction with image recognition technology and storing it in a database" refers to a system that acquires images of food ingredients through cameras installed inside the store, analyzes them, and stores them in a database as food ingredient data.

[1908] "Methods for collecting user emotional data using emotion recognition technology and analyzing it with a data analysis engine" refer to technologies that use devices such as cameras and microphones to analyze the user's facial expressions and tone of voice and estimate their emotional state.

[1909] "A means of adjusting the priority of ingredient use and menu suggestions based on analysis results" refers to a system that adjusts the priority of ingredient use and menu suggestions according to the user's current emotional state, based on collected emotional data.

[1910] This invention relates to a refrigerator food management system that provides suggestions while considering the user's emotional state. The system acquires image data from inside the refrigerator, recognizes food items, calculates usage priority based on their expiration dates, and generates alerts. Furthermore, it retrieves current food data from a database, automatically generates available menus, and creates necessary shopping lists. It also uses emotion recognition technology to collect and analyze user emotional data, adjusting menu suggestions and alert notifications based on the results.

[1911] Hardware and software configuration

[1912] The server acquires image data from a camera inside the refrigerator (e.g., Logitech HD Pro Webcam). It uses an image recognition engine (e.g., OpenCV) to recognize the food items and stores the expiration date information of the recognized items in a database (e.g., MySQL). It retrieves all food item data from the database, calculates usage priority, and generates alerts for high-priority items. It also automatically generates menus from a recipe database based on the food item data and adds missing ingredients to the shopping list. For emotion recognition technology, it uses an engine that performs facial expression analysis and voice analysis (e.g., Microsoft Azure Emotion API) to analyze emotional data collected from the user. Based on the analysis results, the system adjusts the suggestions and notifications according to the user's emotional state.

[1913] example:

[1914] The server analyzes image data acquired by a camera inside the refrigerator and recognizes that the expiration date for the food item "milk" is two days away (October 12, 2023). Because the expiration date is approaching, it is determined that it has a high priority for use and generates an alert saying, "The milk is about to expire. Please consume it as soon as possible." Furthermore, it automatically generates a menu based on the current food data and suggests a "cream pasta" recipe to the user. If the user selects "cream pasta," the missing "pasta" is automatically added to the shopping list. In addition, if the emotion recognition engine detects that the user is experiencing stress, the notification wording is softened, for example, to "The milk is about to expire. Please consume it when you have time."

[1915] Specific example

[1916] Example of a prompt:

[1917] Design a system that manages the contents of a refrigerator and provides emotionally sensitive suggestions. The system will capture images of ingredients using a camera and identify them using an image recognition engine. It will then calculate usage priority based on expiration dates and generate alerts as needed. Additionally, it will offer menu suggestions that take user emotions into consideration.

[1918] Through these specific embodiments, the present invention can achieve both efficient food management and proposals that take into account the emotional state of the user, thereby reducing food waste and improving user satisfaction.

[1919] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1920] Step 1:

[1921] Acquisition of image data and food recognition inside the refrigerator

[1922] The server retrieves the latest image data from a camera installed inside the refrigerator.

[1923] Input: Image data from inside the refrigerator

[1924] Processing: Use an image recognition engine (e.g., OpenCV) to identify ingredients and label each ingredient.

[1925] Output: Data on recognized food items (e.g., milk, eggs, spinach)

[1926] Step 2:

[1927] Reading and saving expiration date information of food ingredients to a database.

[1928] The server reads the expiration date of the recognized food item from the image data and saves that information to the database.

[1929] Input: Data on recognized food ingredients and expiration date information

[1930] Processing: Store the expiration dates of ingredients in a database (e.g., MySQL).

[1931] Output: Food ingredient data stored in the database (Example: Milk: Expiration date 2023-10-12)

[1932] Step 3:

[1933] Calculation of usage priority and generation of alerts

[1934] The server retrieves all ingredient data from the database, compares the current date with the expiration date, and calculates the usage priority. It then generates alerts for ingredients with high usage priority.

[1935] Input: Ingredient data retrieved from the database and the current date

[1936] Processing: Prioritize food items nearing their expiration date and create alert messages for those items.

[1937] Output: Alert message (Example: The milk is nearing its expiration date. Please consume it as soon as possible.)

[1938] Step 4:

[1939] Automatic menu generation

[1940] The server retrieves current ingredient data from the database and automatically generates available menus from the recipe database.

[1941] Input: Food ingredient data retrieved from the database

[1942] Process: Search the recipe database and identify possible menus based on currently available ingredients.

[1943] Output: Automatically generated menu list (Example: Cream Pasta)

[1944] Step 5:

[1945] Automatically generate shopping lists

[1946] The server adds any missing ingredients to the shopping list based on the generated menu.

[1947] Input: Automatically generated menu and current ingredient data

[1948] Process: Identify missing ingredients and create a shopping list.

[1949] Output: Shopping list (Example: Please buy 200g of pasta)

[1950] Step 6:

[1951] Collection and analysis of emotional data

[1952] The device (mobile device) analyzes the user's facial expressions and voice tone through its built-in camera and sensors, and collects emotional data.

[1953] Input: User's facial expressions and tone of voice

[1954] Processing: Analyze emotion data using an emotion recognition engine (e.g., Microsoft Azure Emotion API).

[1955] Output: Analyzed sentiment data (e.g., User is relaxed)

[1956] Step 7:

[1957] Adjusting suggestions based on sentiment data

[1958] Based on the analysis results, the server adjusts the priority of ingredient usage, menu suggestions, and alert notifications.

[1959] Input: Analyzed sentiment data

[1960] Processing: Adjust notification wording and timing according to emotional state (e.g., change to a gentler expression).

[1961] Output: Customized notification message (e.g., "Your milk is nearing its expiration date. Please consume it when you have time.")

[1962] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1963] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include those described above. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions shown by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1964] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[1965] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1966] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. In the upper and lower directions of the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. Also, the upper side of the concentric circles is where "pleasant" emotions are located, and the lower side is where "unpleasant" emotions are located. In this way, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[1967] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[1968] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[1969] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[1970] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[1971] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[1972] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[1973] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[1974] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[1975] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1976] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[1977] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[1978] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[1979] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[1980] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[1981] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[1982] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[1983] The following is further disclosed regarding the embodiments described above.

[1984] (Claim 1)

[1985] A means of acquiring image data from inside a refrigerator and recognizing the food items,

[1986] A method for calculating the priority of use based on the expiration date of recognized ingredients,

[1987] A means of generating alerts based on usage priority,

[1988] A method for retrieving current ingredient data from a database and automatically generating available menus,

[1989] A means of creating a shopping list based on the generated menu,

[1990] A system that includes this.

[1991] (Claim 2)

[1992] The system according to claim 1, which works in conjunction with a mobile device to allow the user to check the status of ingredients, alerts, menus, and shopping lists.

[1993] (Claim 3)

[1994] The system according to claim 1, which uses image recognition technology to automatically label food items in a refrigerator and read their expiration dates.

[1995] "Example 1"

[1996] (Claim 1)

[1997] A means of acquiring image data from inside a refrigerator and recognizing the food items,

[1998] A method for calculating the priority of use based on the expiration date of recognized ingredients,

[1999] A means of generating alerts based on usage priority,

[2000] A method for retrieving current ingredient data from a database and automatically generating available menus,

[2001] A means of creating a shopping list based on the generated menu,

[2002] A method for acquiring high-resolution video data, labeling detailed information about ingredients using image recognition technology, and storing it in a database,

[2003] A method for automatically generating menus based on prioritizing ingredients with high usage priority,

[2004] A method to identify missing ingredients and automatically add them to the shopping list,

[2005] A system that includes this.

[2006] (Claim 2)

[2007] The system according to claim 1, which works in conjunction with a mobile device to allow the user to check the status of ingredients, alerts, menus, and shopping lists.

[2008] (Claim 3)

[2009] A system according to claim 1 that automatically labels food items in a refrigerator using image recognition technology and reads their expiration dates. A system according to claim 1 that automatically generates available menus using a generation AI model based on these labels.

[2010] "Application Example 1"

[2011] (Claim 1)

[2012] A means of acquiring image data from inside a refrigerator and recognizing the food items,

[2013] A method for calculating the priority of use based on the expiration date of recognized ingredients,

[2014] A means of generating alerts based on usage priority,

[2015] A method for retrieving current ingredient data from a database and automatically generating available menus,

[2016] A means of creating a shopping list based on the generated menu,

[2017] A method for suggesting recipes using available ingredients based on the user's refrigerator contents,

[2018] A means of integrating with a food delivery service that automatically orders missing ingredients based on the suggested recipe,

[2019] A system that includes this.

[2020] (Claim 2)

[2021] The system according to claim 1, which works in conjunction with a mobile device to allow the user to check the status of ingredients, alerts, menus, and shopping lists.

[2022] (Claim 3)

[2023] The system according to claim 1, which uses image recognition technology to automatically label food items in a refrigerator and read their expiration dates.

[2024] "Example 2 of combining an emotion engine"

[2025] (Claim 1)

[2026] A means of acquiring video data from inside a refrigerator and recognizing the food items,

[2027] A method for calculating the priority of use based on the expiration date of recognized ingredients,

[2028] A means of generating alerts based on usage priority,

[2029] A method for retrieving current ingredient data from a database and automatically generating available menus,

[2030] A means of creating a shopping list based on the generated menu,

[2031] A method for analyzing emotional data collected from a device and adjusting ingredient selection and menu suggestions based on that information,

[2032] A means of adjusting alert messages and notification timing based on sentiment data,

[2033] A system that includes this.

[2034] (Claim 2)

[2035] The system according to claim 1, which works in conjunction with a mobile device to allow the user to check the status of ingredients, alerts, menus, and shopping lists.

[2036] (Claim 3)

[2037] The system according to claim 1, which uses image recognition technology to automatically label food items in a refrigerator and read their expiration dates.

[2038] "Application example 2 when combining with an emotional engine"

[2039] (Claim 1)

[2040] A means of acquiring image data from inside a refrigerator and recognizing the food items,

[2041] A method for calculating the priority of use based on the expiration date of recognized ingredients,

[2042] A means of generating alerts based on usage priority,

[2043] A method for retrieving current ingredient data from a database and automatically generating available menus,

[2044] A means of creating a shopping list based on the generated menu,

[2045] A method for collecting and storing ingredient data from within a store in conjunction with image recognition technology,

[2046] A method for collecting user emotional data using emotion recognition technology and analyzing it with a data analysis engine,

[2047] Based on the analysis results, a means of adjusting the priority of ingredient use and menu suggestions,

[2048] A system that includes this.

[2049] (Claim 2)

[2050] A means of connecting with mobile devices to allow users to check the status of ingredients, alerts, menus, and shopping lists,

[2051] The system according to claim 1, comprising means for adjusting notification expressions and timing based on the user's emotional state using emotion recognition technology.

[2052] (Claim 3)

[2053] A method for automatically labeling food items in a refrigerator using image recognition technology and reading their expiration dates,

[2054] The system ...

Claims

1. A means of acquiring image data from inside a refrigerator and recognizing the food items, A method for calculating the priority of use based on the expiration date of recognized ingredients, A means of generating alerts based on usage priority, A method for retrieving current ingredient data from a database and automatically generating available menus, A means of creating a shopping list based on the generated menu, A system that includes this.

2. The system according to claim 1, which works in conjunction with a mobile device to allow the user to check the status of ingredients, alerts, menus, and shopping lists.

3. The system according to claim 1, which uses image recognition technology to automatically label food items in a refrigerator and read their expiration dates.

Images