System

Abstract

A system is provided.SOLUTION: The refrigerator includes a unit that periodically acquires video data from a video input device installed in the refrigerator, a unit that analyzes the video data to identify a type, a state, and a deterioration speed of an ingredient, a unit that calculates a use priority of each ingredient on the basis of the identified information, a unit that visualizes the use priority and displays the use priority on a display of the refrigerator or a mobile terminal, a unit that automatically generates a menu of a dish on the basis of the use priority, a unit that specifies a deficient part of an ingredient and displays the deficient part as a shopping list, and a unit that predicts an appropriate consumption timing on the basis of the deterioration speed of the ingredient. Means for notifying a user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In modern households, food management in the refrigerator is often inadequate, resulting in food expiring or being wasted frequently. These problems lead to increased food waste and increased household expenses, negatively impacting the environment and the economy. This invention aims to improve the efficiency of food management, reduce food waste, and reduce household expenses. [Means for solving the problem]

[0005] The present invention provides a system that installs a video input device inside a refrigerator and periodically acquires video data. The acquired video data is analyzed on the server side to identify the type, condition, and deterioration rate of ingredients. Based on this identification data, the use priority of each ingredient is calculated and visualized and displayed on the refrigerator's display or a mobile device. Furthermore, a menu is automatically generated based on the use priority, and ingredients that are in short supply are identified and displayed as a shopping list. By predicting the appropriate time to consume ingredients based on the deterioration rate and notifying the user, the system prevents unnecessary food waste, contributing to reducing food loss and household expenses.

[0006] The "video input device" is a device that is installed inside the refrigerator and is used to acquire video data of ingredients.

[0007] "Video data" is image information that records the type, condition, and location of food.

[0008] "Analysis" refers to the process of using an image recognition algorithm to identify the type, condition, and rate of deterioration of ingredients based on the acquired video data.

[0009] "Type of food" refers to the category or name of the food stored in the refrigerator.

[0010] "Condition" refers to the current physical state of the food material, such as color, shape, and degree of deterioration.

[0011] "Deterioration rate" is an indicator of how quickly food quality deteriorates over time.

[0012] "Usage priority" is an evaluation value that indicates how much priority should be given to the consumption of ingredients. Higher priority is set for ingredients that are close to their expiration date or have deteriorated.

[0013] "Visualization" is the process of visually displaying the calculated usage priorities.

[0014] A "display" is a screen device that is installed on a refrigerator or mobile device to display information.

[0015] A "mobile device" is an electronic device that a user can carry with them, such as a smartphone or tablet.

[0016] A "menu" is a specific dish or cooking suggestion made using ingredients.

[0017] "Auto-generation" is the process of using an algorithm, such as an AI model, to automatically create something based on specific inputs.

[0018] A "shopping list" is a list of ingredients that are in short supply and is provided as a reference for purchasing.

[0019] "Deterioration rate-based notification" is a process that tracks the deterioration status of food ingredients in real time and notifies users when it is appropriate to consume them. [Brief explanation of the drawings]

[0020] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7]FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0021] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0022] First, the terms used in the following description will be explained.

[0023] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0024] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0025] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0026] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0027] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0028] [First embodiment]

[0029] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0030] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0031] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0032] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0033] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0034] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0035] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0036] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0037] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0038] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0039] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0040] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0041] The system embodying this invention is designed to effectively manage food ingredients in a refrigerator, reduce food loss, and prevent food waste. This system links a video input device installed in the refrigerator, a server, and a user's mobile terminal.

[0042] 1. Acquiring video data

[0043] The server periodically acquires video data from a video input device (camera) inside the refrigerator, including the location and status of the food items stored inside the refrigerator.

[0044] 2. Recognizing ingredients and understanding their condition

[0045] The server analyzes the captured video data using image recognition technology to identify the type of food and its condition (such as the rate of deterioration and expiration date). For example, apples, milk, eggs, etc. in the refrigerator are automatically detected and their deterioration status is calculated.

[0046] 3. Calculation of usage priority

[0047] The server calculates the usage priority of each ingredient based on the type and condition data of the ingredient. Ingredients that are deteriorating or approaching their expiration date are given a higher priority.

[0048] 4. Visualization of usage priority

[0049] The device receives the usage priority information sent from the server and displays it on the refrigerator display or mobile device. This display uses color to indicate the usage priority of ingredients, allowing users to understand which ingredients are important at a glance. For example, high priority ingredients are displayed in red, medium priority in yellow, and low priority in green.

[0050] 5. Automatic menu generation

[0051] The server automatically generates menus using an AI model based on the priority data for ingredients. For example, it suggests a recipe for "French toast" that uses milk and eggs, which are the high priorities. This information is sent to the device and notified to the user.

[0052] 6. Listing ingredients you are lacking and making shopping suggestions

[0053] The server compares the current ingredient information with the generated menu, lists any missing ingredients, and suggests them as a shopping list. For example, if there is a shortage of butter for "French toast," that information is displayed on the user's mobile device.

[0054] 7. Prediction of deterioration rate and suggestion of consumption timing

[0055] The server monitors the rate at which food items deteriorate and notifies the user when it is time to consume them. For example, if it detects that an apple is deteriorating faster than normal, it will notify the user to consume it as soon as possible.

[0056] Specific examples

[0057] For example, when a user opens a refrigerator, the display shows the following information:

[0058] High priority (red): Milk (expiration date 2 days later)

[0059] Medium priority (yellow): Apple (changing color)

[0060] Low priority (green): Eggs (expiration date one week later)

[0061] Additionally, your mobile device will receive the following information:

[0062] French toast recipe (milk and eggs)

[0063] Missing Ingredient List: Butter

[0064] Consumption timing notification: Please consume the apples as soon as possible

[0065] In this way, users can manage the food in their refrigerators without waste and consume it in a planned manner, reducing food waste and household expenses.

[0066] The processing flow will be explained below.

[0067] Step 1:

[0068] The server periodically acquires video data from a video input device installed inside the refrigerator. Specifically, the server sends a command to the camera to capture the latest video data every hour and saves it in temporary storage.

[0069] Step 2:

[0070] The server analyzes the stored video data and identifies the type and condition of the ingredients. Specifically, the server processes the video data using a machine learning model to identify ingredients such as apples, milk, and eggs. It also analyzes the color and shape of each ingredient to estimate their deterioration rate and expiration date.

[0071] Step 3:

[0072] The server calculates the usage priority of each ingredient based on the identified ingredient data. Specifically, the server assigns a priority score to each ingredient according to its expiration date and deterioration state, and classifies it into three categories: high, medium, and low.

[0073] Step 4:

[0074] The server stores the calculated usage priority in a database and prepares the data for visualization. Specifically, the server records the priority score and analysis results for each ingredient in the database.

[0075] Step 5:

[0076] The device retrieves ingredient usage priority data from the server and displays it on the refrigerator display or mobile device. Specifically, the device sends an API request, receives the latest data from the server, and displays it in color. High-priority ingredients are displayed in red, medium-priority in yellow, and low-priority in green.

[0077] Step 6:

[0078] The server automatically generates the optimal menu using an AI model based on the usage priority data. Specifically, the server passes ingredient information to the AI ​​model as input and generates a recipe for "French toast" using milk and eggs, for example.

[0079] Step 7:

[0080] The server sends the generated menu to the device and notifies the user. Specifically, the server sends the menu information to the device via API and notifies the user through the mobile app's notification function.

[0081] Step 8:

[0082] The server compares the current ingredient information with the generated menu and lists any missing ingredients. Specifically, the server compares the list of ingredients needed for "French toast" with the current state of the refrigerator and identifies any missing items (e.g., butter).

[0083] Step 9:

[0084] The server creates a shopping list of the identified ingredients that are in short supply and sends it to the device. Specifically, the server creates a list of ingredients that are in short supply and sends it to the device via API.

[0085] Step 10:

[0086] The device displays the shopping list to the user. Specifically, the device displays "You are running low on the following ingredients: Butter" on the mobile app.

[0087] Step 11:

[0088] The server monitors the rate at which food deteriorates and notifies the user of the appropriate time to consume it. Specifically, the server continuously collects data such as the temperature, weight, and color of the food, and predicts the rate at which it will deteriorate. It notifies the user before deterioration progresses. For example, if it determines that an apple is changing color rapidly, the mobile app will notify the user, saying, "Please consume the apple as soon as possible."

[0089] Step 12:

[0090] The device receives the notification from the server and displays it to the user. Specifically, the device displays the notification from the server through the notification function of the mobile app, informing the user when it is time to consume the ingredients.

[0091] Example 1

[0092] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0093] In modern households, food waste is a common problem due to inadequate management of food in the refrigerator. It is particularly difficult to accurately grasp the rate at which food deteriorates and the priority of its use, and then create menu suggestions and shopping lists based on this information. It is also important to accurately predict when food will be consumed in order to avoid food waste. A system that can solve these problems and effectively manage food is needed.

[0094] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0095] In this invention, the server includes means for periodically acquiring video data from a video input device installed in the refrigerator, means for analyzing the video data to identify the type, condition, and deterioration rate of ingredients, means for calculating the use priority of each ingredient based on the identified information, means for visualizing the use priority and displaying it on the refrigerator's display device or a mobile information terminal, means for automatically generating a menu based on the use priority, means for identifying ingredients that are in short supply and displaying it as a shopping list, means for predicting the appropriate consumption timing based on the deterioration rate of ingredients and notifying the user, means for proposing a menu using an AI model that generates the ingredient use priority and its analysis data, and means for automatically activating the camera to capture video data when the refrigerator is opened and closed. This allows for effective management of ingredients in the refrigerator, reducing food waste and household expenses.

[0096] A "video input device" is a device that is installed inside a refrigerator and is used to acquire still image and video data.

[0097] "Video data" refers to image or video data that shows the location and condition of ingredients in the refrigerator, captured by a video input device.

[0098] "Type of food" refers to the specific classification of food stored in the refrigerator, such as fruit, dairy products, eggs, etc.

[0099] "Status of ingredients" refers to the current condition of ingredients in the refrigerator, including, for example, fresh, deteriorating, or close to expiry date.

[0100] "Deterioration rate" is an indicator of how quickly food deteriorates, tracking changes such as from green to yellow to red.

[0101] "Usage priority" indicates the priority for consumption, taking into consideration the rate at which food ingredients deteriorate and their expiration dates.

[0102] The "display device" is a display device that is installed inside the refrigerator and that allows the user to check the status of ingredients and the priority of their use.

[0103] A "personal digital assistant" is a device that a user can carry around and is used to display the status of ingredients and the priority for consumption.

[0104] A "generative AI model" is an artificial intelligence technology that uses pre-trained algorithms to automatically generate new information, such as recipes, based on data input.

[0105] The "purchase list" displays a list of ingredients that the user needs to purchase.

[0106] "Consumption timing" predicts the optimal time to consume food based on the condition of the food and the rate of deterioration.

[0107] The system embodying this invention is designed to effectively manage food ingredients in a refrigerator, reduce food loss, and prevent food waste. This system links a video input device installed in the refrigerator, a server, and a user's mobile information terminal.

[0108] Hardware and Software Configuration

[0109] 1. Video input device

[0110] A camera (video input device) installed inside the refrigerator acquires video data. This camera operates automatically every time the refrigerator door is opened or closed, capturing internal video. For example, a standard refrigerator camera or a smart camera can be used.

[0111] 2. Server

[0112] The server periodically receives and analyzes the video data sent from the video input device. The server uses the following technologies:

[0113] Image recognition technology: Using OpenCV and TensorFlow, the type, condition, and rate of deterioration of ingredients are identified from video data.

[0114] Generative AI model: For example, a generative AI model such as GPT-4 is used to automatically generate menus based on ingredient data.

[0115] 3. Mobile Information Terminals

[0116] The user's mobile information terminal has an application installed that displays the information sent from the server, and the terminal also displays the data on the refrigerator's display.

[0117] Data processing and calculation

[0118] 1. Acquiring video data

[0119] The server acquires video data from the camera inside the refrigerator, for example, at 9:00 AM and 6:00 PM every day. This data includes the location and condition of the food items stored inside the refrigerator.

[0120] 2. Recognizing ingredients and understanding their condition

[0121] The acquired video data is analyzed using image recognition technology (OpenCV and TensorFlow) to identify the type of food and its condition. For example, apples, milk, eggs, etc. in the refrigerator are automatically detected, and their deterioration status and expiration date are estimated.

[0122] 3. Calculation of usage priority

[0123] Based on the analyzed food data, the server calculates the usage priority of ingredients that are deteriorating or approaching their expiration date. For example, the priority is calculated on a scale of 0 to 100, with the higher the number, the higher the usage priority.

[0124] 4. Visualization of usage priority

[0125] The server calculates the usage priority and sends it to the terminal in JSON format, where it is displayed on the refrigerator display or mobile information terminal. For example, it is displayed in red (high priority), yellow (medium priority), or green (low priority).

[0126] 5. Automatic menu generation

[0127] The server uses a generative AI model based on the ingredient usage priority data to automatically generate a menu. An example of a prompt sentence is "Suggest a simple breakfast recipe using milk and eggs," and the generated recipe is provided to the user.

[0128] 6. Listing ingredients you are lacking and making shopping suggestions

[0129] The server compares the recipe with the information on ingredients in the refrigerator, lists any missing ingredients, and displays the list on the mobile information terminal as a shopping list. For example, if the user does not have butter for "French toast," the server notifies the user.

[0130] 7. Prediction of deterioration rate and suggestion of consumption timing

[0131] The server monitors the rate at which food items deteriorate and notifies the user when it is appropriate to consume them. For example, if it detects that an apple is deteriorating faster than normal, a notification will be sent to the mobile information terminal saying, "Please consume the apple as soon as possible."

[0132] Specific examples

[0133] For example, when a user opens a refrigerator, the display shows the following information:

[0134] High priority (red): Milk (expiration date 2 days later)

[0135] Medium priority (yellow): Apple (changing color)

[0136] Low priority (green): Eggs (expiration date one week later)

[0137] The mobile device will be notified of the following:

[0138] French toast recipe (milk and eggs)

[0139] Missing Ingredient List: Butter

[0140] Consumption timing notification: Please consume the apples as soon as possible

[0141] This system allows users to manage food in their refrigerators without waste and consume it in a planned manner, thereby reducing food waste and household expenses.

[0142] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0143] Step 1:

[0144] The server periodically acquires video data from a video input device installed inside the refrigerator.

[0145] Specific behavior:

[0146] The camera automatically activates and captures video data every time the refrigerator door is opened or closed, and the server collects this video data at 9:00 AM and 6:00 PM every day.

[0147] Input: Video data (images and videos showing the location and condition of ingredients in the refrigerator)

[0148] Output: Saved video data file

[0149] Step 2:

[0150] The server analyzes the acquired video data and identifies the type and condition of the ingredients.

[0151] Specific behavior:

[0152] The server uses image recognition technologies such as OpenCV and TensorFlow to classify and recognize ingredients from video data. For example, it analyzes video data to automatically identify ingredients such as apples, milk, and eggs and evaluate their deterioration status.

[0153] Input: Saved video data file

[0154] Output: Ingredient list (including type, condition, and deterioration rate)

[0155] Step 3:

[0156] The server calculates the use priority of each ingredient based on the identified ingredient data.

[0157] Specific behavior:

[0158] The server calculates the priority of ingredients based on their deterioration rate and expiration date, using a score ranging from 0 to 100. Ingredients that are deteriorating or nearing their expiration date are given a higher score.

[0159] Input: Ingredient list (including type, condition, and deterioration rate)

[0160] Output: Usage priority list

[0161] Step 4:

[0162] The server visualizes the usage priority and displays it on the terminal.

[0163] Specific behavior:

[0164] The server sends the usage priority list in JSON format to the terminal. The terminal receives this data and displays it on the refrigerator display and on the mobile information terminal. For example, each ingredient is displayed in red (high priority), yellow (medium priority), and green (low priority).

[0165] Input: Usage priority list

[0166] Output: Ingredient information displayed on a display and a mobile information terminal

[0167] Step 5:

[0168] The server automatically generates a menu based on the usage priority data.

[0169] Specific behavior:

[0170] The server uses a generative AI model such as GPT-4 to create a menu based on the priority of ingredients. For example, it inputs a prompt such as "Suggest a simple breakfast recipe using milk and eggs" into the AI ​​model and presents the generated recipe to the user.

[0171] Input: Usage priority list

[0172] Output: Recipe

[0173] Step 6:

[0174] The server compares the generated menu with the ingredient information, lists any missing ingredients, and displays them as a shopping list.

[0175] Specific behavior:

[0176] The server compares the list of ingredients needed for the created menu with the list of ingredients currently in the refrigerator to identify any missing ingredients. Based on this, it creates a shopping list and sends it to the user's mobile information terminal.

[0177] Input: Menu, ingredient list

[0178] Output: Shopping list

[0179] Step 7:

[0180] The server monitors the rate at which ingredients deteriorate and notifies the user of the appropriate time to consume them.

[0181] Specific behavior:

[0182] The server analyzes the rate at which food deteriorates and determines the appropriate time to consume it. For example, if it determines that an apple is deteriorating faster than normal, it sends a notification to the user's mobile information terminal saying, "Please consume the apple as soon as possible."

[0183] Input: Ingredient list, deterioration rate data

[0184] Output: Information message

[0185] (Application example 1)

[0186] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0187] In modern households, efficiently managing food in the refrigerator and reducing waste are major challenges. However, manual food management is time-consuming, and it is difficult to understand expiration dates and determine the appropriate time to consume food. In addition, food delivery services also require inventory management and optimization of consumption times, but achieving this requires advanced systems.

[0188] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0189] In this invention, the server includes means for periodically acquiring video data from a video input device installed in the refrigerator, means for analyzing the video data to identify the type, condition, and deterioration rate of ingredients, means for calculating the use priority of each ingredient based on the identified information, means for visualizing the use priority and displaying it on an information display device, means for automatically generating cooking recipes based on the use priority, means for identifying shortages of ingredients and displaying it as a shopping list, means for predicting appropriate consumption times based on the deterioration rate of ingredients and notifying the user, means for acquiring consumption data in the refrigerator in real time and automating menu suggestions according to the deterioration status of ingredients, and means for automatically generating cooking recipes based on the consumption priority of ingredients using a generative AI model.This improves the efficiency of food management in the refrigerator, enabling food waste reduction and consumption optimization.

[0190] The "video input device" is a device that is installed inside the refrigerator and is used to acquire video data from inside the refrigerator.

[0191] "Video data" is image information of the inside of the refrigerator acquired by a video input device.

[0192] "Type of foodstuff" is information that indicates the specific classification of food stored in the refrigerator.

[0193] "Condition" is information that indicates the freshness and degree of deterioration of the food material.

[0194] "Deterioration rate" is information indicating the rate at which food ingredients deteriorate.

[0195] "Use priority" is information indicating the consumption priority of each ingredient.

[0196] An "information display device" is a device for displaying acquired information to a user.

[0197] A "cooking recipe" is information that shows the steps for cooking a dish using ingredients.

[0198] A "shopping list" is a list of ingredients that are needed but in short supply.

[0199] "Consumption timing" is information indicating the appropriate time to consume the food material.

[0200] "Consumption data" is information relating to the usage of ingredients.

[0201] "Menu suggestion" means suggesting cooking recipes based on the current ingredients.

[0202] A "generative AI model" is a model that uses artificial intelligence to analyze data and generate results.

[0203] The system embodying this invention is designed to effectively manage food ingredients in a refrigerator and reduce food waste. This system links a video input device installed in the refrigerator, a server, and the user's mobile device.

[0204] The server periodically acquires video data from a video input device (camera) installed inside the refrigerator. The server then analyzes the video data and uses image recognition algorithms to identify the type, condition, and deterioration rate of ingredients. This allows the server to detect the specific classification of ingredients in the refrigerator and their freshness or degree of deterioration.

[0205] The server then calculates the usage priority of each ingredient based on the identified information. Ingredients with a high usage priority are those that deteriorate quickly or are close to their expiration date. This information is displayed on an information display device such as a refrigerator display or a mobile device.

[0206] The server automatically generates cooking recipes based on the priority of use. The generated recipes are sent to the terminal and displayed to the user. The server also identifies missing ingredients and displays them as a shopping list. This information is also sent to the user's terminal in real time.

[0207] Furthermore, the server predicts the best time to consume ingredients based on the rate at which ingredients deteriorate and notifies the user, allowing them to consume ingredients at the appropriate time.The server also uses a generative AI model to automatically generate cooking recipes based on ingredient consumption priorities, automating menu suggestions.

[0208] For example, a camera inside a refrigerator captures video data and sends it to a server. The server analyzes the data and determines that the ingredients are A (deteriorating quickly), B (nearly expiring), and C (fresh). Based on this, an AI model generates recipes using ingredients A and B, which have a high priority for use, and notifies the user. A recipe such as "French toast" is then suggested, and butter, which is in short supply, is also displayed on the shopping list. Notifications such as "Please consume the apples soon" are also given to remind users when to consume them.

[0209] An example of a prompt to input to a generative AI model would be:

[0210] High priority ingredients: milk, apples

[0211] Medium priority ingredients: Eggs

[0212] Please suggest a recipe using these ingredients.

[0213] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0214] Step 1:

[0215] The server periodically acquires video data from a video input device (camera) installed inside the refrigerator.

[0216] Input: Video data acquired from a camera inside the refrigerator

[0217] Data processing: Send the video data to the server

[0218] Output: Video data stored on the server

[0219] Step 2:

[0220] The server analyzes the captured video data and uses image recognition algorithms to identify the type, condition, and rate of deterioration of the food.

[0221] Input: Video data stored on the server

[0222] Data Computing: Using image recognition algorithms to identify the type, condition, and rate of deterioration of ingredients

[0223] Output: Information on the type, condition, and deterioration rate of identified ingredients

[0224] Step 3:

[0225] The server calculates the use priority of each ingredient based on the identified information.

[0226] Input: Information on the type, condition, and deterioration rate of ingredients

[0227] Data calculation: Calculate the usage priority taking into account the rate of deterioration and expiration date

[0228] Output: Usage priority of each ingredient

[0229] Step 4:

[0230] The server visualizes the usage priority and displays it on an information display device.

[0231] Input: Usage priority of each ingredient

[0232] Data processing: Visualization of usage priority (e.g., color-coding)

[0233] Output: Usage priority displayed on the information display

[0234] Step 5:

[0235] The server automatically generates cooking recipes based on the usage priority and notifies the terminal.

[0236] Input: Usage priority of each ingredient

[0237] Data Computing: Automatically generating cooking recipes using generative AI models

[0238] Output: Cooking recipe notified to the device

[0239] Step 6:

[0240] The server identifies missing ingredients and displays them as a shopping list.

[0241] Input: Cooking recipe

[0242] Data calculation: Compare the current ingredient information with the generated menu and list any missing ingredients

[0243] Output: Shopping list displayed on the terminal

[0244] Step 7:

[0245] The server predicts the appropriate time to consume the food based on the rate at which the food deteriorates and notifies the user.

[0246] Input: Food deterioration rate and consumption priority

[0247] Data calculation: Predicting the optimal consumption timing based on the rate of deterioration and usage priority

[0248] Output: Consumption timing suggestions sent to the device

[0249] This will improve the efficiency of food management in the refrigerator, reducing food waste and optimizing consumption.

[0250] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0251] The system embodying this invention recognizes the user's emotional state by combining an emotion engine and customizes services based on this to effectively manage ingredients in a refrigerator. The system includes a video input device installed in the refrigerator, a server, a user's mobile terminal, and an emotion engine.

[0252] 1. Acquiring video data

[0253] The server periodically acquires video data from a video input device (camera) inside the refrigerator, including the type and condition of the food stored in the refrigerator.

[0254] 2. Recognizing ingredients and understanding their condition

[0255] The server analyzes the captured video data using image recognition technology to identify the type of food and its condition (such as the rate of deterioration and expiration date). For example, apples, milk, eggs, etc. in the refrigerator are automatically detected and their deterioration status is calculated.

[0256] 3. Calculation of usage priority

[0257] The server calculates the usage priority of each ingredient based on the type and condition data of the ingredient. Ingredients that are deteriorating or approaching their expiration date are given a higher priority.

[0258] 4. Visualization of usage priority

[0259] The device receives the usage priority information sent from the server and displays it on the refrigerator display or mobile device. This display uses color to indicate the usage priority of ingredients, allowing users to understand which ingredients are important at a glance. For example, high priority ingredients are displayed in red, medium priority in yellow, and low priority in green.

[0260] 5. Automatic menu generation

[0261] The server automatically generates menus using an AI model based on the priority data for ingredients. For example, it suggests a recipe for "French toast" that uses milk and eggs, which are the high priorities. This information is sent to the device and notified to the user.

[0262] 6. Listing ingredients you are lacking and making shopping suggestions

[0263] The server compares the current ingredient information with the generated menu, lists any missing ingredients, and suggests them as a shopping list. For example, if there is a shortage of butter for "French toast," that information is displayed on the user's mobile device.

[0264] 7. Prediction of deterioration rate and suggestion of consumption timing

[0265] The server monitors the rate at which food items deteriorate and notifies the user when it is time to consume them. For example, if it detects that an apple is deteriorating faster than normal, it will notify the user to consume it as soon as possible.

[0266] 8. User Emotion Recognition

[0267] The server uses an emotion engine to recognize the user's emotions. This emotion data is obtained, for example, from a smartphone camera or other sensors. It analyzes the user's facial expressions and voice and stores the results as emotion data.

[0268] 9. Emotionally-based menu adjustments

[0269] The server adjusts the menu based on the user's emotional data. For example, if the user is under stress, it will suggest simple and relaxing meals.

[0270] 10. Emotion-Based Information Display

[0271] The device customizes the information shown on the refrigerator display or mobile device depending on the user's emotional state. For example, if the emotion engine recognizes that the user is tired, it will prioritize showing easy-to-prepare recipes.

[0272] Specific examples

[0273] For example, when a user opens a refrigerator, the display shows the following information:

[0274] High priority (red): Milk (expiration date 2 days later)

[0275] Medium priority (yellow): Apple (changing color)

[0276] Low priority (green): Eggs (expiration date one week later)

[0277] When a user smiles through the camera in front of the refrigerator, the emotion engine recognizes this information. The server determines that the user is relaxed and suggests an easy and enjoyable recipe for "French toast."

[0278] Additionally, your mobile device will display the following information:

[0279] French toast recipe (milk and eggs)

[0280] Missing Ingredient List: Butter

[0281] Consumption timing notification: Please consume the apples as soon as possible

[0282] In this way, users can manage the food in their refrigerators without waste and consume it in a planned manner. In addition, cooking suggestions are made based on the user's emotions, allowing for more personalized food management. This system will reduce food waste and household expenses.

[0283] The processing flow will be explained below.

[0284] Step 1:

[0285] The server periodically acquires video data from a video input device installed inside the refrigerator. Specifically, the server sends a command to the camera to capture the latest video data every hour and saves it in temporary storage.

[0286] Step 2:

[0287] The server analyzes the stored video data and identifies the type and condition of the ingredients. Specifically, the server processes the video data using a machine learning model to identify ingredients such as apples, milk, and eggs. It also analyzes the color and shape of each ingredient to estimate their deterioration rate and expiration date.

[0288] Step 3:

[0289] The server calculates the usage priority of each ingredient based on the identified ingredient data. Specifically, the server assigns a priority score to each ingredient according to its expiration date and deterioration state, and classifies it into three categories: high, medium, and low.

[0290] Step 4:

[0291] The server stores the calculated usage priority in a database and prepares the data for visualization. Specifically, the server records the priority score and analysis results for each ingredient in the database.

[0292] Step 5:

[0293] The device retrieves ingredient usage priority data from the server and displays it on the refrigerator display or mobile device. Specifically, the device sends an API request, receives the latest data from the server, and displays it in color. High-priority ingredients are displayed in red, medium-priority in yellow, and low-priority in green.

[0294] Step 6:

[0295] The server uses an AI model to automatically generate the optimal menu based on the usage priority data. Specifically, the server passes ingredient information to the AI ​​model as input and generates a recipe for "French toast" using milk and eggs, for example.

[0296] Step 7:

[0297] The server sends the generated menu to the device and notifies the user. Specifically, the server sends the menu information to the device via API and notifies the user through the mobile app's notification function.

[0298] Step 8:

[0299] The server compares the current ingredient information with the generated menu and lists any missing ingredients. Specifically, the server compares the list of ingredients needed for "French toast" with the current state of the refrigerator and identifies any missing items (e.g., butter).

[0300] Step 9:

[0301] The server creates a shopping list of the identified ingredients that are in short supply and sends it to the device. Specifically, the server creates a list of ingredients that are in short supply and sends it to the device via API.

[0302] Step 10:

[0303] The device displays the shopping list to the user. Specifically, the device displays "You are running low on the following ingredients: Butter" on the mobile app.

[0304] Step 11:

[0305] The server monitors the rate at which food deteriorates and notifies the user of the appropriate time to consume it. Specifically, the server continuously collects data such as the temperature, weight, and color of the food, and predicts the rate at which it will deteriorate. It notifies the user before deterioration progresses. For example, if it determines that an apple is changing color rapidly, the mobile app will notify the user, saying, "Please consume the apple as soon as possible."

[0306] Step 12:

[0307] The device receives the notification from the server and displays it to the user. Specifically, the device displays the notification from the server through the notification function of the mobile app, informing the user when it is time to consume the ingredients.

[0308] Step 13:

[0309] The server uses an emotion engine to recognize the user's emotions. Specifically, the server analyzes data acquired from the smartphone's camera and microphone, and generates emotion data from the user's facial expressions and voice.

[0310] Step 14:

[0311] The server adjusts the generated menu based on the emotional data. Specifically, if the server determines that the user is under stress, it will prioritize suggesting ingredients that have a relaxing effect and easy-to-cook recipes.

[0312] Step 15:

[0313] The device customizes the information displayed on the refrigerator display or mobile device based on the emotional data. Specifically, if the device recognizes that the user is tired, it will prioritize easy-to-prepare recipes. If the user's emotional state is positive, it will suggest new recipes to try.

[0314] Example flow

[0315] For example, when a user opens a refrigerator, the display shows the following information:

[0316] High priority (red): Milk (expiration date 2 days later)

[0317] Medium priority (yellow): Apple (changing color)

[0318] Low priority (green): Eggs (expiration date one week later)

[0319] When a user smiles through the camera in front of the refrigerator, the emotion engine recognizes this information. The server determines that the user is relaxed and suggests an easy and enjoyable recipe for "French toast."

[0320] Additionally, your mobile device will display the following information:

[0321] French toast recipe (milk and eggs)

[0322] Missing Ingredient List: Butter

[0323] Consumption timing notification: Please consume the apples as soon as possible

[0324] In this way, users can manage the food in their refrigerators without waste and consume it in a planned manner. In addition, cooking suggestions are made based on the user's emotions, allowing for more personalized food management. This system will reduce food waste and household expenses.

[0325] Example 2

[0326] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0327] While conventional refrigerator food management systems can grasp the type and condition of ingredients, they have difficulty managing ingredients and suggesting menus that match the user's emotions and individual lifestyles. Furthermore, while they can predict the rate of deterioration and calculate usage priorities, they do not support adjusting menus or suggesting appropriate consumption times based on the user's emotions. As a result, user satisfaction is low, food waste is often generated, and unnecessary stress is often generated. To solve these issues, a system that can recognize the user's emotional state and customize services based on that state was needed.

[0328] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for periodically acquiring video data from a video input device installed in the refrigerator, means for analyzing the video data to identify the type, condition, and deterioration rate of ingredients, means for calculating the use priority of each ingredient based on the identified information, means for visualizing the use priority and displaying it on the refrigerator's display device or a mobile information terminal, means for generating a menu of dishes based on the use priority, means for identifying shortages of ingredients and displaying it as a shopping list, means for predicting appropriate consumption times based on the deterioration rate of ingredients and notifying the user, means for recognizing the user's emotions using an emotion engine, means for adjusting the menu based on the user's emotions, and means for customizing display information based on the user's emotions. This enables personalized ingredient management and menu suggestions based on the user's emotional state, thereby reducing ingredient waste and improving user satisfaction.

[0329] The "video input device installed inside the refrigerator" refers to a camera or other image capturing device that is installed inside the refrigerator and that captures video data of the food ingredients inside the refrigerator.

[0330] "Video data" is digital data that includes image information of food items in the refrigerator.

[0331] "Type of food" refers to the specific item names of food stored in the refrigerator, such as apples, milk, eggs, etc.

[0332] "Condition of ingredients" refers to the current quality and degree of deterioration of ingredients in the refrigerator.

[0333] "Deterioration rate" is an index that indicates how quickly each food ingredient stored in a refrigerator deteriorates.

[0334] "Usage priority" is calculated based on the priority of using food items in the refrigerator, taking into account factors such as the rate of deterioration and expiration date.

[0335] A "display device" is a display or monitor attached to a refrigerator and is used to visually display information about ingredients and menu items.

[0336] A "mobile information terminal" is an electronic device such as a smartphone or tablet that can be carried by a user and that displays ingredient information and menus.

[0337] "Cooking Menu" is a meal menu suggested based on the ingredients in your refrigerator.

[0338] A "shopping list" is a list of ingredients that need to be purchased based on the ingredients in the refrigerator and the menu.

[0339] "Consumption timing" refers to the predicted optimal time to consume a food item based on its deterioration rate.

[0340] An "emotion engine" is an analysis device or software that analyzes a user's emotional state, stores it as data, and uses it for subsequent processing.

[0341] "User emotion" means the user's psychological state analyzed through the emotion engine.

[0342] "Customizing display information" means adjusting the information displayed on the refrigerator display device or mobile information terminal based on the user's emotional state and priorities.

[0343] The system embodying this invention recognizes the user's emotional state by combining an emotion engine and customizes services based on this to effectively manage ingredients in a refrigerator. The system includes a video input device installed in the refrigerator, a server, a user's mobile terminal, and an emotion engine.

[0344] Hardware and Software Configuration

[0345] 1. Video input device (camera) inside the refrigerator

[0346] A camera is installed inside the refrigerator to periodically capture video data of the ingredients. The camera can take high-resolution images and send the data to a server every hour, for example.

[0347] 2. Server

[0348] The server acquires, analyzes, stores, and processes video data. Specific software used for image recognition is OpenCV and TensorFlow. The emotion engine is used to analyze user emotion data.

[0349] 3. Terminal (refrigerator display or mobile terminal)

[0350] The terminal displays the information sent from the server, visually conveying information such as the priority of ingredients, menus, shopping lists, and consumption timing to the user.

[0351] Data processing and calculation

[0352] 1. Acquiring video data

[0353] The server periodically acquires video data from the video input device (camera) inside the refrigerator, including the type and condition of the food stored in the refrigerator.

[0354] 2. Recognizing ingredients and understanding their condition

[0355] The server uses image recognition software (e.g., OpenCV or TensorFlow) to analyze the captured video data, thereby identifying the type and condition of the food (rate of deterioration and expiration date).

[0356] 3. Calculation of usage priority

[0357] The server calculates the usage priority of each ingredient based on the type and condition data of the ingredient, assigning a higher priority to ingredients that are deteriorating or approaching their expiration date.

[0358] 4. Visualization of usage priority

[0359] The device receives the usage priority information sent from the server and displays it on the refrigerator display or mobile device. The priority is indicated by a color, allowing the user to understand the importance at a glance.

[0360] 5. Automatic menu generation

[0361] The server automatically generates menus using a generative AI model (e.g., GPT-4) based on the ingredient usage priority data, and suggests recipes that include high-priority ingredients.

[0362] 6. Listing ingredients you are lacking and making shopping suggestions

[0363] The server compares the current ingredient information with the generated menu, lists any missing ingredients, and suggests a shopping list to the user.

[0364] 7. Prediction of deterioration rate and suggestion of consumption timing

[0365] The server monitors the rate at which ingredients deteriorate and notifies the user when it is time to consume them.

[0366] 8. User Emotion Recognition

[0367] The server uses an emotion engine to recognize the user's emotions, and this data is obtained from the smartphone's camera and other sensors.

[0368] 9. Emotionally-based menu adjustments

[0369] The server uses an AI model to adjust the menu based on the user's emotional data.

[0370] 10. Emotion-Based Information Display

[0371] The device customizes the information displayed on the refrigerator display and mobile device based on the user's emotional state.

[0372] Specific examples

[0373] For example, when a user opens a refrigerator, the display shows the following information:

[0374] High priority (red): Milk (expiration date 2 days later)

[0375] Medium priority (yellow): Apple (changing color)

[0376] Low priority (green): Eggs (expiration date one week later)

[0377] When the user smiles through the camera in front of the refrigerator, the emotion engine recognizes this information. The server determines that the user is relaxed and suggests a simple and enjoyable recipe for "French toast." The mobile device also displays the following information:

[0378] French toast recipe (milk and eggs)

[0379] Missing Ingredient List: Butter

[0380] Consumption timing notification: Please consume the apples as soon as possible

[0381] Example prompts to input to the generative AI model

[0382] "Suggest a simple recipe using the milk and eggs in the fridge. The user is currently relaxed."

[0383] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0384] Step 1:

[0385] The server periodically acquires video data from a video input device (camera) installed inside the refrigerator.

[0386] Input: Footage from inside a refrigerator.

[0387] Data processing / calculation: The camera takes high-resolution images every hour and sends them to the server as digital data.

[0388] Output: Video data of food in the refrigerator.

[0389] Step 2:

[0390] The server uses image recognition software (e.g., OpenCV or TensorFlow) to analyze the captured video data, thereby identifying the type and condition of the food ingredients (rate of deterioration and expiration date).

[0391] Input: Video data of food in a refrigerator.

[0392] Data processing / calculation: Detects food ingredients such as apples, milk, and eggs from the video and calculates the degree of deterioration and expiration date of each.

[0393] Output: Data on the type, condition and deterioration rate of identified ingredients.

[0394] Step 3:

[0395] The server calculates the use priority of each ingredient based on the type and state data of the ingredient.

[0396] Input: Data on the type, condition and deterioration rate of identified ingredients.

[0397] Data processing / calculation: A mathematical algorithm is used to assign a priority score based on the degree of deterioration and expiration date. Food items that are in an advanced state of deterioration or approaching their expiration date are given a higher priority.

[0398] Output: Usage priority data for each ingredient.

[0399] Step 4:

[0400] The terminal receives the information on the usage priority sent from the server and displays it on the refrigerator's display device or the mobile terminal.

[0401] Input: Usage priority data for each ingredient.

[0402] Data processing / calculation: Usage priority is displayed in color (e.g., high priority is red, medium priority is yellow, and low priority is green).

[0403] Output: Visually displayed ingredient usage priority information.

[0404] Step 5:

[0405] The server automatically generates menus using a generative AI model (e.g., GPT-4) based on ingredient usage priority data.

[0406] Input: Usage priority data for each ingredient.

[0407] Data processing / calculation: Send the prompt "Please suggest a simple recipe using milk and eggs. The user is currently in a relaxed state" to the generative AI model and receive the generated recipe.

[0408] Output: Automatically generated menu information.

[0409] Step 6:

[0410] The server compares the current ingredient information with the generated menu, lists any missing ingredients, and suggests a shopping list to the user.

[0411] Input: Current ingredient information, automatically generated menu information.

[0412] Data processing / calculation: Matching ingredient information with the menu, identifying missing ingredients, and generating a shopping list.

[0413] Output: Shopping list.

[0414] Step 7:

[0415] The server monitors the rate at which ingredients deteriorate and notifies the user when it is time to consume the ingredients.

[0416] Input: Food deterioration rate data.

[0417] Data processing / calculation: Analyzes deterioration rate data and calculates the appropriate consumption timing. For example, generates a notification saying "Please consume the apples as soon as possible."

[0418] Output: Consumption timing notification.

[0419] Step 8:

[0420] The server uses an emotion engine to recognize the user's emotions, and this data is obtained from the smartphone's camera and other sensors.

[0421] Input: Emotion data from smartphone cameras and sensors.

[0422] Data processing / calculation: Analyze the user's facial expressions and voice, and save the results as emotional data.

[0423] Output: User emotion data.

[0424] Step 9:

[0425] The server uses an AI model to adjust the menu based on the user's emotional data.

[0426] Input: User emotion data, automatically generated menu information.

[0427] Data processing / calculation: Emotional data is taken into account and menus are adjusted to be more relaxing (e.g., simple dishes when under stress).

[0428] Output: Menu information tailored based on emotions.

[0429] Step 10:

[0430] The terminal customizes the information displayed on the refrigerator display and the mobile terminal according to the user's emotional state.

[0431] Input: User emotion data.

[0432] Data processing / calculation: Based on emotional data, information to be displayed preferentially (e.g., easy recipes) is selected.

[0433] Output: Customized display information.

[0434] (Application example 2)

[0435] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0436] Conventional refrigerator food management systems focus on calculating the priority of ingredients based on their condition and deterioration rate and notifying the user of this priority. However, because information is provided unilaterally without considering the user's emotional state, there is a problem in that it is not possible to display optimal menu suggestions or product information for the user. The present invention aims to more effectively manage ingredients and suggest products in stores by recognizing the user's emotional state in real time and providing personalized services based on that.

[0437] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0438] In this invention, the server includes means for periodically acquiring video data from a video input device installed in the refrigerator, means for analyzing the video data to identify the type, condition, and deterioration rate of ingredients, means for calculating the use priority of each ingredient based on the identified information, means for visualizing the use priority and displaying it on the refrigerator display or a mobile terminal, means for automatically generating a recipe based on the use priority, means for identifying shortages of ingredients and displaying it as a shopping list, means for predicting appropriate consumption times based on the deterioration rate of ingredients and notifying the user, means including an emotion recognition engine for recognizing the user's emotional state, means for adjusting the recipe based on the user's emotional data, and means for displaying information about products in the store in real time. This enables optimal ingredient management and personalized product suggestions according to the user's emotional state.

[0439] The "video input device" is a device that is installed inside the refrigerator and periodically acquires video data of the food ingredients inside.

[0440] An "image recognition algorithm" is a calculation method for analyzing acquired video data and identifying the type, condition, and rate of deterioration of ingredients.

[0441] "Use priority" is an index that indicates the priority of consumption based on the type, condition, and deterioration rate of food ingredients.

[0442] A "display" is a device for visually displaying information, and may be installed on the surface of a refrigerator or on a mobile device.

[0443] A "mobile terminal" is a portable electronic device such as a smartphone or tablet that receives and displays information.

[0444] A "menu" is a plan of dishes using ingredients, and is automatically generated based on the priority of using ingredients.

[0445] A "shopping list" is a list of information that lists ingredients that are in short supply and is presented for purchase.

[0446] An "emotion recognition engine" is a software or hardware element that analyzes a user's facial expressions and voice and recognizes their emotional state.

[0447] "User emotion data" is data that indicates the temporary or persistent emotional state of the user, obtained through analysis by the emotion recognition engine.

[0448] "Product suggestion" is an act or system that presents product information in a store based on the user's emotional state.

[0449] "Real-time" is a concept that indicates that information is processed immediately and provided without delay.

[0450] In this invention, the following system is implemented to realize the management of ingredients in a refrigerator and the provision of personalized services based on the user's emotional state.

[0451] First, the server periodically acquires video data from a video input device installed inside the refrigerator. An image recognition algorithm is used to analyze this video data. The image recognition algorithm identifies the type, condition, and deterioration rate of ingredients, and calculates the usage priority of each ingredient based on this information. The usage priority is visualized on the refrigerator or mobile device display, allowing users to understand the status of ingredients at a glance.

[0452] Next, the server automatically generates a menu of dishes based on the calculated usage priority. This automatic generation uses a generative AI model. The generated menu is notified to the mobile device and displayed. The server also identifies any missing ingredients and displays them as a shopping list on the mobile device.

[0453] Furthermore, the server monitors the rate at which food items deteriorate and notifies the user of the appropriate time to consume them. For example, if an apple is deteriorating faster than usual, the server will notify the user, "Consume the apple sooner." This information encourages the user to consume food appropriately according to their lifestyle.

[0454] Next, an emotion recognition engine recognizes the user's emotional state. This engine captures and analyzes facial and voice data from the smartphone's camera and other sensors. Based on the user's emotional data, the system adjusts the menu accordingly. For example, if the user is tired, it will suggest simple, relaxing dishes.

[0455] Smart glasses and head-mounted displays (HMDs) are also used to display real-time product information in stores. When a user approaches or picks up a particular product, information about that product and recommended recipes are displayed.

[0456] As a concrete example, suppose a user wearing smart glasses is walking through a supermarket and stops in front of a bottle of milk. The camera in the smart glasses captures the user's smile, and the system determines that the user is relaxed and enjoying themselves. In this case, the smart glasses will display "Fresh milk. Recommended recipe: Panna cotta."

[0457] Prompt Sentence Examples

[0458] "A camera in smart glasses captures the user's facial expressions and suggests supermarket products to the user based on their emotional state. The system analyzes the user's emotional data in real time and displays appropriate product and recipe information based on the results."

[0459] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0460] Flow of the system program that realizes the application example

[0461] Step 1:

[0462] The server periodically acquires video data from a video input device installed inside the refrigerator.

[0463] Input: Video data from a video input device.

[0464] Data processing: Video data is converted to the appropriate resolution and saved as an image file on the server.

[0465] Output: Video data in image file format.

[0466] Step 2:

[0467] The server analyzes the captured video data and uses image recognition algorithms to identify the type, condition and rate of deterioration of the food.

[0468] Input: Stored video data.

[0469] Data processing: Apply image recognition algorithms to identify the type, condition, and rate of deterioration of ingredients.

[0470] Output: Identification data including type, condition and rate of deterioration of the food material.

[0471] Step 3:

[0472] The server calculates the use priority of each ingredient based on the identified information.

[0473] Input: Identification data for food type, condition, and rate of deterioration.

[0474] Data calculation: Calculates the usage priority score based on the deterioration rate and condition.

[0475] Output: Usage priority score for each ingredient.

[0476] Step 4:

[0477] The terminal receives the usage priority information sent from the server and displays it on the refrigerator display or mobile terminal.

[0478] Input: Usage priority score.

[0479] Data transformation: Converting usage priority scores into visual formats such as color coding.

[0480] Output: Visually enhanced usage priority information.

[0481] Step 5:

[0482] The server automatically generates a menu of dishes based on the priority of use.

[0483] Input: Usage priority score for each ingredient.

[0484] Data computation: Uses generative AI models to generate optimal menus.

[0485] Output: The generated menu data.

[0486] Step 6:

[0487] The terminal receives the generated menu data and notifies the mobile terminal.

[0488] Input: Generated menu data.

[0489] Data processing: Convert menu data into notification format.

[0490] Output: Notification message to mobile device.

[0491] Step 7:

[0492] The server compares the current ingredient information with the generated menu, identifies any missing ingredients, and displays them as a shopping list.

[0493] Input: Current ingredient information, generated menu data.

[0494] Data calculation: Compare the ingredients needed for the menu with the ingredients currently available.

[0495] Output: A list of missing ingredients.

[0496] Step 8:

[0497] The server monitors the rate at which ingredients deteriorate and notifies the user of the appropriate time to consume them.

[0498] Input: Food deterioration rate data.

[0499] Data calculation: Analyzes deterioration rate data and calculates the appropriate consumption timing.

[0500] Output: Consumption timing notification message.

[0501] Step 9:

[0502] The server uses an emotion recognition engine to recognize the emotional state of the user.

[0503] Input: Facial expression and voice data captured from a smartphone's camera and sensors.

[0504] Data calculation: Apply emotion recognition algorithms to analyze the user's emotional state.

[0505] Output: User emotion data.

[0506] Step 10:

[0507] The server adjusts the menu based on the user's emotional data.

[0508] Input: User emotion data.

[0509] Data calculation: Optimal menu adjustment based on emotional data.

[0510] Output: Adjusted menu data.

[0511] Step 11:

[0512] The server displays information about products in the store in real time.

[0513] Input: Camera image data from smart glasses or head-mounted displays, and user location information.

[0514] Data calculation: Analyzes camera footage and location information to display appropriate product information.

[0515] Output: Product information shown on the user's display.

[0516] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0517] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0518] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0519] [Second embodiment]

[0520] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0521] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0522] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0523] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0524] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0525] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0526] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0527] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0528] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0529] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0530] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0531] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0532] The system embodying this invention is designed to effectively manage food ingredients in a refrigerator, reduce food loss, and prevent food waste. This system links a video input device installed in the refrigerator, a server, and a user's mobile terminal.

[0533] 1. Acquiring video data

[0534] The server periodically acquires video data from a video input device (camera) inside the refrigerator, including the location and status of the food items stored inside the refrigerator.

[0535] 2. Recognizing ingredients and understanding their condition

[0536] The server analyzes the captured video data using image recognition technology to identify the type of food and its condition (such as the rate of deterioration and expiration date). For example, apples, milk, eggs, etc. in the refrigerator are automatically detected and their deterioration status is calculated.

[0537] 3. Calculation of usage priority

[0538] The server calculates the usage priority of each ingredient based on the type and condition data of the ingredient. Ingredients that are deteriorating or approaching their expiration date are given a higher priority.

[0539] 4. Visualization of usage priority

[0540] The device receives the usage priority information sent from the server and displays it on the refrigerator display or mobile device. This display uses color to indicate the usage priority of ingredients, allowing users to understand which ingredients are important at a glance. For example, high priority ingredients are displayed in red, medium priority in yellow, and low priority in green.

[0541] 5. Automatic menu generation

[0542] The server automatically generates menus using an AI model based on the priority data for ingredients. For example, it suggests a recipe for "French toast" that uses milk and eggs, which are the high priorities. This information is sent to the device and notified to the user.

[0543] 6. Listing ingredients you are lacking and making shopping suggestions

[0544] The server compares the current ingredient information with the generated menu, lists any missing ingredients, and suggests them as a shopping list. For example, if there is a shortage of butter for "French toast," that information is displayed on the user's mobile device.

[0545] 7. Prediction of deterioration rate and suggestion of consumption timing

[0546] The server monitors the rate at which food items deteriorate and notifies the user when it is time to consume them. For example, if it detects that an apple is deteriorating faster than normal, it will notify the user to consume it as soon as possible.

[0547] Specific examples

[0548] For example, when a user opens a refrigerator, the display shows the following information:

[0549] High priority (red): Milk (expiration date 2 days later)

[0550] Medium priority (yellow): Apple (changing color)

[0551] Low priority (green): Eggs (expiration date one week later)

[0552] Additionally, your mobile device will receive the following information:

[0553] French toast recipe (milk and eggs)

[0554] Missing Ingredient List: Butter

[0555] Consumption timing notification: Please consume the apples as soon as possible

[0556] In this way, users can manage the food in their refrigerators without waste and consume it in a planned manner, reducing food waste and household expenses.

[0557] The processing flow will be explained below.

[0558] Step 1:

[0559] The server periodically acquires video data from a video input device installed inside the refrigerator. Specifically, the server sends a command to the camera to capture the latest video data every hour and saves it in temporary storage.

[0560] Step 2:

[0561] The server analyzes the stored video data and identifies the type and condition of the ingredients. Specifically, the server processes the video data using a machine learning model to identify ingredients such as apples, milk, and eggs. It also analyzes the color and shape of each ingredient to estimate their deterioration rate and expiration date.

[0562] Step 3:

[0563] The server calculates the usage priority of each ingredient based on the identified ingredient data. Specifically, the server assigns a priority score to each ingredient according to its expiration date and deterioration state, and classifies it into three categories: high, medium, and low.

[0564] Step 4:

[0565] The server stores the calculated usage priority in a database and prepares the data for visualization. Specifically, the server records the priority score and analysis results for each ingredient in the database.

[0566] Step 5:

[0567] The device retrieves ingredient usage priority data from the server and displays it on the refrigerator display or mobile device. Specifically, the device sends an API request, receives the latest data from the server, and displays it in color. High-priority ingredients are displayed in red, medium-priority in yellow, and low-priority in green.

[0568] Step 6:

[0569] The server automatically generates the optimal menu using an AI model based on the usage priority data. Specifically, the server passes ingredient information to the AI ​​model as input and generates a recipe for "French toast" using milk and eggs, for example.

[0570] Step 7:

[0571] The server sends the generated menu to the device and notifies the user. Specifically, the server sends the menu information to the device via API and notifies the user through the mobile app's notification function.

[0572] Step 8:

[0573] The server compares the current ingredient information with the generated menu and lists any missing ingredients. Specifically, the server compares the list of ingredients needed for "French toast" with the current state of the refrigerator and identifies any missing items (e.g., butter).

[0574] Step 9:

[0575] The server creates a shopping list of the identified ingredients that are in short supply and sends it to the device. Specifically, the server creates a list of ingredients that are in short supply and sends it to the device via API.

[0576] Step 10:

[0577] The device displays the shopping list to the user. Specifically, the device displays "You are running low on the following ingredients: Butter" on the mobile app.

[0578] Step 11:

[0579] The server monitors the rate at which food deteriorates and notifies the user of the appropriate time to consume it. Specifically, the server continuously collects data such as the temperature, weight, and color of the food, and predicts the rate at which it will deteriorate. It notifies the user before deterioration progresses. For example, if it determines that an apple is changing color rapidly, the mobile app will notify the user, saying, "Please consume the apple as soon as possible."

[0580] Step 12:

[0581] The device receives the notification from the server and displays it to the user. Specifically, the device displays the notification from the server through the notification function of the mobile app, informing the user when it is time to consume the ingredients.

[0582] Example 1

[0583] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0584] In modern households, food waste is a common problem due to inadequate management of food in the refrigerator. It is particularly difficult to accurately grasp the rate at which food deteriorates and the priority of its use, and then create menu suggestions and shopping lists based on this information. It is also important to accurately predict when food will be consumed in order to avoid food waste. A system that can solve these problems and effectively manage food is needed.

[0585] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0586] In this invention, the server includes means for periodically acquiring video data from a video input device installed in the refrigerator, means for analyzing the video data to identify the type, condition, and deterioration rate of ingredients, means for calculating the use priority of each ingredient based on the identified information, means for visualizing the use priority and displaying it on the refrigerator's display device or a mobile information terminal, means for automatically generating a menu based on the use priority, means for identifying ingredients that are in short supply and displaying it as a shopping list, means for predicting the appropriate consumption timing based on the deterioration rate of ingredients and notifying the user, means for proposing a menu using an AI model that generates the ingredient use priority and its analysis data, and means for automatically activating the camera to capture video data when the refrigerator is opened and closed. This allows for effective management of ingredients in the refrigerator, reducing food waste and household expenses.

[0587] A "video input device" is a device that is installed inside a refrigerator and is used to acquire still image and video data.

[0588] "Video data" refers to image or video data that shows the location and condition of ingredients in the refrigerator, captured by a video input device.

[0589] "Type of food" refers to the specific classification of food stored in the refrigerator, such as fruit, dairy products, eggs, etc.

[0590] "Status of ingredients" refers to the current condition of ingredients in the refrigerator, including, for example, fresh, deteriorating, or close to expiry date.

[0591] "Deterioration rate" is an indicator of how quickly food deteriorates, tracking changes such as from green to yellow to red.

[0592] "Usage priority" indicates the priority for consumption, taking into consideration the rate at which food ingredients deteriorate and their expiration dates.

[0593] The "display device" is a display device that is installed inside the refrigerator and that allows the user to check the status of ingredients and the priority of their use.

[0594] A "personal digital assistant" is a device that a user can carry around and is used to display the status of ingredients and the priority for consumption.

[0595] A "generative AI model" is an artificial intelligence technology that uses pre-trained algorithms to automatically generate new information, such as recipes, based on data input.

[0596] The "purchase list" displays a list of ingredients that the user needs to purchase.

[0597] "Consumption timing" predicts the optimal time to consume food based on the condition of the food and the rate of deterioration.

[0598] The system embodying this invention is designed to effectively manage food ingredients in a refrigerator, reduce food loss, and prevent food waste. This system links a video input device installed in the refrigerator, a server, and a user's mobile information terminal.

[0599] Hardware and Software Configuration

[0600] 1. Video input device

[0601] A camera (video input device) installed inside the refrigerator acquires video data. This camera operates automatically every time the refrigerator door is opened or closed, capturing internal video. For example, a standard refrigerator camera or a smart camera can be used.

[0602] 2. Server

[0603] The server periodically receives and analyzes the video data sent from the video input device. The server uses the following technologies:

[0604] Image recognition technology: Using OpenCV and TensorFlow, the type, condition, and rate of deterioration of ingredients are identified from video data.

[0605] Generative AI model: For example, a generative AI model such as GPT-4 is used to automatically generate menus based on ingredient data.

[0606] 3. Mobile Information Terminals

[0607] The user's mobile information terminal has an application installed that displays the information sent from the server, and the terminal also displays the data on the refrigerator's display.

[0608] Data processing and calculation

[0609] 1. Acquiring video data

[0610] The server acquires video data from the camera inside the refrigerator, for example, at 9:00 AM and 6:00 PM every day. This data includes the location and condition of the food items stored inside the refrigerator.

[0611] 2. Recognizing ingredients and understanding their condition

[0612] The acquired video data is analyzed using image recognition technology (OpenCV and TensorFlow) to identify the type of food and its condition. For example, apples, milk, eggs, etc. in the refrigerator are automatically detected, and their deterioration status and expiration date are estimated.

[0613] 3. Calculation of usage priority

[0614] Based on the analyzed food data, the server calculates the usage priority of ingredients that are deteriorating or approaching their expiration date. For example, the priority is calculated on a scale of 0 to 100, with the higher the number, the higher the usage priority.

[0615] 4. Visualization of usage priority

[0616] The server calculates the usage priority and sends it to the terminal in JSON format, where it is displayed on the refrigerator display or mobile information terminal. For example, it is displayed in red (high priority), yellow (medium priority), or green (low priority).

[0617] 5. Automatic menu generation

[0618] The server uses a generative AI model based on the ingredient usage priority data to automatically generate a menu. An example of a prompt sentence is "Suggest a simple breakfast recipe using milk and eggs," and the generated recipe is provided to the user.

[0619] 6. Listing ingredients you are lacking and making shopping suggestions

[0620] The server compares the recipe with the information on ingredients in the refrigerator, lists any missing ingredients, and displays the list on the mobile information terminal as a shopping list. For example, if the user does not have butter for "French toast," the server notifies the user.

[0621] 7. Prediction of deterioration rate and suggestion of consumption timing

[0622] The server monitors the rate at which food items deteriorate and notifies the user when it is appropriate to consume them. For example, if it detects that an apple is deteriorating faster than normal, a notification will be sent to the mobile information terminal saying, "Please consume the apple as soon as possible."

[0623] Specific examples

[0624] For example, when a user opens a refrigerator, the display shows the following information:

[0625] High priority (red): Milk (expiration date 2 days later)

[0626] Medium priority (yellow): Apple (changing color)

[0627] Low priority (green): Eggs (expiration date one week later)

[0628] The mobile device will be notified of the following:

[0629] French toast recipe (milk and eggs)

[0630] Missing Ingredient List: Butter

[0631] Consumption timing notification: Please consume the apples as soon as possible

[0632] This system allows users to manage food in their refrigerators without waste and consume it in a planned manner, thereby reducing food waste and household expenses.

[0633] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0634] Step 1:

[0635] The server periodically acquires video data from a video input device installed inside the refrigerator.

[0636] Specific behavior:

[0637] The camera automatically activates and captures video data every time the refrigerator door is opened or closed, and the server collects this video data at 9:00 AM and 6:00 PM every day.

[0638] Input: Video data (images and videos showing the location and condition of ingredients in the refrigerator)

[0639] Output: Saved video data file

[0640] Step 2:

[0641] The server analyzes the acquired video data and identifies the type and condition of the ingredients.

[0642] Specific behavior:

[0643] The server uses image recognition technologies such as OpenCV and TensorFlow to classify and recognize ingredients from video data. For example, it analyzes video data to automatically identify ingredients such as apples, milk, and eggs and evaluate their deterioration status.

[0644] Input: Saved video data file

[0645] Output: Ingredient list (including type, condition, and deterioration rate)

[0646] Step 3:

[0647] The server calculates the use priority of each ingredient based on the identified ingredient data.

[0648] Specific behavior:

[0649] The server calculates the priority of ingredients based on their deterioration rate and expiration date, using a score ranging from 0 to 100. Ingredients that are deteriorating or nearing their expiration date are given a higher score.

[0650] Input: Ingredient list (including type, condition, and deterioration rate)

[0651] Output: Usage priority list

[0652] Step 4:

[0653] The server visualizes the usage priority and displays it on the terminal.

[0654] Specific behavior:

[0655] The server sends the usage priority list in JSON format to the terminal. The terminal receives this data and displays it on the refrigerator display and on the mobile information terminal. For example, each ingredient is displayed in red (high priority), yellow (medium priority), and green (low priority).

[0656] Input: Usage priority list

[0657] Output: Ingredient information displayed on a display and a mobile information terminal

[0658] Step 5:

[0659] The server automatically generates a menu based on the usage priority data.

[0660] Specific behavior:

[0661] The server uses a generative AI model such as GPT-4 to create a menu based on the priority of ingredients. For example, it inputs a prompt such as "Suggest a simple breakfast recipe using milk and eggs" into the AI ​​model and presents the generated recipe to the user.

[0662] Input: Usage priority list

[0663] Output: Recipe

[0664] Step 6:

[0665] The server compares the generated menu with the ingredient information, lists any missing ingredients, and displays them as a shopping list.

[0666] Specific behavior:

[0667] The server compares the list of ingredients needed for the created menu with the list of ingredients currently in the refrigerator to identify any missing ingredients. Based on this, it creates a shopping list and sends it to the user's mobile information terminal.

[0668] Input: Menu, ingredient list

[0669] Output: Shopping list

[0670] Step 7:

[0671] The server monitors the rate at which ingredients deteriorate and notifies the user of the appropriate time to consume them.

[0672] Specific behavior:

[0673] The server analyzes the rate at which food deteriorates and determines the appropriate time to consume it. For example, if it determines that an apple is deteriorating faster than normal, it sends a notification to the user's mobile information terminal saying, "Please consume the apple as soon as possible."

[0674] Input: Ingredient list, deterioration rate data

[0675] Output: Information message

[0676] (Application example 1)

[0677] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0678] In modern households, efficiently managing food in the refrigerator and reducing waste are major challenges. However, manual food management is time-consuming, and it is difficult to understand expiration dates and determine the appropriate time to consume food. In addition, food delivery services also require inventory management and optimization of consumption times, but achieving this requires advanced systems.

[0679] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0680] In this invention, the server includes means for periodically acquiring video data from a video input device installed in the refrigerator, means for analyzing the video data to identify the type, condition, and deterioration rate of ingredients, means for calculating the use priority of each ingredient based on the identified information, means for visualizing the use priority and displaying it on an information display device, means for automatically generating cooking recipes based on the use priority, means for identifying shortages of ingredients and displaying it as a shopping list, means for predicting appropriate consumption times based on the deterioration rate of ingredients and notifying the user, means for acquiring consumption data in the refrigerator in real time and automating menu suggestions according to the deterioration status of ingredients, and means for automatically generating cooking recipes based on the consumption priority of ingredients using a generative AI model.This improves the efficiency of food management in the refrigerator, enabling food waste reduction and consumption optimization.

[0681] The "video input device" is a device that is installed inside the refrigerator and is used to acquire video data from inside the refrigerator.

[0682] "Video data" is image information of the inside of the refrigerator acquired by a video input device.

[0683] "Type of foodstuff" is information that indicates the specific classification of food stored in the refrigerator.

[0684] "Condition" is information that indicates the freshness and degree of deterioration of the food material.

[0685] "Deterioration rate" is information indicating the rate at which food ingredients deteriorate.

[0686] "Use priority" is information indicating the consumption priority of each ingredient.

[0687] An "information display device" is a device for displaying acquired information to a user.

[0688] A "cooking recipe" is information that shows the steps for cooking a dish using ingredients.

[0689] A "shopping list" is a list of ingredients that are needed but in short supply.

[0690] "Consumption timing" is information indicating the appropriate time to consume the food material.

[0691] "Consumption data" is information relating to the usage of ingredients.

[0692] "Menu suggestion" means suggesting cooking recipes based on the current ingredients.

[0693] A "generative AI model" is a model that uses artificial intelligence to analyze data and generate results.

[0694] The system embodying this invention is designed to effectively manage food ingredients in a refrigerator and reduce food waste. This system links a video input device installed in the refrigerator, a server, and the user's mobile device.

[0695] The server periodically acquires video data from a video input device (camera) installed inside the refrigerator. The server then analyzes the video data and uses image recognition algorithms to identify the type, condition, and deterioration rate of ingredients. This allows the server to detect the specific classification of ingredients in the refrigerator and their freshness or degree of deterioration.

[0696] The server then calculates the usage priority of each ingredient based on the identified information. Ingredients with a high usage priority are those that deteriorate quickly or are close to their expiration date. This information is displayed on an information display device such as a refrigerator display or a mobile device.

[0697] The server automatically generates cooking recipes based on the priority of use. The generated recipes are sent to the terminal and displayed to the user. The server also identifies missing ingredients and displays them as a shopping list. This information is also sent to the user's terminal in real time.

[0698] Furthermore, the server predicts the best time to consume ingredients based on the rate at which ingredients deteriorate and notifies the user, allowing them to consume ingredients at the appropriate time.The server also uses a generative AI model to automatically generate cooking recipes based on ingredient consumption priorities, automating menu suggestions.

[0699] For example, a camera inside a refrigerator captures video data and sends it to a server. The server analyzes the data and determines that the ingredients are A (deteriorating quickly), B (nearly expiring), and C (fresh). Based on this, an AI model generates recipes using ingredients A and B, which have a high priority for use, and notifies the user. A recipe such as "French toast" is then suggested, and butter, which is in short supply, is also displayed on the shopping list. Notifications such as "Please consume the apples soon" are also given to remind users when to consume them.

[0700] An example of a prompt to input to a generative AI model would be:

[0701] High priority ingredients: milk, apples

[0702] Medium priority ingredients: Eggs

[0703] Please suggest a recipe using these ingredients.

[0704] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0705] Step 1:

[0706] The server periodically acquires video data from a video input device (camera) installed inside the refrigerator.

[0707] Input: Video data acquired from a camera inside the refrigerator

[0708] Data processing: Send the video data to the server

[0709] Output: Video data stored on the server

[0710] Step 2:

[0711] The server analyzes the captured video data and uses image recognition algorithms to identify the type, condition, and rate of deterioration of the food.

[0712] Input: Video data stored on the server

[0713] Data Computing: Using image recognition algorithms to identify the type, condition, and rate of deterioration of ingredients

[0714] Output: Information on the type, condition, and deterioration rate of identified ingredients

[0715] Step 3:

[0716] The server calculates the use priority of each ingredient based on the identified information.

[0717] Input: Information on the type, condition, and deterioration rate of ingredients

[0718] Data calculation: Calculate the usage priority taking into account the rate of deterioration and expiration date

[0719] Output: Usage priority of each ingredient

[0720] Step 4:

[0721] The server visualizes the usage priority and displays it on an information display device.

[0722] Input: Usage priority of each ingredient

[0723] Data processing: Visualization of usage priority (e.g., color-coding)

[0724] Output: Usage priority displayed on the information display

[0725] Step 5:

[0726] The server automatically generates cooking recipes based on the usage priority and notifies the terminal.

[0727] Input: Usage priority of each ingredient

[0728] Data Computing: Automatically generating cooking recipes using generative AI models

[0729] Output: Cooking recipe notified to the device

[0730] Step 6:

[0731] The server identifies missing ingredients and displays them as a shopping list.

[0732] Input: Cooking recipe

[0733] Data calculation: Compare the current ingredient information with the generated menu and list any missing ingredients

[0734] Output: Shopping list displayed on the terminal

[0735] Step 7:

[0736] The server predicts the appropriate time to consume the food based on the rate at which the food deteriorates and notifies the user.

[0737] Input: Food deterioration rate and consumption priority

[0738] Data calculation: Predicting the optimal consumption timing based on the rate of deterioration and usage priority

[0739] Output: Consumption timing suggestions sent to the device

[0740] This will improve the efficiency of food management in the refrigerator, reducing food waste and optimizing consumption.

[0741] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0742] The system embodying this invention recognizes the user's emotional state by combining an emotion engine and customizes services based on this to effectively manage ingredients in a refrigerator. The system includes a video input device installed in the refrigerator, a server, a user's mobile terminal, and an emotion engine.

[0743] 1. Acquiring video data

[0744] The server periodically acquires video data from a video input device (camera) inside the refrigerator, including the type and condition of the food stored in the refrigerator.

[0745] 2. Recognizing ingredients and understanding their condition

[0746] The server analyzes the captured video data using image recognition technology to identify the type of food and its condition (such as the rate of deterioration and expiration date). For example, apples, milk, eggs, etc. in the refrigerator are automatically detected and their deterioration status is calculated.

[0747] 3. Calculation of usage priority

[0748] The server calculates the usage priority of each ingredient based on the type and condition data of the ingredient. Ingredients that are deteriorating or approaching their expiration date are given a higher priority.

[0749] 4. Visualization of usage priority

[0750] The device receives the usage priority information sent from the server and displays it on the refrigerator display or mobile device. This display uses color to indicate the usage priority of ingredients, allowing users to understand which ingredients are important at a glance. For example, high priority ingredients are displayed in red, medium priority in yellow, and low priority in green.

[0751] 5. Automatic menu generation

[0752] The server automatically generates menus using an AI model based on the priority data for ingredients. For example, it suggests a recipe for "French toast" that uses milk and eggs, which are the high priorities. This information is sent to the device and notified to the user.

[0753] 6. Listing ingredients you are lacking and making shopping suggestions

[0754] The server compares the current ingredient information with the generated menu, lists any missing ingredients, and suggests them as a shopping list. For example, if there is a shortage of butter for "French toast," that information is displayed on the user's mobile device.

[0755] 7. Prediction of deterioration rate and suggestion of consumption timing

[0756] The server monitors the rate at which food items deteriorate and notifies the user when it is time to consume them. For example, if it detects that an apple is deteriorating faster than normal, it will notify the user to consume it as soon as possible.

[0757] 8. User Emotion Recognition

[0758] The server uses an emotion engine to recognize the user's emotions. This emotion data is obtained, for example, from a smartphone camera or other sensors. It analyzes the user's facial expressions and voice and stores the results as emotion data.

[0759] 9. Emotionally-based menu adjustments

[0760] The server adjusts the menu based on the user's emotional data. For example, if the user is under stress, it will suggest simple and relaxing meals.

[0761] 10. Emotion-Based Information Display

[0762] The device customizes the information shown on the refrigerator display or mobile device depending on the user's emotional state. For example, if the emotion engine recognizes that the user is tired, it will prioritize showing easy-to-prepare recipes.

[0763] Specific examples

[0764] For example, when a user opens a refrigerator, the display shows the following information:

[0765] High priority (red): Milk (expiration date 2 days later)

[0766] Medium priority (yellow): Apple (changing color)

[0767] Low priority (green): Eggs (expiration date one week later)

[0768] When a user smiles through the camera in front of the refrigerator, the emotion engine recognizes this information. The server determines that the user is relaxed and suggests an easy and enjoyable recipe for "French toast."

[0769] Additionally, your mobile device will display the following information:

[0770] French toast recipe (milk and eggs)

[0771] Missing Ingredient List: Butter

[0772] Consumption timing notification: Please consume the apples as soon as possible

[0773] In this way, users can manage the food in their refrigerators without waste and consume it in a planned manner. In addition, cooking suggestions are made based on the user's emotions, allowing for more personalized food management. This system will reduce food waste and household expenses.

[0774] The processing flow will be explained below.

[0775] Step 1:

[0776] The server periodically acquires video data from a video input device installed inside the refrigerator. Specifically, the server sends a command to the camera to capture the latest video data every hour and saves it in temporary storage.

[0777] Step 2:

[0778] The server analyzes the stored video data and identifies the type and condition of the ingredients. Specifically, the server processes the video data using a machine learning model to identify ingredients such as apples, milk, and eggs. It also analyzes the color and shape of each ingredient to estimate their deterioration rate and expiration date.

[0779] Step 3:

[0780] The server calculates the usage priority of each ingredient based on the identified ingredient data. Specifically, the server assigns a priority score to each ingredient according to its expiration date and deterioration state, and classifies it into three categories: high, medium, and low.

[0781] Step 4:

[0782] The server stores the calculated usage priority in a database and prepares the data for visualization. Specifically, the server records the priority score and analysis results for each ingredient in the database.

[0783] Step 5:

[0784] The device retrieves ingredient usage priority data from the server and displays it on the refrigerator display or mobile device. Specifically, the device sends an API request, receives the latest data from the server, and displays it in color. High-priority ingredients are displayed in red, medium-priority in yellow, and low-priority in green.

[0785] Step 6:

[0786] The server uses an AI model to automatically generate the optimal menu based on the usage priority data. Specifically, the server passes ingredient information to the AI ​​model as input and generates a recipe for "French toast" using milk and eggs, for example.

[0787] Step 7:

[0788] The server sends the generated menu to the device and notifies the user. Specifically, the server sends the menu information to the device via API and notifies the user through the mobile app's notification function.

[0789] Step 8:

[0790] The server compares the current ingredient information with the generated menu and lists any missing ingredients. Specifically, the server compares the list of ingredients needed for "French toast" with the current state of the refrigerator and identifies any missing items (e.g., butter).

[0791] Step 9:

[0792] The server creates a shopping list of the identified ingredients that are in short supply and sends it to the device. Specifically, the server creates a list of ingredients that are in short supply and sends it to the device via API.

[0793] Step 10:

[0794] The device displays the shopping list to the user. Specifically, the device displays "You are running low on the following ingredients: Butter" on the mobile app.

[0795] Step 11:

[0796] The server monitors the rate at which food deteriorates and notifies the user of the appropriate time to consume it. Specifically, the server continuously collects data such as the temperature, weight, and color of the food, and predicts the rate at which it will deteriorate. It notifies the user before deterioration progresses. For example, if it determines that an apple is changing color rapidly, the mobile app will notify the user, saying, "Please consume the apple as soon as possible."

[0797] Step 12:

[0798] The device receives the notification from the server and displays it to the user. Specifically, the device displays the notification from the server through the notification function of the mobile app, informing the user when it is time to consume the ingredients.

[0799] Step 13:

[0800] The server uses an emotion engine to recognize the user's emotions. Specifically, the server analyzes data acquired from the smartphone's camera and microphone, and generates emotion data from the user's facial expressions and voice.

[0801] Step 14:

[0802] The server adjusts the generated menu based on the emotional data. Specifically, if the server determines that the user is under stress, it will prioritize suggesting ingredients that have a relaxing effect and easy-to-cook recipes.

[0803] Step 15:

[0804] The device customizes the information displayed on the refrigerator display or mobile device based on the emotional data. Specifically, if the device recognizes that the user is tired, it will prioritize easy-to-prepare recipes. If the user's emotional state is positive, it will suggest new recipes to try.

[0805] Example flow

[0806] For example, when a user opens a refrigerator, the display shows the following information:

[0807] High priority (red): Milk (expiration date 2 days later)

[0808] Medium priority (yellow): Apple (changing color)

[0809] Low priority (green): Eggs (expiration date one week later)

[0810] When a user smiles through the camera in front of the refrigerator, the emotion engine recognizes this information. The server determines that the user is relaxed and suggests an easy and enjoyable recipe for "French toast."

[0811] Additionally, your mobile device will display the following information:

[0812] French toast recipe (milk and eggs)

[0813] Missing Ingredient List: Butter

[0814] Consumption timing notification: Please consume the apples as soon as possible

[0815] In this way, users can manage the food in their refrigerators without waste and consume it in a planned manner. In addition, cooking suggestions are made based on the user's emotions, allowing for more personalized food management. This system will reduce food waste and household expenses.

[0816] Example 2

[0817] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0818] While conventional refrigerator food management systems can grasp the type and condition of ingredients, they have difficulty managing ingredients and suggesting menus that match the user's emotions and individual lifestyles. Furthermore, while they can predict the rate of deterioration and calculate usage priorities, they do not support adjusting menus or suggesting appropriate consumption times based on the user's emotions. As a result, user satisfaction is low, food waste is often generated, and unnecessary stress is often generated. To solve these issues, a system that can recognize the user's emotional state and customize services based on that state was needed.

[0819] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for periodically acquiring video data from a video input device installed in the refrigerator, means for analyzing the video data to identify the type, condition, and deterioration rate of ingredients, means for calculating the use priority of each ingredient based on the identified information, means for visualizing the use priority and displaying it on the refrigerator's display device or a mobile information terminal, means for generating a menu of dishes based on the use priority, means for identifying shortages of ingredients and displaying it as a shopping list, means for predicting appropriate consumption times based on the deterioration rate of ingredients and notifying the user, means for recognizing the user's emotions using an emotion engine, means for adjusting the menu based on the user's emotions, and means for customizing display information based on the user's emotions. This enables personalized ingredient management and menu suggestions based on the user's emotional state, thereby reducing ingredient waste and improving user satisfaction.

[0820] The "video input device installed inside the refrigerator" refers to a camera or other image capturing device that is installed inside the refrigerator and that captures video data of the food ingredients inside the refrigerator.

[0821] "Video data" is digital data that includes image information of food items in the refrigerator.

[0822] "Type of food" refers to the specific item names of food stored in the refrigerator, such as apples, milk, eggs, etc.

[0823] "Condition of ingredients" refers to the current quality and degree of deterioration of ingredients in the refrigerator.

[0824] "Deterioration rate" is an index that indicates how quickly each food ingredient stored in a refrigerator deteriorates.

[0825] "Usage priority" is calculated based on the priority of using food items in the refrigerator, taking into account factors such as the rate of deterioration and expiration date.

[0826] A "display device" is a display or monitor attached to a refrigerator and is used to visually display information about ingredients and menu items.

[0827] A "mobile information terminal" is an electronic device such as a smartphone or tablet that can be carried by a user and that displays ingredient information and menus.

[0828] "Cooking Menu" is a meal menu suggested based on the ingredients in your refrigerator.

[0829] A "shopping list" is a list of ingredients that need to be purchased based on the ingredients in the refrigerator and the menu.

[0830] "Consumption timing" refers to the predicted optimal time to consume a food item based on its deterioration rate.

[0831] An "emotion engine" is an analysis device or software that analyzes a user's emotional state, stores it as data, and uses it for subsequent processing.

[0832] "User emotion" means the user's psychological state analyzed through the emotion engine.

[0833] "Customizing display information" means adjusting the information displayed on the refrigerator display device or mobile information terminal based on the user's emotional state and priorities.

[0834] The system embodying this invention recognizes the user's emotional state by combining an emotion engine and customizes services based on this to effectively manage ingredients in a refrigerator. The system includes a video input device installed in the refrigerator, a server, a user's mobile terminal, and an emotion engine.

[0835] Hardware and Software Configuration

[0836] 1. Video input device (camera) inside the refrigerator

[0837] A camera is installed inside the refrigerator to periodically capture video data of the ingredients. The camera can take high-resolution images and send the data to a server every hour, for example.

[0838] 2. Server

[0839] The server acquires, analyzes, stores, and processes video data. Specific software used for image recognition is OpenCV and TensorFlow. The emotion engine is used to analyze user emotion data.

[0840] 3. Terminal (refrigerator display or mobile terminal)

[0841] The terminal displays the information sent from the server, visually conveying information such as the priority of ingredients, menus, shopping lists, and consumption timing to the user.

[0842] Data processing and calculation

[0843] 1. Acquiring video data

[0844] The server periodically acquires video data from the video input device (camera) inside the refrigerator, including the type and condition of the food stored in the refrigerator.

[0845] 2. Recognizing ingredients and understanding their condition

[0846] The server uses image recognition software (e.g., OpenCV or TensorFlow) to analyze the captured video data, thereby identifying the type and condition of the food (rate of deterioration and expiration date).

[0847] 3. Calculation of usage priority

[0848] The server calculates the usage priority of each ingredient based on the type and condition data of the ingredient, assigning a higher priority to ingredients that are deteriorating or approaching their expiration date.

[0849] 4. Visualization of usage priority

[0850] The device receives the usage priority information sent from the server and displays it on the refrigerator display or mobile device. The priority is indicated by a color, allowing the user to understand the importance at a glance.

[0851] 5. Automatic menu generation

[0852] The server automatically generates menus using a generative AI model (e.g., GPT-4) based on the ingredient usage priority data, and suggests recipes that include high-priority ingredients.

[0853] 6. Listing ingredients you are lacking and making shopping suggestions

[0854] The server compares the current ingredient information with the generated menu, lists any missing ingredients, and suggests a shopping list to the user.

[0855] 7. Prediction of deterioration rate and suggestion of consumption timing

[0856] The server monitors the rate at which ingredients deteriorate and notifies the user when it is time to consume them.

[0857] 8. User Emotion Recognition

[0858] The server uses an emotion engine to recognize the user's emotions, and this data is obtained from the smartphone's camera and other sensors.

[0859] 9. Emotionally-based menu adjustments

[0860] The server uses an AI model to adjust the menu based on the user's emotional data.

[0861] 10. Emotion-Based Information Display

[0862] The device customizes the information displayed on the refrigerator display and mobile device based on the user's emotional state.

[0863] Specific examples

[0864] For example, when a user opens a refrigerator, the display shows the following information:

[0865] High priority (red): Milk (expiration date 2 days later)

[0866] Medium priority (yellow): Apple (changing color)

[0867] Low priority (green): Eggs (expiration date one week later)

[0868] When the user smiles through the camera in front of the refrigerator, the emotion engine recognizes this information. The server determines that the user is relaxed and suggests a simple and enjoyable recipe for "French toast." The mobile device also displays the following information:

[0869] French toast recipe (milk and eggs)

[0870] Missing Ingredient List: Butter

[0871] Consumption timing notification: Please consume the apples as soon as possible

[0872] Example prompts to input to the generative AI model

[0873] "Suggest a simple recipe using the milk and eggs in the fridge. The user is currently relaxed."

[0874] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0875] Step 1:

[0876] The server periodically acquires video data from a video input device (camera) installed inside the refrigerator.

[0877] Input: Footage from inside a refrigerator.

[0878] Data processing / calculation: The camera takes high-resolution images every hour and sends them to the server as digital data.

[0879] Output: Video data of food in the refrigerator.

[0880] Step 2:

[0881] The server uses image recognition software (e.g., OpenCV or TensorFlow) to analyze the captured video data, thereby identifying the type and condition of the food ingredients (rate of deterioration and expiration date).

[0882] Input: Video data of food in a refrigerator.

[0883] Data processing / calculation: Detects food ingredients such as apples, milk, and eggs from the video and calculates the degree of deterioration and expiration date of each.

[0884] Output: Data on the type, condition and deterioration rate of identified ingredients.

[0885] Step 3:

[0886] The server calculates the use priority of each ingredient based on the type and state data of the ingredient.

[0887] Input: Data on the type, condition and deterioration rate of identified ingredients.

[0888] Data processing / calculation: A mathematical algorithm is used to assign a priority score based on the degree of deterioration and expiration date. Food items that are in an advanced state of deterioration or approaching their expiration date are given a higher priority.

[0889] Output: Usage priority data for each ingredient.

[0890] Step 4:

[0891] The terminal receives the information on the usage priority sent from the server and displays it on the refrigerator's display device or the mobile terminal.

[0892] Input: Usage priority data for each ingredient.

[0893] Data processing / calculation: Usage priority is displayed in color (e.g., high priority is red, medium priority is yellow, and low priority is green).

[0894] Output: Visually displayed ingredient usage priority information.

[0895] Step 5:

[0896] The server automatically generates menus using a generative AI model (e.g., GPT-4) based on ingredient usage priority data.

[0897] Input: Usage priority data for each ingredient.

[0898] Data processing / calculation: Send the prompt "Please suggest a simple recipe using milk and eggs. The user is currently in a relaxed state" to the generative AI model and receive the generated recipe.

[0899] Output: Automatically generated menu information.

[0900] Step 6:

[0901] The server compares the current ingredient information with the generated menu, lists any missing ingredients, and suggests a shopping list to the user.

[0902] Input: Current ingredient information, automatically generated menu information.

[0903] Data processing / calculation: Matching ingredient information with the menu, identifying missing ingredients, and generating a shopping list.

[0904] Output: Shopping list.

[0905] Step 7:

[0906] The server monitors the rate at which ingredients deteriorate and notifies the user when it is time to consume the ingredients.

[0907] Input: Food deterioration rate data.

[0908] Data processing / calculation: Analyzes deterioration rate data and calculates the appropriate consumption timing. For example, generates a notification saying "Please consume the apples as soon as possible."

[0909] Output: Consumption timing notification.

[0910] Step 8:

[0911] The server uses an emotion engine to recognize the user's emotions, and this data is obtained from the smartphone's camera and other sensors.

[0912] Input: Emotion data from smartphone cameras and sensors.

[0913] Data processing / calculation: Analyze the user's facial expressions and voice, and save the results as emotional data.

[0914] Output: User emotion data.

[0915] Step 9:

[0916] The server uses an AI model to adjust the menu based on the user's emotional data.

[0917] Input: User emotion data, automatically generated menu information.

[0918] Data processing / calculation: Emotional data is taken into account and menus are adjusted to be more relaxing (e.g., simple dishes when under stress).

[0919] Output: Menu information tailored based on emotions.

[0920] Step 10:

[0921] The terminal customizes the information displayed on the refrigerator display and the mobile terminal according to the user's emotional state.

[0922] Input: User emotion data.

[0923] Data processing / calculation: Based on emotional data, information to be displayed preferentially (e.g., easy recipes) is selected.

[0924] Output: Customized display information.

[0925] (Application example 2)

[0926] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0927] Conventional refrigerator food management systems focus on calculating the priority of ingredients based on their condition and deterioration rate and notifying the user of this priority. However, because information is provided unilaterally without considering the user's emotional state, there is a problem in that it is not possible to display optimal menu suggestions or product information for the user. The present invention aims to more effectively manage ingredients and suggest products in stores by recognizing the user's emotional state in real time and providing personalized services based on that.

[0928] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0929] In this invention, the server includes means for periodically acquiring video data from a video input device installed in the refrigerator, means for analyzing the video data to identify the type, condition, and deterioration rate of ingredients, means for calculating the use priority of each ingredient based on the identified information, means for visualizing the use priority and displaying it on the refrigerator display or a mobile terminal, means for automatically generating a recipe based on the use priority, means for identifying shortages of ingredients and displaying it as a shopping list, means for predicting appropriate consumption times based on the deterioration rate of ingredients and notifying the user, means including an emotion recognition engine for recognizing the user's emotional state, means for adjusting the recipe based on the user's emotional data, and means for displaying information about products in the store in real time. This enables optimal ingredient management and personalized product suggestions according to the user's emotional state.

[0930] The "video input device" is a device that is installed inside the refrigerator and periodically acquires video data of the food ingredients inside.

[0931] An "image recognition algorithm" is a calculation method for analyzing acquired video data and identifying the type, condition, and rate of deterioration of ingredients.

[0932] "Use priority" is an index that indicates the priority of consumption based on the type, condition, and deterioration rate of food ingredients.

[0933] A "display" is a device for visually displaying information, and may be installed on the surface of a refrigerator or on a mobile device.

[0934] A "mobile terminal" is a portable electronic device such as a smartphone or tablet that receives and displays information.

[0935] A "menu" is a plan of dishes using ingredients, and is automatically generated based on the priority of using ingredients.

[0936] A "shopping list" is a list of information that lists ingredients that are in short supply and is presented for purchase.

[0937] An "emotion recognition engine" is a software or hardware element that analyzes a user's facial expressions and voice and recognizes their emotional state.

[0938] "User emotion data" is data that indicates the temporary or persistent emotional state of the user, obtained through analysis by the emotion recognition engine.

[0939] "Product suggestion" is an act or system that presents product information in a store based on the user's emotional state.

[0940] "Real-time" is a concept that indicates that information is processed immediately and provided without delay.

[0941] In this invention, the following system is implemented to realize the management of ingredients in a refrigerator and the provision of personalized services based on the user's emotional state.

[0942] First, the server periodically acquires video data from a video input device installed inside the refrigerator. An image recognition algorithm is used to analyze this video data. The image recognition algorithm identifies the type, condition, and deterioration rate of ingredients, and calculates the usage priority of each ingredient based on this information. The usage priority is visualized on the refrigerator or mobile device display, allowing users to understand the status of ingredients at a glance.

[0943] Next, the server automatically generates a menu of dishes based on the calculated usage priority. This automatic generation uses a generative AI model. The generated menu is notified to the mobile device and displayed. The server also identifies any missing ingredients and displays them as a shopping list on the mobile device.

[0944] Furthermore, the server monitors the rate at which food items deteriorate and notifies the user of the appropriate time to consume them. For example, if an apple is deteriorating faster than usual, the server will notify the user, "Consume the apple sooner." This information encourages the user to consume food appropriately according to their lifestyle.

[0945] Next, an emotion recognition engine recognizes the user's emotional state. This engine captures and analyzes facial and voice data from the smartphone's camera and other sensors. Based on the user's emotional data, the system adjusts the menu accordingly. For example, if the user is tired, it will suggest simple, relaxing dishes.

[0946] Smart glasses and head-mounted displays (HMDs) are also used to display real-time product information in stores. When a user approaches or picks up a particular product, information about that product and recommended recipes are displayed.

[0947] As a concrete example, suppose a user wearing smart glasses is walking through a supermarket and stops in front of a bottle of milk. The camera in the smart glasses captures the user's smile, and the system determines that the user is relaxed and enjoying themselves. In this case, the smart glasses will display "Fresh milk. Recommended recipe: Panna cotta."

[0948] Prompt Sentence Examples

[0949] "A camera in smart glasses captures the user's facial expressions and suggests supermarket products to the user based on their emotional state. The system analyzes the user's emotional data in real time and displays appropriate product and recipe information based on the results."

[0950] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0951] Flow of the system program that realizes the application example

[0952] Step 1:

[0953] The server periodically acquires video data from a video input device installed inside the refrigerator.

[0954] Input: Video data from a video input device.

[0955] Data processing: Video data is converted to the appropriate resolution and saved as an image file on the server.

[0956] Output: Video data in image file format.

[0957] Step 2:

[0958] The server analyzes the captured video data and uses image recognition algorithms to identify the type, condition and rate of deterioration of the food.

[0959] Input: Stored video data.

[0960] Data processing: Apply image recognition algorithms to identify the type, condition, and rate of deterioration of ingredients.

[0961] Output: Identification data including type, condition and rate of deterioration of the food material.

[0962] Step 3:

[0963] The server calculates the use priority of each ingredient based on the identified information.

[0964] Input: Identification data for food type, condition, and rate of deterioration.

[0965] Data calculation: Calculates the usage priority score based on the deterioration rate and condition.

[0966] Output: Usage priority score for each ingredient.

[0967] Step 4:

[0968] The terminal receives the usage priority information sent from the server and displays it on the refrigerator display or mobile terminal.

[0969] Input: Usage priority score.

[0970] Data transformation: Converting usage priority scores into visual formats such as color coding.

[0971] Output: Visually enhanced usage priority information.

[0972] Step 5:

[0973] The server automatically generates a menu of dishes based on the priority of use.

[0974] Input: Usage priority score for each ingredient.

[0975] Data computation: Uses generative AI models to generate optimal menus.

[0976] Output: The generated menu data.

[0977] Step 6:

[0978] The terminal receives the generated menu data and notifies the mobile terminal.

[0979] Input: Generated menu data.

[0980] Data processing: Convert menu data into notification format.

[0981] Output: Notification message to mobile device.

[0982] Step 7:

[0983] The server compares the current ingredient information with the generated menu, identifies any missing ingredients, and displays them as a shopping list.

[0984] Input: Current ingredient information, generated menu data.

[0985] Data calculation: Compare the ingredients needed for the menu with the ingredients currently available.

[0986] Output: A list of missing ingredients.

[0987] Step 8:

[0988] The server monitors the rate at which ingredients deteriorate and notifies the user of the appropriate time to consume them.

[0989] Input: Food deterioration rate data.

[0990] Data calculation: Analyzes deterioration rate data and calculates the appropriate consumption timing.

[0991] Output: Consumption timing notification message.

[0992] Step 9:

[0993] The server uses an emotion recognition engine to recognize the emotional state of the user.

[0994] Input: Facial expression and voice data captured from a smartphone's camera and sensors.

[0995] Data calculation: Apply emotion recognition algorithms to analyze the user's emotional state.

[0996] Output: User emotion data.

[0997] Step 10:

[0998] The server adjusts the menu based on the user's emotional data.

[0999] Input: User emotion data.

[1000] Data calculation: Optimal menu adjustment based on emotional data.

[1001] Output: Adjusted menu data.

[1002] Step 11:

[1003] The server displays information about products in the store in real time.

[1004] Input: Camera image data from smart glasses or head-mounted displays, and user location information.

[1005] Data calculation: Analyzes camera footage and location information to display appropriate product information.

[1006] Output: Product information shown on the user's display.

[1007] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1008] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1009] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1010] [Third embodiment]

[1011] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1012] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1013] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1014] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1015] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1016] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1017] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1018] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1019] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1020] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1021] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1022] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1023] The system embodying this invention is designed to effectively manage food ingredients in a refrigerator, reduce food loss, and prevent food waste. This system links a video input device installed in the refrigerator, a server, and a user's mobile terminal.

[1024] 1. Acquiring video data

[1025] The server periodically acquires video data from a video input device (camera) inside the refrigerator, including the location and status of the food items stored inside the refrigerator.

[1026] 2. Recognizing ingredients and understanding their condition

[1027] The server analyzes the captured video data using image recognition technology to identify the type of food and its condition (such as the rate of deterioration and expiration date). For example, apples, milk, eggs, etc. in the refrigerator are automatically detected and their deterioration status is calculated.

[1028] 3. Calculation of usage priority

[1029] The server calculates the usage priority of each ingredient based on the type and condition data of the ingredient. Ingredients that are deteriorating or approaching their expiration date are given a higher priority.

[1030] 4. Visualization of usage priority

[1031] The device receives the usage priority information sent from the server and displays it on the refrigerator display or mobile device. This display uses color to indicate the usage priority of ingredients, allowing users to understand which ingredients are important at a glance. For example, high priority ingredients are displayed in red, medium priority in yellow, and low priority in green.

[1032] 5. Automatic menu generation

[1033] The server automatically generates menus using an AI model based on the priority data for ingredients. For example, it suggests a recipe for "French toast" that uses milk and eggs, which are the high priorities. This information is sent to the device and notified to the user.

[1034] 6. Listing ingredients you are lacking and making shopping suggestions

[1035] The server compares the current ingredient information with the generated menu, lists any missing ingredients, and suggests them as a shopping list. For example, if there is a shortage of butter for "French toast," that information is displayed on the user's mobile device.

[1036] 7. Prediction of deterioration rate and suggestion of consumption timing

[1037] The server monitors the rate at which food items deteriorate and notifies the user when it is time to consume them. For example, if it detects that an apple is deteriorating faster than normal, it will notify the user to consume it as soon as possible.

[1038] Specific examples

[1039] For example, when a user opens a refrigerator, the display shows the following information:

[1040] High priority (red): Milk (expiration date 2 days later)

[1041] Medium priority (yellow): Apple (changing color)

[1042] Low priority (green): Eggs (expiration date one week later)

[1043] Additionally, your mobile device will receive the following information:

[1044] French toast recipe (milk and eggs)

[1045] Missing Ingredient List: Butter

[1046] Consumption timing notification: Please consume the apples as soon as possible

[1047] In this way, users can manage the food in their refrigerators without waste and consume it in a planned manner, reducing food waste and household expenses.

[1048] The processing flow will be explained below.

[1049] Step 1:

[1050] The server periodically acquires video data from a video input device installed inside the refrigerator. Specifically, the server sends a command to the camera to capture the latest video data every hour and saves it in temporary storage.

[1051] Step 2:

[1052] The server analyzes the stored video data and identifies the type and condition of the ingredients. Specifically, the server processes the video data using a machine learning model to identify ingredients such as apples, milk, and eggs. It also analyzes the color and shape of each ingredient to estimate their deterioration rate and expiration date.

[1053] Step 3:

[1054] The server calculates the usage priority of each ingredient based on the identified ingredient data. Specifically, the server assigns a priority score to each ingredient according to its expiration date and deterioration state, and classifies it into three categories: high, medium, and low.

[1055] Step 4:

[1056] The server stores the calculated usage priority in a database and prepares the data for visualization. Specifically, the server records the priority score and analysis results for each ingredient in the database.

[1057] Step 5:

[1058] The device retrieves ingredient usage priority data from the server and displays it on the refrigerator display or mobile device. Specifically, the device sends an API request, receives the latest data from the server, and displays it in color. High-priority ingredients are displayed in red, medium-priority in yellow, and low-priority in green.

[1059] Step 6:

[1060] The server automatically generates the optimal menu using an AI model based on the usage priority data. Specifically, the server passes ingredient information to the AI ​​model as input and generates a recipe for "French toast" using milk and eggs, for example.

[1061] Step 7:

[1062] The server sends the generated menu to the device and notifies the user. Specifically, the server sends the menu information to the device via API and notifies the user through the mobile app's notification function.

[1063] Step 8:

[1064] The server compares the current ingredient information with the generated menu and lists any missing ingredients. Specifically, the server compares the list of ingredients needed for "French toast" with the current state of the refrigerator and identifies any missing items (e.g., butter).

[1065] Step 9:

[1066] The server creates a shopping list of the identified ingredients that are in short supply and sends it to the device. Specifically, the server creates a list of ingredients that are in short supply and sends it to the device via API.

[1067] Step 10:

[1068] The device displays the shopping list to the user. Specifically, the device displays "You are running low on the following ingredients: Butter" on the mobile app.

[1069] Step 11:

[1070] The server monitors the rate at which food deteriorates and notifies the user of the appropriate time to consume it. Specifically, the server continuously collects data such as the temperature, weight, and color of the food, and predicts the rate at which it will deteriorate. It notifies the user before deterioration progresses. For example, if it determines that an apple is changing color rapidly, the mobile app will notify the user, saying, "Please consume the apple as soon as possible."

[1071] Step 12:

[1072] The device receives the notification from the server and displays it to the user. Specifically, the device displays the notification from the server through the notification function of the mobile app, informing the user when it is time to consume the ingredients.

[1073] Example 1

[1074] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1075] In modern households, food waste is a common problem due to inadequate management of food in the refrigerator. It is particularly difficult to accurately grasp the rate at which food deteriorates and the priority of its use, and then create menu suggestions and shopping lists based on this information. It is also important to accurately predict when food will be consumed in order to avoid food waste. A system that can solve these problems and effectively manage food is needed.

[1076] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1077] In this invention, the server includes means for periodically acquiring video data from a video input device installed in the refrigerator, means for analyzing the video data to identify the type, condition, and deterioration rate of ingredients, means for calculating the use priority of each ingredient based on the identified information, means for visualizing the use priority and displaying it on the refrigerator's display device or a mobile information terminal, means for automatically generating a menu based on the use priority, means for identifying ingredients that are in short supply and displaying it as a shopping list, means for predicting the appropriate consumption timing based on the deterioration rate of ingredients and notifying the user, means for proposing a menu using an AI model that generates the ingredient use priority and its analysis data, and means for automatically activating the camera to capture video data when the refrigerator is opened and closed. This allows for effective management of ingredients in the refrigerator, reducing food waste and household expenses.

[1078] A "video input device" is a device that is installed inside a refrigerator and is used to acquire still image and video data.

[1079] "Video data" refers to image or video data that shows the location and condition of ingredients in the refrigerator, captured by a video input device.

[1080] "Type of food" refers to the specific classification of food stored in the refrigerator, such as fruit, dairy products, eggs, etc.

[1081] "Status of ingredients" refers to the current condition of ingredients in the refrigerator, including, for example, fresh, deteriorating, or close to expiry date.

[1082] "Deterioration rate" is an indicator of how quickly food deteriorates, tracking changes such as from green to yellow to red.

[1083] "Usage priority" indicates the priority for consumption, taking into consideration the rate at which food ingredients deteriorate and their expiration dates.

[1084] The "display device" is a display device that is installed inside the refrigerator and that allows the user to check the status of ingredients and the priority of their use.

[1085] A "personal digital assistant" is a device that a user can carry around and is used to display the status of ingredients and the priority for consumption.

[1086] A "generative AI model" is an artificial intelligence technology that uses pre-trained algorithms to automatically generate new information, such as recipes, based on data input.

[1087] The "purchase list" displays a list of ingredients that the user needs to purchase.

[1088] "Consumption timing" predicts the optimal time to consume food based on the condition of the food and the rate of deterioration.

[1089] The system embodying this invention is designed to effectively manage food ingredients in a refrigerator, reduce food loss, and prevent food waste. This system links a video input device installed in the refrigerator, a server, and a user's mobile information terminal.

[1090] Hardware and Software Configuration

[1091] 1. Video input device

[1092] A camera (video input device) installed inside the refrigerator acquires video data. This camera operates automatically every time the refrigerator door is opened or closed, capturing internal video. For example, a standard refrigerator camera or a smart camera can be used.

[1093] 2. Server

[1094] The server periodically receives and analyzes the video data sent from the video input device. The server uses the following technologies:

[1095] Image recognition technology: Using OpenCV and TensorFlow, the type, condition, and rate of deterioration of ingredients are identified from video data.

[1096] Generative AI model: For example, a generative AI model such as GPT-4 is used to automatically generate menus based on ingredient data.

[1097] 3. Mobile Information Terminals

[1098] The user's mobile information terminal has an application installed that displays the information sent from the server, and the terminal also displays the data on the refrigerator's display.

[1099] Data processing and calculation

[1100] 1. Acquiring video data

[1101] The server acquires video data from the camera inside the refrigerator, for example, at 9:00 AM and 6:00 PM every day. This data includes the location and condition of the food items stored inside the refrigerator.

[1102] 2. Recognizing ingredients and understanding their condition

[1103] The acquired video data is analyzed using image recognition technology (OpenCV and TensorFlow) to identify the type of food and its condition. For example, apples, milk, eggs, etc. in the refrigerator are automatically detected, and their deterioration status and expiration date are estimated.

[1104] 3. Calculation of usage priority

[1105] Based on the analyzed food data, the server calculates the usage priority of ingredients that are deteriorating or approaching their expiration date. For example, the priority is calculated on a scale of 0 to 100, with the higher the number, the higher the usage priority.

[1106] 4. Visualization of usage priority

[1107] The server calculates the usage priority and sends it to the terminal in JSON format, where it is displayed on the refrigerator display or mobile information terminal. For example, it is displayed in red (high priority), yellow (medium priority), or green (low priority).

[1108] 5. Automatic menu generation

[1109] The server uses a generative AI model based on the ingredient usage priority data to automatically generate a menu. An example of a prompt sentence is "Suggest a simple breakfast recipe using milk and eggs," and the generated recipe is provided to the user.

[1110] 6. Listing ingredients you are lacking and making shopping suggestions

[1111] The server compares the recipe with the information on ingredients in the refrigerator, lists any missing ingredients, and displays the list on the mobile information terminal as a shopping list. For example, if the user does not have butter for "French toast," the server notifies the user.

[1112] 7. Prediction of deterioration rate and suggestion of consumption timing

[1113] The server monitors the rate at which food items deteriorate and notifies the user when it is appropriate to consume them. For example, if it detects that an apple is deteriorating faster than normal, a notification will be sent to the mobile information terminal saying, "Please consume the apple as soon as possible."

[1114] Specific examples

[1115] For example, when a user opens a refrigerator, the display shows the following information:

[1116] High priority (red): Milk (expiration date 2 days later)

[1117] Medium priority (yellow): Apple (changing color)

[1118] Low priority (green): Eggs (expiration date one week later)

[1119] The mobile device will be notified of the following:

[1120] French toast recipe (milk and eggs)

[1121] Missing Ingredient List: Butter

[1122] Consumption timing notification: Please consume the apples as soon as possible

[1123] This system allows users to manage food in their refrigerators without waste and consume it in a planned manner, thereby reducing food waste and household expenses.

[1124] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1125] Step 1:

[1126] The server periodically acquires video data from a video input device installed inside the refrigerator.

[1127] Specific behavior:

[1128] The camera automatically activates and captures video data every time the refrigerator door is opened or closed, and the server collects this video data at 9:00 AM and 6:00 PM every day.

[1129] Input: Video data (images and videos showing the location and condition of ingredients in the refrigerator)

[1130] Output: Saved video data file

[1131] Step 2:

[1132] The server analyzes the acquired video data and identifies the type and condition of the ingredients.

[1133] Specific behavior:

[1134] The server uses image recognition technologies such as OpenCV and TensorFlow to classify and recognize ingredients from video data. For example, it analyzes video data to automatically identify ingredients such as apples, milk, and eggs and evaluate their deterioration status.

[1135] Input: Saved video data file

[1136] Output: Ingredient list (including type, condition, and deterioration rate)

[1137] Step 3:

[1138] The server calculates the use priority of each ingredient based on the identified ingredient data.

[1139] Specific behavior:

[1140] The server calculates the priority of ingredients based on their deterioration rate and expiration date, using a score ranging from 0 to 100. Ingredients that are deteriorating or nearing their expiration date are given a higher score.

[1141] Input: Ingredient list (including type, condition, and deterioration rate)

[1142] Output: Usage priority list

[1143] Step 4:

[1144] The server visualizes the usage priority and displays it on the terminal.

[1145] Specific behavior:

[1146] The server sends the usage priority list in JSON format to the terminal. The terminal receives this data and displays it on the refrigerator display and on the mobile information terminal. For example, each ingredient is displayed in red (high priority), yellow (medium priority), and green (low priority).

[1147] Input: Usage priority list

[1148] Output: Ingredient information displayed on a display and a mobile information terminal

[1149] Step 5:

[1150] The server automatically generates a menu based on the usage priority data.

[1151] Specific behavior:

[1152] The server uses a generative AI model such as GPT-4 to create a menu based on the priority of ingredients. For example, it inputs a prompt such as "Suggest a simple breakfast recipe using milk and eggs" into the AI ​​model and presents the generated recipe to the user.

[1153] Input: Usage priority list

[1154] Output: Recipe

[1155] Step 6:

[1156] The server compares the generated menu with the ingredient information, lists any missing ingredients, and displays them as a shopping list.

[1157] Specific behavior:

[1158] The server compares the list of ingredients needed for the created menu with the list of ingredients currently in the refrigerator to identify any missing ingredients. Based on this, it creates a shopping list and sends it to the user's mobile information terminal.

[1159] Input: Menu, ingredient list

[1160] Output: Shopping list

[1161] Step 7:

[1162] The server monitors the rate at which ingredients deteriorate and notifies the user of the appropriate time to consume them.

[1163] Specific behavior:

[1164] The server analyzes the rate at which food deteriorates and determines the appropriate time to consume it. For example, if it determines that an apple is deteriorating faster than normal, it sends a notification to the user's mobile information terminal saying, "Please consume the apple as soon as possible."

[1165] Input: Ingredient list, deterioration rate data

[1166] Output: Information message

[1167] (Application example 1)

[1168] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1169] In modern households, efficiently managing food in the refrigerator and reducing waste are major challenges. However, manual food management is time-consuming, and it is difficult to understand expiration dates and determine the appropriate time to consume food. In addition, food delivery services also require inventory management and optimization of consumption times, but achieving this requires advanced systems.

[1170] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1171] In this invention, the server includes means for periodically acquiring video data from a video input device installed in the refrigerator, means for analyzing the video data to identify the type, condition, and deterioration rate of ingredients, means for calculating the use priority of each ingredient based on the identified information, means for visualizing the use priority and displaying it on an information display device, means for automatically generating cooking recipes based on the use priority, means for identifying shortages of ingredients and displaying it as a shopping list, means for predicting appropriate consumption times based on the deterioration rate of ingredients and notifying the user, means for acquiring consumption data in the refrigerator in real time and automating menu suggestions according to the deterioration status of ingredients, and means for automatically generating cooking recipes based on the consumption priority of ingredients using a generative AI model.This improves the efficiency of food management in the refrigerator, enabling food waste reduction and consumption optimization.

[1172] The "video input device" is a device that is installed inside the refrigerator and is used to acquire video data from inside the refrigerator.

[1173] "Video data" is image information of the inside of the refrigerator acquired by a video input device.

[1174] "Type of foodstuff" is information that indicates the specific classification of food stored in the refrigerator.

[1175] "Condition" is information that indicates the freshness and degree of deterioration of the food material.

[1176] "Deterioration rate" is information indicating the rate at which food ingredients deteriorate.

[1177] "Use priority" is information indicating the consumption priority of each ingredient.

[1178] An "information display device" is a device for displaying acquired information to a user.

[1179] A "cooking recipe" is information that shows the steps for cooking a dish using ingredients.

[1180] A "shopping list" is a list of ingredients that are needed but in short supply.

[1181] "Consumption timing" is information indicating the appropriate time to consume the food material.

[1182] "Consumption data" is information relating to the usage of ingredients.

[1183] "Menu suggestion" means suggesting cooking recipes based on the current ingredients.

[1184] A "generative AI model" is a model that uses artificial intelligence to analyze data and generate results.

[1185] The system embodying this invention is designed to effectively manage food ingredients in a refrigerator and reduce food waste. This system links a video input device installed in the refrigerator, a server, and the user's mobile device.

[1186] The server periodically acquires video data from a video input device (camera) installed inside the refrigerator. The server then analyzes the video data and uses image recognition algorithms to identify the type, condition, and deterioration rate of ingredients. This allows the server to detect the specific classification of ingredients in the refrigerator and their freshness or degree of deterioration.

[1187] The server then calculates the usage priority of each ingredient based on the identified information. Ingredients with a high usage priority are those that deteriorate quickly or are close to their expiration date. This information is displayed on an information display device such as a refrigerator display or a mobile device.

[1188] The server automatically generates cooking recipes based on the priority of use. The generated recipes are sent to the terminal and displayed to the user. The server also identifies missing ingredients and displays them as a shopping list. This information is also sent to the user's terminal in real time.

[1189] Furthermore, the server predicts the best time to consume ingredients based on the rate at which ingredients deteriorate and notifies the user, allowing them to consume ingredients at the appropriate time.The server also uses a generative AI model to automatically generate cooking recipes based on ingredient consumption priorities, automating menu suggestions.

[1190] For example, a camera inside a refrigerator captures video data and sends it to a server. The server analyzes the data and determines that the ingredients are A (deteriorating quickly), B (nearly expiring), and C (fresh). Based on this, an AI model generates recipes using ingredients A and B, which have a high priority for use, and notifies the user. A recipe such as "French toast" is then suggested, and butter, which is in short supply, is also displayed on the shopping list. Notifications such as "Please consume the apples soon" are also given to remind users when to consume them.

[1191] An example of a prompt to input to a generative AI model would be:

[1192] High priority ingredients: milk, apples

[1193] Medium priority ingredients: Eggs

[1194] Please suggest a recipe using these ingredients.

[1195] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1196] Step 1:

[1197] The server periodically acquires video data from a video input device (camera) installed inside the refrigerator.

[1198] Input: Video data acquired from a camera inside the refrigerator

[1199] Data processing: Send the video data to the server

[1200] Output: Video data stored on the server

[1201] Step 2:

[1202] The server analyzes the captured video data and uses image recognition algorithms to identify the type, condition, and rate of deterioration of the food.

[1203] Input: Video data stored on the server

[1204] Data Computing: Using image recognition algorithms to identify the type, condition, and rate of deterioration of ingredients

[1205] Output: Information on the type, condition, and deterioration rate of identified ingredients

[1206] Step 3:

[1207] The server calculates the use priority of each ingredient based on the identified information.

[1208] Input: Information on the type, condition, and deterioration rate of ingredients

[1209] Data calculation: Calculate the usage priority taking into account the rate of deterioration and expiration date

[1210] Output: Usage priority of each ingredient

[1211] Step 4:

[1212] The server visualizes the usage priority and displays it on an information display device.

[1213] Input: Usage priority of each ingredient

[1214] Data processing: Visualization of usage priority (e.g., color-coding)

[1215] Output: Usage priority displayed on the information display

[1216] Step 5:

[1217] The server automatically generates cooking recipes based on the usage priority and notifies the terminal.

[1218] Input: Usage priority of each ingredient

[1219] Data Computing: Automatically generating cooking recipes using generative AI models

[1220] Output: Cooking recipe notified to the device

[1221] Step 6:

[1222] The server identifies missing ingredients and displays them as a shopping list.

[1223] Input: Cooking recipe

[1224] Data calculation: Compare the current ingredient information with the generated menu and list any missing ingredients

[1225] Output: Shopping list displayed on the terminal

[1226] Step 7:

[1227] The server predicts the appropriate time to consume the food based on the rate at which the food deteriorates and notifies the user.

[1228] Input: Food deterioration rate and consumption priority

[1229] Data calculation: Predicting the optimal consumption timing based on the rate of deterioration and usage priority

[1230] Output: Consumption timing suggestions sent to the device

[1231] This will improve the efficiency of food management in the refrigerator, reducing food waste and optimizing consumption.

[1232] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1233] The system embodying this invention recognizes the user's emotional state by combining an emotion engine and customizes services based on this to effectively manage ingredients in a refrigerator. The system includes a video input device installed in the refrigerator, a server, a user's mobile terminal, and an emotion engine.

[1234] 1. Acquiring video data

[1235] The server periodically acquires video data from a video input device (camera) inside the refrigerator, including the type and condition of the food stored in the refrigerator.

[1236] 2. Recognizing ingredients and understanding their condition

[1237] The server analyzes the captured video data using image recognition technology to identify the type of food and its condition (such as the rate of deterioration and expiration date). For example, apples, milk, eggs, etc. in the refrigerator are automatically detected and their deterioration status is calculated.

[1238] 3. Calculation of usage priority

[1239] The server calculates the usage priority of each ingredient based on the type and condition data of the ingredient. Ingredients that are deteriorating or approaching their expiration date are given a higher priority.

[1240] 4. Visualization of usage priority

[1241] The device receives the usage priority information sent from the server and displays it on the refrigerator display or mobile device. This display uses color to indicate the usage priority of ingredients, allowing users to understand which ingredients are important at a glance. For example, high priority ingredients are displayed in red, medium priority in yellow, and low priority in green.

[1242] 5. Automatic menu generation

[1243] The server automatically generates menus using an AI model based on the priority data for ingredients. For example, it suggests a recipe for "French toast" that uses milk and eggs, which are the high priorities. This information is sent to the device and notified to the user.

[1244] 6. Listing ingredients you are lacking and making shopping suggestions

[1245] The server compares the current ingredient information with the generated menu, lists any missing ingredients, and suggests them as a shopping list. For example, if there is a shortage of butter for "French toast," that information is displayed on the user's mobile device.

[1246] 7. Prediction of deterioration rate and suggestion of consumption timing

[1247] The server monitors the rate at which food items deteriorate and notifies the user when it is time to consume them. For example, if it detects that an apple is deteriorating faster than normal, it will notify the user to consume it as soon as possible.

[1248] 8. User Emotion Recognition

[1249] The server uses an emotion engine to recognize the user's emotions. This emotion data is obtained, for example, from a smartphone camera or other sensors. It analyzes the user's facial expressions and voice and stores the results as emotion data.

[1250] 9. Emotionally-based menu adjustments

[1251] The server adjusts the menu based on the user's emotional data. For example, if the user is under stress, it will suggest simple and relaxing meals.

[1252] 10. Emotion-Based Information Display

[1253] The device customizes the information shown on the refrigerator display or mobile device depending on the user's emotional state. For example, if the emotion engine recognizes that the user is tired, it will prioritize showing easy-to-prepare recipes.

[1254] Specific examples

[1255] For example, when a user opens a refrigerator, the display shows the following information:

[1256] High priority (red): Milk (expiration date 2 days later)

[1257] Medium priority (yellow): Apple (changing color)

[1258] Low priority (green): Eggs (expiration date one week later)

[1259] When a user smiles through the camera in front of the refrigerator, the emotion engine recognizes this information. The server determines that the user is relaxed and suggests an easy and enjoyable recipe for "French toast."

[1260] Additionally, your mobile device will display the following information:

[1261] French toast recipe (milk and eggs)

[1262] Missing Ingredient List: Butter

[1263] Consumption timing notification: Please consume the apples as soon as possible

[1264] In this way, users can manage the food in their refrigerators without waste and consume it in a planned manner. In addition, cooking suggestions are made based on the user's emotions, allowing for more personalized food management. This system will reduce food waste and household expenses.

[1265] The processing flow will be explained below.

[1266] Step 1:

[1267] The server periodically acquires video data from a video input device installed inside the refrigerator. Specifically, the server sends a command to the camera to capture the latest video data every hour and saves it in temporary storage.

[1268] Step 2:

[1269] The server analyzes the stored video data and identifies the type and condition of the ingredients. Specifically, the server processes the video data using a machine learning model to identify ingredients such as apples, milk, and eggs. It also analyzes the color and shape of each ingredient to estimate their deterioration rate and expiration date.

[1270] Step 3:

[1271] The server calculates the usage priority of each ingredient based on the identified ingredient data. Specifically, the server assigns a priority score to each ingredient according to its expiration date and deterioration state, and classifies it into three categories: high, medium, and low.

[1272] Step 4:

[1273] The server stores the calculated usage priority in a database and prepares the data for visualization. Specifically, the server records the priority score and analysis results for each ingredient in the database.

[1274] Step 5:

[1275] The device retrieves ingredient usage priority data from the server and displays it on the refrigerator display or mobile device. Specifically, the device sends an API request, receives the latest data from the server, and displays it in color. High-priority ingredients are displayed in red, medium-priority in yellow, and low-priority in green.

[1276] Step 6:

[1277] The server uses an AI model to automatically generate the optimal menu based on the usage priority data. Specifically, the server passes ingredient information to the AI ​​model as input and generates a recipe for "French toast" using milk and eggs, for example.

[1278] Step 7:

[1279] The server sends the generated menu to the device and notifies the user. Specifically, the server sends the menu information to the device via API and notifies the user through the mobile app's notification function.

[1280] Step 8:

[1281] The server compares the current ingredient information with the generated menu and lists any missing ingredients. Specifically, the server compares the list of ingredients needed for "French toast" with the current state of the refrigerator and identifies any missing items (e.g., butter).

[1282] Step 9:

[1283] The server creates a shopping list of the identified ingredients that are in short supply and sends it to the device. Specifically, the server creates a list of ingredients that are in short supply and sends it to the device via API.

[1284] Step 10:

[1285] The device displays the shopping list to the user. Specifically, the device displays "You are running low on the following ingredients: Butter" on the mobile app.

[1286] Step 11:

[1287] The server monitors the rate at which food deteriorates and notifies the user of the appropriate time to consume it. Specifically, the server continuously collects data such as the temperature, weight, and color of the food, and predicts the rate at which it will deteriorate. It notifies the user before deterioration progresses. For example, if it determines that an apple is changing color rapidly, the mobile app will notify the user, saying, "Please consume the apple as soon as possible."

[1288] Step 12:

[1289] The device receives the notification from the server and displays it to the user. Specifically, the device displays the notification from the server through the notification function of the mobile app, informing the user when it is time to consume the ingredients.

[1290] Step 13:

[1291] The server uses an emotion engine to recognize the user's emotions. Specifically, the server analyzes data acquired from the smartphone's camera and microphone, and generates emotion data from the user's facial expressions and voice.

[1292] Step 14:

[1293] The server adjusts the generated menu based on the emotional data. Specifically, if the server determines that the user is under stress, it will prioritize suggesting ingredients that have a relaxing effect and easy-to-cook recipes.

[1294] Step 15:

[1295] The device customizes the information displayed on the refrigerator display or mobile device based on the emotional data. Specifically, if the device recognizes that the user is tired, it will prioritize easy-to-prepare recipes. If the user's emotional state is positive, it will suggest new recipes to try.

[1296] Example flow

[1297] For example, when a user opens a refrigerator, the display shows the following information:

[1298] High priority (red): Milk (expiration date 2 days later)

[1299] Medium priority (yellow): Apple (changing color)

[1300] Low priority (green): Eggs (expiration date one week later)

[1301] When a user smiles through the camera in front of the refrigerator, the emotion engine recognizes this information. The server determines that the user is relaxed and suggests an easy and enjoyable recipe for "French toast."

[1302] Additionally, your mobile device will display the following information:

[1303] French toast recipe (milk and eggs)

[1304] Missing Ingredient List: Butter

[1305] Consumption timing notification: Please consume the apples as soon as possible

[1306] In this way, users can manage the food in their refrigerators without waste and consume it in a planned manner. In addition, cooking suggestions are made based on the user's emotions, allowing for more personalized food management. This system will reduce food waste and household expenses.

[1307] Example 2

[1308] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1309] While conventional refrigerator food management systems can grasp the type and condition of ingredients, they have difficulty managing ingredients and suggesting menus that match the user's emotions and individual lifestyles. Furthermore, while they can predict the rate of deterioration and calculate usage priorities, they do not support adjusting menus or suggesting appropriate consumption times based on the user's emotions. As a result, user satisfaction is low, food waste is often generated, and unnecessary stress is often generated. To solve these issues, a system that can recognize the user's emotional state and customize services based on that state was needed.

[1310] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for periodically acquiring video data from a video input device installed in the refrigerator, means for analyzing the video data to identify the type, condition, and deterioration rate of ingredients, means for calculating the use priority of each ingredient based on the identified information, means for visualizing the use priority and displaying it on the refrigerator's display device or a mobile information terminal, means for generating a menu of dishes based on the use priority, means for identifying shortages of ingredients and displaying it as a shopping list, means for predicting appropriate consumption times based on the deterioration rate of ingredients and notifying the user, means for recognizing the user's emotions using an emotion engine, means for adjusting the menu based on the user's emotions, and means for customizing display information based on the user's emotions. This enables personalized ingredient management and menu suggestions based on the user's emotional state, thereby reducing ingredient waste and improving user satisfaction.

[1311] The "video input device installed inside the refrigerator" refers to a camera or other image capturing device that is installed inside the refrigerator and that captures video data of the food ingredients inside the refrigerator.

[1312] "Video data" is digital data that includes image information of food items in the refrigerator.

[1313] "Type of food" refers to the specific item names of food stored in the refrigerator, such as apples, milk, eggs, etc.

[1314] "Condition of ingredients" refers to the current quality and degree of deterioration of ingredients in the refrigerator.

[1315] "Deterioration rate" is an index that indicates how quickly each food ingredient stored in a refrigerator deteriorates.

[1316] "Usage priority" is calculated based on the priority of using food items in the refrigerator, taking into account factors such as the rate of deterioration and expiration date.

[1317] A "display device" is a display or monitor attached to a refrigerator and is used to visually display information about ingredients and menu items.

[1318] A "mobile information terminal" is an electronic device such as a smartphone or tablet that can be carried by a user and that displays ingredient information and menus.

[1319] "Cooking Menu" is a meal menu suggested based on the ingredients in your refrigerator.

[1320] A "shopping list" is a list of ingredients that need to be purchased based on the ingredients in the refrigerator and the menu.

[1321] "Consumption timing" refers to the predicted optimal time to consume a food item based on its deterioration rate.

[1322] An "emotion engine" is an analysis device or software that analyzes a user's emotional state, stores it as data, and uses it for subsequent processing.

[1323] "User emotion" means the user's psychological state analyzed through the emotion engine.

[1324] "Customizing display information" means adjusting the information displayed on the refrigerator display device or mobile information terminal based on the user's emotional state and priorities.

[1325] The system embodying this invention recognizes the user's emotional state by combining an emotion engine and customizes services based on this to effectively manage ingredients in a refrigerator. The system includes a video input device installed in the refrigerator, a server, a user's mobile terminal, and an emotion engine.

[1326] Hardware and Software Configuration

[1327] 1. Video input device (camera) inside the refrigerator

[1328] A camera is installed inside the refrigerator to periodically capture video data of the ingredients. The camera can take high-resolution images and send the data to a server every hour, for example.

[1329] 2. Server

[1330] The server acquires, analyzes, stores, and processes video data. Specific software used for image recognition is OpenCV and TensorFlow. The emotion engine is used to analyze user emotion data.

[1331] 3. Terminal (refrigerator display or mobile terminal)

[1332] The terminal displays the information sent from the server, visually conveying information such as the priority of ingredients, menus, shopping lists, and consumption timing to the user.

[1333] Data processing and calculation

[1334] 1. Acquiring video data

[1335] The server periodically acquires video data from the video input device (camera) inside the refrigerator, including the type and condition of the food stored in the refrigerator.

[1336] 2. Recognizing ingredients and understanding their condition

[1337] The server uses image recognition software (e.g., OpenCV or TensorFlow) to analyze the captured video data, thereby identifying the type and condition of the food (rate of deterioration and expiration date).

[1338] 3. Calculation of usage priority

[1339] The server calculates the usage priority of each ingredient based on the type and condition data of the ingredient, assigning a higher priority to ingredients that are deteriorating or approaching their expiration date.

[1340] 4. Visualization of usage priority

[1341] The device receives the usage priority information sent from the server and displays it on the refrigerator display or mobile device. The priority is indicated by a color, allowing the user to understand the importance at a glance.

[1342] 5. Automatic menu generation

[1343] The server automatically generates menus using a generative AI model (e.g., GPT-4) based on the ingredient usage priority data, and suggests recipes that include high-priority ingredients.

[1344] 6. Listing ingredients you are lacking and making shopping suggestions

[1345] The server compares the current ingredient information with the generated menu, lists any missing ingredients, and suggests a shopping list to the user.

[1346] 7. Prediction of deterioration rate and suggestion of consumption timing

[1347] The server monitors the rate at which ingredients deteriorate and notifies the user when it is time to consume them.

[1348] 8. User Emotion Recognition

[1349] The server uses an emotion engine to recognize the user's emotions, and this data is obtained from the smartphone's camera and other sensors.

[1350] 9. Emotionally-based menu adjustments

[1351] The server uses an AI model to adjust the menu based on the user's emotional data.

[1352] 10. Emotion-Based Information Display

[1353] The device customizes the information displayed on the refrigerator display and mobile device based on the user's emotional state.

[1354] Specific examples

[1355] For example, when a user opens a refrigerator, the display shows the following information:

[1356] High priority (red): Milk (expiration date 2 days later)

[1357] Medium priority (yellow): Apple (changing color)

[1358] Low priority (green): Eggs (expiration date one week later)

[1359] When the user smiles through the camera in front of the refrigerator, the emotion engine recognizes this information. The server determines that the user is relaxed and suggests a simple and enjoyable recipe for "French toast." The mobile device also displays the following information:

[1360] French toast recipe (milk and eggs)

[1361] Missing Ingredient List: Butter

[1362] Consumption timing notification: Please consume the apples as soon as possible

[1363] Example prompts to input to the generative AI model

[1364] "Suggest a simple recipe using the milk and eggs in the fridge. The user is currently relaxed."

[1365] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1366] Step 1:

[1367] The server periodically acquires video data from a video input device (camera) installed inside the refrigerator.

[1368] Input: Footage from inside a refrigerator.

[1369] Data processing / calculation: The camera takes high-resolution images every hour and sends them to the server as digital data.

[1370] Output: Video data of food in the refrigerator.

[1371] Step 2:

[1372] The server uses image recognition software (e.g., OpenCV or TensorFlow) to analyze the captured video data, thereby identifying the type and condition of the food ingredients (rate of deterioration and expiration date).

[1373] Input: Video data of food in a refrigerator.

[1374] Data processing / calculation: Detects food ingredients such as apples, milk, and eggs from the video and calculates the degree of deterioration and expiration date of each.

[1375] Output: Data on the type, condition and deterioration rate of identified ingredients.

[1376] Step 3:

[1377] The server calculates the use priority of each ingredient based on the type and state data of the ingredient.

[1378] Input: Data on the type, condition and deterioration rate of identified ingredients.

[1379] Data processing / calculation: A mathematical algorithm is used to assign a priority score based on the degree of deterioration and expiration date. Food items that are in an advanced state of deterioration or approaching their expiration date are given a higher priority.

[1380] Output: Usage priority data for each ingredient.

[1381] Step 4:

[1382] The terminal receives the information on the usage priority sent from the server and displays it on the refrigerator's display device or the mobile terminal.

[1383] Input: Usage priority data for each ingredient.

[1384] Data processing / calculation: Usage priority is displayed in color (e.g., high priority is red, medium priority is yellow, and low priority is green).

[1385] Output: Visually displayed ingredient usage priority information.

[1386] Step 5:

[1387] The server automatically generates menus using a generative AI model (e.g., GPT-4) based on ingredient usage priority data.

[1388] Input: Usage priority data for each ingredient.

[1389] Data processing / calculation: Send the prompt "Please suggest a simple recipe using milk and eggs. The user is currently in a relaxed state" to the generative AI model and receive the generated recipe.

[1390] Output: Automatically generated menu information.

[1391] Step 6:

[1392] The server compares the current ingredient information with the generated menu, lists any missing ingredients, and suggests a shopping list to the user.

[1393] Input: Current ingredient information, automatically generated menu information.

[1394] Data processing / calculation: Matching ingredient information with the menu, identifying missing ingredients, and generating a shopping list.

[1395] Output: Shopping list.

[1396] Step 7:

[1397] The server monitors the rate at which ingredients deteriorate and notifies the user when it is time to consume the ingredients.

[1398] Input: Food deterioration rate data.

[1399] Data processing / calculation: Analyzes deterioration rate data and calculates the appropriate consumption timing. For example, generates a notification saying "Please consume the apples as soon as possible."

[1400] Output: Consumption timing notification.

[1401] Step 8:

[1402] The server uses an emotion engine to recognize the user's emotions, and this data is obtained from the smartphone's camera and other sensors.

[1403] Input: Emotion data from smartphone cameras and sensors.

[1404] Data processing / calculation: Analyze the user's facial expressions and voice, and save the results as emotional data.

[1405] Output: User emotion data.

[1406] Step 9:

[1407] The server uses an AI model to adjust the menu based on the user's emotional data.

[1408] Input: User emotion data, automatically generated menu information.

[1409] Data processing / calculation: Emotional data is taken into account and menus are adjusted to be more relaxing (e.g., simple dishes when under stress).

[1410] Output: Menu information tailored based on emotions.

[1411] Step 10:

[1412] The terminal customizes the information displayed on the refrigerator display and the mobile terminal according to the user's emotional state.

[1413] Input: User emotion data.

[1414] Data processing / calculation: Based on emotional data, information to be displayed preferentially (e.g., easy recipes) is selected.

[1415] Output: Customized display information.

[1416] (Application example 2)

[1417] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1418] Conventional refrigerator food management systems focus on calculating the priority of ingredients based on their condition and deterioration rate and notifying the user of this priority. However, because information is provided unilaterally without considering the user's emotional state, there is a problem in that it is not possible to display optimal menu suggestions or product information for the user. The present invention aims to more effectively manage ingredients and suggest products in stores by recognizing the user's emotional state in real time and providing personalized services based on that.

[1419] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1420] In this invention, the server includes means for periodically acquiring video data from a video input device installed in the refrigerator, means for analyzing the video data to identify the type, condition, and deterioration rate of ingredients, means for calculating the use priority of each ingredient based on the identified information, means for visualizing the use priority and displaying it on the refrigerator display or a mobile terminal, means for automatically generating a recipe based on the use priority, means for identifying shortages of ingredients and displaying it as a shopping list, means for predicting appropriate consumption times based on the deterioration rate of ingredients and notifying the user, means including an emotion recognition engine for recognizing the user's emotional state, means for adjusting the recipe based on the user's emotional data, and means for displaying information about products in the store in real time. This enables optimal ingredient management and personalized product suggestions according to the user's emotional state.

[1421] The "video input device" is a device that is installed inside the refrigerator and periodically acquires video data of the food ingredients inside.

[1422] An "image recognition algorithm" is a calculation method for analyzing acquired video data and identifying the type, condition, and rate of deterioration of ingredients.

[1423] "Use priority" is an index that indicates the priority of consumption based on the type, condition, and deterioration rate of food ingredients.

[1424] A "display" is a device for visually displaying information, and may be installed on the surface of a refrigerator or on a mobile device.

[1425] A "mobile terminal" is a portable electronic device such as a smartphone or tablet that receives and displays information.

[1426] A "menu" is a plan of dishes using ingredients, and is automatically generated based on the priority of using ingredients.

[1427] A "shopping list" is a list of information that lists ingredients that are in short supply and is presented for purchase.

[1428] An "emotion recognition engine" is a software or hardware element that analyzes a user's facial expressions and voice and recognizes their emotional state.

[1429] "User emotion data" is data that indicates the temporary or persistent emotional state of the user, obtained through analysis by the emotion recognition engine.

[1430] "Product suggestion" is an act or system that presents product information in a store based on the user's emotional state.

[1431] "Real-time" is a concept that indicates that information is processed immediately and provided without delay.

[1432] In this invention, the following system is implemented to realize the management of ingredients in a refrigerator and the provision of personalized services based on the user's emotional state.

[1433] First, the server periodically acquires video data from a video input device installed inside the refrigerator. An image recognition algorithm is used to analyze this video data. The image recognition algorithm identifies the type, condition, and deterioration rate of ingredients, and calculates the usage priority of each ingredient based on this information. The usage priority is visualized on the refrigerator or mobile device display, allowing users to understand the status of ingredients at a glance.

[1434] Next, the server automatically generates a menu of dishes based on the calculated usage priority. This automatic generation uses a generative AI model. The generated menu is notified to the mobile device and displayed. The server also identifies any missing ingredients and displays them as a shopping list on the mobile device.

[1435] Furthermore, the server monitors the rate at which food items deteriorate and notifies the user of the appropriate time to consume them. For example, if an apple is deteriorating faster than usual, the server will notify the user, "Consume the apple sooner." This information encourages the user to consume food appropriately according to their lifestyle.

[1436] Next, an emotion recognition engine recognizes the user's emotional state. This engine captures and analyzes facial and voice data from the smartphone's camera and other sensors. Based on the user's emotional data, the system adjusts the menu accordingly. For example, if the user is tired, it will suggest simple, relaxing dishes.

[1437] Smart glasses and head-mounted displays (HMDs) are also used to display real-time product information in stores. When a user approaches or picks up a particular product, information about that product and recommended recipes are displayed.

[1438] As a concrete example, suppose a user wearing smart glasses is walking through a supermarket and stops in front of a bottle of milk. The camera in the smart glasses captures the user's smile, and the system determines that the user is relaxed and enjoying themselves. In this case, the smart glasses will display "Fresh milk. Recommended recipe: Panna cotta."

[1439] Prompt Sentence Examples

[1440] "A camera in smart glasses captures the user's facial expressions and suggests supermarket products to the user based on their emotional state. The system analyzes the user's emotional data in real time and displays appropriate product and recipe information based on the results."

[1441] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1442] Flow of the system program that realizes the application example

[1443] Step 1:

[1444] The server periodically acquires video data from a video input device installed inside the refrigerator.

[1445] Input: Video data from a video input device.

[1446] Data processing: Video data is converted to the appropriate resolution and saved as an image file on the server.

[1447] Output: Video data in image file format.

[1448] Step 2:

[1449] The server analyzes the captured video data and uses image recognition algorithms to identify the type, condition and rate of deterioration of the food.

[1450] Input: Stored video data.

[1451] Data processing: Apply image recognition algorithms to identify the type, condition, and rate of deterioration of ingredients.

[1452] Output: Identification data including type, condition and rate of deterioration of the food material.

[1453] Step 3:

[1454] The server calculates the use priority of each ingredient based on the identified information.

[1455] Input: Identification data for food type, condition, and rate of deterioration.

[1456] Data calculation: Calculates the usage priority score based on the deterioration rate and condition.

[1457] Output: Usage priority score for each ingredient.

[1458] Step 4:

[1459] The terminal receives the usage priority information sent from the server and displays it on the refrigerator display or mobile terminal.

[1460] Input: Usage priority score.

[1461] Data transformation: Converting usage priority scores into visual formats such as color coding.

[1462] Output: Visually enhanced usage priority information.

[1463] Step 5:

[1464] The server automatically generates a menu of dishes based on the priority of use.

[1465] Input: Usage priority score for each ingredient.

[1466] Data computation: Uses generative AI models to generate optimal menus.

[1467] Output: The generated menu data.

[1468] Step 6:

[1469] The terminal receives the generated menu data and notifies the mobile terminal.

[1470] Input: Generated menu data.

[1471] Data processing: Convert menu data into notification format.

[1472] Output: Notification message to mobile device.

[1473] Step 7:

[1474] The server compares the current ingredient information with the generated menu, identifies any missing ingredients, and displays them as a shopping list.

[1475] Input: Current ingredient information, generated menu data.

[1476] Data calculation: Compare the ingredients needed for the menu with the ingredients currently available.

[1477] Output: A list of missing ingredients.

[1478] Step 8:

[1479] The server monitors the rate at which ingredients deteriorate and notifies the user of the appropriate time to consume them.

[1480] Input: Food deterioration rate data.

[1481] Data calculation: Analyzes deterioration rate data and calculates the appropriate consumption timing.

[1482] Output: Consumption timing notification message.

[1483] Step 9:

[1484] The server uses an emotion recognition engine to recognize the emotional state of the user.

[1485] Input: Facial expression and voice data captured from a smartphone's camera and sensors.

[1486] Data calculation: Apply emotion recognition algorithms to analyze the user's emotional state.

[1487] Output: User emotion data.

[1488] Step 10:

[1489] The server adjusts the menu based on the user's emotional data.

[1490] Input: User emotion data.

[1491] Data calculation: Optimal menu adjustment based on emotional data.

[1492] Output: Adjusted menu data.

[1493] Step 11:

[1494] The server displays information about products in the store in real time.

[1495] Input: Camera image data from smart glasses or head-mounted displays, and user location information.

[1496] Data calculation: Analyzes camera footage and location information to display appropriate product information.

[1497] Output: Product information shown on the user's display.

[1498] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1499] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1500] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1501] [Fourth embodiment]

[1502] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1503] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1504] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1505] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1506] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1507] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1508] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1509] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1510] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1511] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1512] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1513] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1514] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1515] The system embodying this invention is designed to effectively manage food ingredients in a refrigerator, reduce food loss, and prevent food waste. This system links a video input device installed in the refrigerator, a server, and a user's mobile terminal.

[1516] 1. Acquiring video data

[1517] The server periodically acquires video data from a video input device (camera) inside the refrigerator, including the location and status of the food items stored inside the refrigerator.

[1518] 2. Recognizing ingredients and understanding their condition

[1519] The server analyzes the captured video data using image recognition technology to identify the type of food and its condition (such as the rate of deterioration and expiration date). For example, apples, milk, eggs, etc. in the refrigerator are automatically detected and their deterioration status is calculated.

[1520] 3. Calculation of usage priority

[1521] The server calculates the usage priority of each ingredient based on the type and condition data of the ingredient. Ingredients that are deteriorating or approaching their expiration date are given a higher priority.

[1522] 4. Visualization of usage priority

[1523] The device receives the usage priority information sent from the server and displays it on the refrigerator display or mobile device. This display uses color to indicate the usage priority of ingredients, allowing users to understand which ingredients are important at a glance. For example, high priority ingredients are displayed in red, medium priority in yellow, and low priority in green.

[1524] 5. Automatic menu generation

[1525] The server automatically generates menus using an AI model based on the priority data for ingredients. For example, it suggests a recipe for "French toast" that uses milk and eggs, which are the high priorities. This information is sent to the device and notified to the user.

[1526] 6. Listing ingredients you are lacking and making shopping suggestions

[1527] The server compares the current ingredient information with the generated menu, lists any missing ingredients, and suggests them as a shopping list. For example, if there is a shortage of butter for "French toast," that information is displayed on the user's mobile device.

[1528] 7. Prediction of deterioration rate and suggestion of consumption timing

[1529] The server monitors the rate at which food items deteriorate and notifies the user when it is time to consume them. For example, if it detects that an apple is deteriorating faster than normal, it will notify the user to consume it as soon as possible.

[1530] Specific examples

[1531] For example, when a user opens a refrigerator, the display shows the following information:

[1532] High priority (red): Milk (expiration date 2 days later)

[1533] Medium priority (yellow): Apple (changing color)

[1534] Low priority (green): Eggs (expiration date one week later)

[1535] Additionally, your mobile device will receive the following information:

[1536] French toast recipe (milk and eggs)

[1537] Missing Ingredient List: Butter

[1538] Consumption timing notification: Please consume the apples as soon as possible

[1539] In this way, users can manage the food in their refrigerators without waste and consume it in a planned manner, reducing food waste and household expenses.

[1540] The processing flow will be explained below.

[1541] Step 1:

[1542] The server periodically acquires video data from a video input device installed inside the refrigerator. Specifically, the server sends a command to the camera to capture the latest video data every hour and saves it in temporary storage.

[1543] Step 2:

[1544] The server analyzes the stored video data and identifies the type and condition of the ingredients. Specifically, the server processes the video data using a machine learning model to identify ingredients such as apples, milk, and eggs. It also analyzes the color and shape of each ingredient to estimate their deterioration rate and expiration date.

[1545] Step 3:

[1546] The server calculates the usage priority of each ingredient based on the identified ingredient data. Specifically, the server assigns a priority score to each ingredient according to its expiration date and deterioration state, and classifies it into three categories: high, medium, and low.

[1547] Step 4:

[1548] The server stores the calculated usage priority in a database and prepares the data for visualization. Specifically, the server records the priority score and analysis results for each ingredient in the database.

[1549] Step 5:

[1550] The device retrieves ingredient usage priority data from the server and displays it on the refrigerator display or mobile device. Specifically, the device sends an API request, receives the latest data from the server, and displays it in color. High-priority ingredients are displayed in red, medium-priority in yellow, and low-priority in green.

[1551] Step 6:

[1552] The server automatically generates the optimal menu using an AI model based on the usage priority data. Specifically, the server passes ingredient information to the AI ​​model as input and generates a recipe for "French toast" using milk and eggs, for example.

[1553] Step 7:

[1554] The server sends the generated menu to the device and notifies the user. Specifically, the server sends the menu information to the device via API and notifies the user through the mobile app's notification function.

[1555] Step 8:

[1556] The server compares the current ingredient information with the generated menu and lists any missing ingredients. Specifically, the server compares the list of ingredients needed for "French toast" with the current state of the refrigerator and identifies any missing items (e.g., butter).

[1557] Step 9:

[1558] The server creates a shopping list of the identified ingredients that are in short supply and sends it to the device. Specifically, the server creates a list of ingredients that are in short supply and sends it to the device via API.

[1559] Step 10:

[1560] The device displays the shopping list to the user. Specifically, the device displays "You are running low on the following ingredients: Butter" on the mobile app.

[1561] Step 11:

[1562] The server monitors the rate at which food deteriorates and notifies the user of the appropriate time to consume it. Specifically, the server continuously collects data such as the temperature, weight, and color of the food, and predicts the rate at which it will deteriorate. It notifies the user before deterioration progresses. For example, if it determines that an apple is changing color rapidly, the mobile app will notify the user, saying, "Please consume the apple as soon as possible."

[1563] Step 12:

[1564] The device receives the notification from the server and displays it to the user. Specifically, the device displays the notification from the server through the notification function of the mobile app, informing the user when it is time to consume the ingredients.

[1565] Example 1

[1566] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1567] In modern households, food waste is a common problem due to inadequate management of food in the refrigerator. It is particularly difficult to accurately grasp the rate at which food deteriorates and the priority of its use, and then create menu suggestions and shopping lists based on this information. It is also important to accurately predict when food will be consumed in order to avoid food waste. A system that can solve these problems and effectively manage food is needed.

[1568] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1569] In this invention, the server includes means for periodically acquiring video data from a video input device installed in the refrigerator, means for analyzing the video data to identify the type, condition, and deterioration rate of ingredients, means for calculating the use priority of each ingredient based on the identified information, means for visualizing the use priority and displaying it on the refrigerator's display device or a mobile information terminal, means for automatically generating a menu based on the use priority, means for identifying ingredients that are in short supply and displaying it as a shopping list, means for predicting the appropriate consumption timing based on the deterioration rate of ingredients and notifying the user, means for proposing a menu using an AI model that generates the ingredient use priority and its analysis data, and means for automatically activating the camera to capture video data when the refrigerator is opened and closed. This allows for effective management of ingredients in the refrigerator, reducing food waste and household expenses.

[1570] A "video input device" is a device that is installed inside a refrigerator and is used to acquire still image and video data.

[1571] "Video data" refers to image or video data that shows the location and condition of ingredients in the refrigerator, captured by a video input device.

[1572] "Type of food" refers to the specific classification of food stored in the refrigerator, such as fruit, dairy products, eggs, etc.

[1573] "Status of ingredients" refers to the current condition of ingredients in the refrigerator, including, for example, fresh, deteriorating, or close to expiry date.

[1574] "Deterioration rate" is an indicator of how quickly food deteriorates, tracking changes such as from green to yellow to red.

[1575] "Usage priority" indicates the priority for consumption, taking into consideration the rate at which food ingredients deteriorate and their expiration dates.

[1576] The "display device" is a display device that is installed inside the refrigerator and that allows the user to check the status of ingredients and the priority of their use.

[1577] A "personal digital assistant" is a device that a user can carry around and is used to display the status of ingredients and the priority for consumption.

[1578] A "generative AI model" is an artificial intelligence technology that uses pre-trained algorithms to automatically generate new information, such as recipes, based on data input.

[1579] The "purchase list" displays a list of ingredients that the user needs to purchase.

[1580] "Consumption timing" predicts the optimal time to consume food based on the condition of the food and the rate of deterioration.

[1581] The system embodying this invention is designed to effectively manage food ingredients in a refrigerator, reduce food loss, and prevent food waste. This system links a video input device installed in the refrigerator, a server, and a user's mobile information terminal.

[1582] Hardware and Software Configuration

[1583] 1. Video input device

[1584] A camera (video input device) installed inside the refrigerator acquires video data. This camera operates automatically every time the refrigerator door is opened or closed, capturing internal video. For example, a standard refrigerator camera or a smart camera can be used.

[1585] 2. Server

[1586] The server periodically receives and analyzes the video data sent from the video input device. The server uses the following technologies:

[1587] Image recognition technology: Using OpenCV and TensorFlow, the type, condition, and rate of deterioration of ingredients are identified from video data.

[1588] Generative AI model: For example, a generative AI model such as GPT-4 is used to automatically generate menus based on ingredient data.

[1589] 3. Mobile Information Terminals

[1590] The user's mobile information terminal has an application installed that displays the information sent from the server, and the terminal also displays the data on the refrigerator's display.

[1591] Data processing and calculation

[1592] 1. Acquiring video data

[1593] The server acquires video data from the camera inside the refrigerator, for example, at 9:00 AM and 6:00 PM every day. This data includes the location and condition of the food items stored inside the refrigerator.

[1594] 2. Recognizing ingredients and understanding their condition

[1595] The acquired video data is analyzed using image recognition technology (OpenCV and TensorFlow) to identify the type of food and its condition. For example, apples, milk, eggs, etc. in the refrigerator are automatically detected, and their deterioration status and expiration date are estimated.

[1596] 3. Calculation of usage priority

[1597] Based on the analyzed food data, the server calculates the usage priority of ingredients that are deteriorating or approaching their expiration date. For example, the priority is calculated on a scale of 0 to 100, with the higher the number, the higher the usage priority.

[1598] 4. Visualization of usage priority

[1599] The server calculates the usage priority and sends it to the terminal in JSON format, where it is displayed on the refrigerator display or mobile information terminal. For example, it is displayed in red (high priority), yellow (medium priority), or green (low priority).

[1600] 5. Automatic menu generation

[1601] The server uses a generative AI model based on the ingredient usage priority data to automatically generate a menu. An example of a prompt sentence is "Suggest a simple breakfast recipe using milk and eggs," and the generated recipe is provided to the user.

[1602] 6. Listing ingredients you are lacking and making shopping suggestions

[1603] The server compares the recipe with the information on ingredients in the refrigerator, lists any missing ingredients, and displays the list on the mobile information terminal as a shopping list. For example, if the user does not have butter for "French toast," the server notifies the user.

[1604] 7. Prediction of deterioration rate and suggestion of consumption timing

[1605] The server monitors the rate at which food items deteriorate and notifies the user when it is appropriate to consume them. For example, if it detects that an apple is deteriorating faster than normal, a notification will be sent to the mobile information terminal saying, "Please consume the apple as soon as possible."

[1606] Specific examples

[1607] For example, when a user opens a refrigerator, the display shows the following information:

[1608] High priority (red): Milk (expiration date 2 days later)

[1609] Medium priority (yellow): Apple (changing color)

[1610] Low priority (green): Eggs (expiration date one week later)

[1611] The mobile device will be notified of the following:

[1612] French toast recipe (milk and eggs)

[1613] Missing Ingredient List: Butter

[1614] Consumption timing notification: Please consume the apples as soon as possible

[1615] This system allows users to manage food in their refrigerators without waste and consume it in a planned manner, thereby reducing food waste and household expenses.

[1616] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1617] Step 1:

[1618] The server periodically acquires video data from a video input device installed inside the refrigerator.

[1619] Specific behavior:

[1620] The camera automatically activates and captures video data every time the refrigerator door is opened or closed, and the server collects this video data at 9:00 AM and 6:00 PM every day.

[1621] Input: Video data (images and videos showing the location and condition of ingredients in the refrigerator)

[1622] Output: Saved video data file

[1623] Step 2:

[1624] The server analyzes the acquired video data and identifies the type and condition of the ingredients.

[1625] Specific behavior:

[1626] The server uses image recognition technologies such as OpenCV and TensorFlow to classify and recognize ingredients from video data. For example, it analyzes video data to automatically identify ingredients such as apples, milk, and eggs and evaluate their deterioration status.

[1627] Input: Saved video data file

[1628] Output: Ingredient list (including type, condition, and deterioration rate)

[1629] Step 3:

[1630] The server calculates the use priority of each ingredient based on the identified ingredient data.

[1631] Specific behavior:

[1632] The server calculates the priority of ingredients based on their deterioration rate and expiration date, using a score ranging from 0 to 100. Ingredients that are deteriorating or nearing their expiration date are given a higher score.

[1633] Input: Ingredient list (including type, condition, and deterioration rate)

[1634] Output: Usage priority list

[1635] Step 4:

[1636] The server visualizes the usage priority and displays it on the terminal.

[1637] Specific behavior:

[1638] The server sends the usage priority list in JSON format to the terminal. The terminal receives this data and displays it on the refrigerator display and on the mobile information terminal. For example, each ingredient is displayed in red (high priority), yellow (medium priority), and green (low priority).

[1639] Input: Usage priority list

[1640] Output: Ingredient information displayed on a display and a mobile information terminal

[1641] Step 5:

[1642] The server automatically generates a menu based on the usage priority data.

[1643] Specific behavior:

[1644] The server uses a generative AI model such as GPT-4 to create a menu based on the priority of ingredients. For example, it inputs a prompt such as "Suggest a simple breakfast recipe using milk and eggs" into the AI ​​model and presents the generated recipe to the user.

[1645] Input: Usage priority list

[1646] Output: Recipe

[1647] Step 6:

[1648] The server compares the generated menu with the ingredient information, lists any missing ingredients, and displays them as a shopping list.

[1649] Specific behavior:

[1650] The server compares the list of ingredients needed for the created menu with the list of ingredients currently in the refrigerator to identify any missing ingredients. Based on this, it creates a shopping list and sends it to the user's mobile information terminal.

[1651] Input: Menu, ingredient list

[1652] Output: Shopping list

[1653] Step 7:

[1654] The server monitors the rate at which ingredients deteriorate and notifies the user of the appropriate time to consume them.

[1655] Specific behavior:

[1656] The server analyzes the rate at which food deteriorates and determines the appropriate time to consume it. For example, if it determines that an apple is deteriorating faster than normal, it sends a notification to the user's mobile information terminal saying, "Please consume the apple as soon as possible."

[1657] Input: Ingredient list, deterioration rate data

[1658] Output: Information message

[1659] (Application example 1)

[1660] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1661] In modern households, efficiently managing food in the refrigerator and reducing waste are major challenges. However, manual food management is time-consuming, and it is difficult to understand expiration dates and determine the appropriate time to consume food. In addition, food delivery services also require inventory management and optimization of consumption times, but achieving this requires advanced systems.

[1662] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1663] In this invention, the server includes means for periodically acquiring video data from a video input device installed in the refrigerator, means for analyzing the video data to identify the type, condition, and deterioration rate of ingredients, means for calculating the use priority of each ingredient based on the identified information, means for visualizing the use priority and displaying it on an information display device, means for automatically generating cooking recipes based on the use priority, means for identifying shortages of ingredients and displaying it as a shopping list, means for predicting appropriate consumption times based on the deterioration rate of ingredients and notifying the user, means for acquiring consumption data in the refrigerator in real time and automating menu suggestions according to the deterioration status of ingredients, and means for automatically generating cooking recipes based on the consumption priority of ingredients using a generative AI model.This improves the efficiency of food management in the refrigerator, enabling food waste reduction and consumption optimization.

[1664] The "video input device" is a device that is installed inside the refrigerator and is used to acquire video data from inside the refrigerator.

[1665] "Video data" is image information of the inside of the refrigerator acquired by a video input device.

[1666] "Type of foodstuff" is information that indicates the specific classification of food stored in the refrigerator.

[1667] "Condition" is information that indicates the freshness and degree of deterioration of the food material.

[1668] "Deterioration rate" is information indicating the rate at which food ingredients deteriorate.

[1669] "Use priority" is information indicating the consumption priority of each ingredient.

[1670] An "information display device" is a device for displaying acquired information to a user.

[1671] A "cooking recipe" is information that shows the steps for cooking a dish using ingredients.

[1672] A "shopping list" is a list of ingredients that are needed but in short supply.

[1673] "Consumption timing" is information indicating the appropriate time to consume the food material.

[1674] "Consumption data" is information relating to the usage of ingredients.

[1675] "Menu suggestion" means suggesting cooking recipes based on the current ingredients.

[1676] A "generative AI model" is a model that uses artificial intelligence to analyze data and generate results.

[1677] The system embodying this invention is designed to effectively manage food ingredients in a refrigerator and reduce food waste. This system links a video input device installed in the refrigerator, a server, and the user's mobile device.

[1678] The server periodically acquires video data from a video input device (camera) installed inside the refrigerator. The server then analyzes the video data and uses image recognition algorithms to identify the type, condition, and deterioration rate of ingredients. This allows the server to detect the specific classification of ingredients in the refrigerator and their freshness or degree of deterioration.

[1679] The server then calculates the usage priority of each ingredient based on the identified information. Ingredients with a high usage priority are those that deteriorate quickly or are close to their expiration date. This information is displayed on an information display device such as a refrigerator display or a mobile device.

[1680] The server automatically generates cooking recipes based on the priority of use. The generated recipes are sent to the terminal and displayed to the user. The server also identifies missing ingredients and displays them as a shopping list. This information is also sent to the user's terminal in real time.

[1681] Furthermore, the server predicts the best time to consume ingredients based on the rate at which ingredients deteriorate and notifies the user, allowing them to consume ingredients at the appropriate time.The server also uses a generative AI model to automatically generate cooking recipes based on ingredient consumption priorities, automating menu suggestions.

[1682] For example, a camera inside a refrigerator captures video data and sends it to a server. The server analyzes the data and determines that the ingredients are A (deteriorating quickly), B (nearly expiring), and C (fresh). Based on this, an AI model generates recipes using ingredients A and B, which have a high priority for use, and notifies the user. A recipe such as "French toast" is then suggested, and butter, which is in short supply, is also displayed on the shopping list. Notifications such as "Please consume the apples soon" are also given to remind users when to consume them.

[1683] An example of a prompt to input to a generative AI model would be:

[1684] High priority ingredients: milk, apples

[1685] Medium priority ingredients: Eggs

[1686] Please suggest a recipe using these ingredients.

[1687] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1688] Step 1:

[1689] The server periodically acquires video data from a video input device (camera) installed inside the refrigerator.

[1690] Input: Video data acquired from a camera inside the refrigerator

[1691] Data processing: Send the video data to the server

[1692] Output: Video data stored on the server

[1693] Step 2:

[1694] The server analyzes the captured video data and uses image recognition algorithms to identify the type, condition, and rate of deterioration of the food.

[1695] Input: Video data stored on the server

[1696] Data Computing: Using image recognition algorithms to identify the type, condition, and rate of deterioration of ingredients

[1697] Output: Information on the type, condition, and deterioration rate of identified ingredients

[1698] Step 3:

[1699] The server calculates the use priority of each ingredient based on the identified information.

[1700] Input: Information on the type, condition, and deterioration rate of ingredients

[1701] Data calculation: Calculate the usage priority taking into account the rate of deterioration and expiration date

[1702] Output: Usage priority of each ingredient

[1703] Step 4:

[1704] The server visualizes the usage priority and displays it on an information display device.

[1705] Input: Usage priority of each ingredient

[1706] Data processing: Visualization of usage priority (e.g., color-coding)

[1707] Output: Usage priority displayed on the information display

[1708] Step 5:

[1709] The server automatically generates cooking recipes based on the usage priority and notifies the terminal.

[1710] Input: Usage priority of each ingredient

[1711] Data Computing: Automatically generating cooking recipes using generative AI models

[1712] Output: Cooking recipe notified to the device

[1713] Step 6:

[1714] The server identifies missing ingredients and displays them as a shopping list.

[1715] Input: Cooking recipe

[1716] Data calculation: Compare the current ingredient information with the generated menu and list any missing ingredients

[1717] Output: Shopping list displayed on the terminal

[1718] Step 7:

[1719] The server predicts the appropriate time to consume the food based on the rate at which the food deteriorates and notifies the user.

[1720] Input: Food deterioration rate and consumption priority

[1721] Data calculation: Predicting the optimal consumption timing based on the rate of deterioration and usage priority

[1722] Output: Consumption timing suggestions sent to the device

[1723] This will improve the efficiency of food management in the refrigerator, reducing food waste and optimizing consumption.

[1724] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1725] The system embodying this invention recognizes the user's emotional state by combining an emotion engine and customizes services based on this to effectively manage ingredients in a refrigerator. The system includes a video input device installed in the refrigerator, a server, a user's mobile terminal, and an emotion engine.

[1726] 1. Acquiring video data

[1727] The server periodically acquires video data from a video input device (camera) inside the refrigerator, including the type and condition of the food stored in the refrigerator.

[1728] 2. Recognizing ingredients and understanding their condition

[1729] The server analyzes the captured video data using image recognition technology to identify the type of food and its condition (such as the rate of deterioration and expiration date). For example, apples, milk, eggs, etc. in the refrigerator are automatically detected and their deterioration status is calculated.

[1730] 3. Calculation of usage priority

[1731] The server calculates the usage priority of each ingredient based on the type and condition data of the ingredient. Ingredients that are deteriorating or approaching their expiration date are given a higher priority.

[1732] 4. Visualization of usage priority

[1733] The device receives the usage priority information sent from the server and displays it on the refrigerator display or mobile device. This display uses color to indicate the usage priority of ingredients, allowing users to understand which ingredients are important at a glance. For example, high priority ingredients are displayed in red, medium priority in yellow, and low priority in green.

[1734] 5. Automatic menu generation

[1735] The server automatically generates menus using an AI model based on the priority data for ingredients. For example, it suggests a recipe for "French toast" that uses milk and eggs, which are the high priorities. This information is sent to the device and notified to the user.

[1736] 6. Listing ingredients you are lacking and making shopping suggestions

[1737] The server compares the current ingredient information with the generated menu, lists any missing ingredients, and suggests them as a shopping list. For example, if there is a shortage of butter for "French toast," that information is displayed on the user's mobile device.

[1738] 7. Prediction of deterioration rate and suggestion of consumption timing

[1739] The server monitors the rate at which food items deteriorate and notifies the user when it is time to consume them. For example, if it detects that an apple is deteriorating faster than normal, it will notify the user to consume it as soon as possible.

[1740] 8. User Emotion Recognition

[1741] The server uses an emotion engine to recognize the user's emotions. This emotion data is obtained, for example, from a smartphone camera or other sensors. It analyzes the user's facial expressions and voice and stores the results as emotion data.

[1742] 9. Emotionally-based menu adjustments

[1743] The server adjusts the menu based on the user's emotional data. For example, if the user is under stress, it will suggest simple and relaxing meals.

[1744] 10. Emotion-Based Information Display

[1745] The device customizes the information shown on the refrigerator display or mobile device depending on the user's emotional state. For example, if the emotion engine recognizes that the user is tired, it will prioritize showing easy-to-prepare recipes.

[1746] Specific examples

[1747] For example, when a user opens a refrigerator, the display shows the following information:

[1748] High priority (red): Milk (expiration date 2 days later)

[1749] Medium priority (yellow): Apple (changing color)

[1750] Low priority (green): Eggs (expiration date one week later)

[1751] When a user smiles through the camera in front of the refrigerator, the emotion engine recognizes this information. The server determines that the user is relaxed and suggests an easy and enjoyable recipe for "French toast."

[1752] Additionally, your mobile device will display the following information:

[1753] French toast recipe (milk and eggs)

[1754] Missing Ingredient List: Butter

[1755] Consumption timing notification: Please consume the apples as soon as possible

[1756] In this way, users can manage the food in their refrigerators without waste and consume it in a planned manner. In addition, cooking suggestions are made based on the user's emotions, allowing for more personalized food management. This system will reduce food waste and household expenses.

[1757] The processing flow will be explained below.

[1758] Step 1:

[1759] The server periodically acquires video data from a video input device installed inside the refrigerator. Specifically, the server sends a command to the camera to capture the latest video data every hour and saves it in temporary storage.

[1760] Step 2:

[1761] The server analyzes the stored video data and identifies the type and condition of the ingredients. Specifically, the server processes the video data using a machine learning model to identify ingredients such as apples, milk, and eggs. It also analyzes the color and shape of each ingredient to estimate their deterioration rate and expiration date.

[1762] Step 3:

[1763] The server calculates the usage priority of each ingredient based on the identified ingredient data. Specifically, the server assigns a priority score to each ingredient according to its expiration date and deterioration state, and classifies it into three categories: high, medium, and low.

[1764] Step 4:

[1765] The server stores the calculated usage priority in a database and prepares the data for visualization. Specifically, the server records the priority score and analysis results for each ingredient in the database.

[1766] Step 5:

[1767] The device retrieves ingredient usage priority data from the server and displays it on the refrigerator display or mobile device. Specifically, the device sends an API request, receives the latest data from the server, and displays it in color. High-priority ingredients are displayed in red, medium-priority in yellow, and low-priority in green.

[1768] Step 6:

[1769] The server uses an AI model to automatically generate the optimal menu based on the usage priority data. Specifically, the server passes ingredient information to the AI ​​model as input and generates a recipe for "French toast" using milk and eggs, for example.

[1770] Step 7:

[1771] The server sends the generated menu to the device and notifies the user. Specifically, the server sends the menu information to the device via API and notifies the user through the mobile app's notification function.

[1772] Step 8:

[1773] The server compares the current ingredient information with the generated menu and lists any missing ingredients. Specifically, the server compares the list of ingredients needed for "French toast" with the current state of the refrigerator and identifies any missing items (e.g., butter).

[1774] Step 9:

[1775] The server creates a shopping list of the identified ingredients that are in short supply and sends it to the device. Specifically, the server creates a list of ingredients that are in short supply and sends it to the device via API.

[1776] Step 10:

[1777] The device displays the shopping list to the user. Specifically, the device displays "You are running low on the following ingredients: Butter" on the mobile app.

[1778] Step 11:

[1779] The server monitors the rate at which food deteriorates and notifies the user of the appropriate time to consume it. Specifically, the server continuously collects data such as the temperature, weight, and color of the food, and predicts the rate at which it will deteriorate. It notifies the user before deterioration progresses. For example, if it determines that an apple is changing color rapidly, the mobile app will notify the user, saying, "Please consume the apple as soon as possible."

[1780] Step 12:

[1781] The device receives the notification from the server and displays it to the user. Specifically, the device displays the notification from the server through the notification function of the mobile app, informing the user when it is time to consume the ingredients.

[1782] Step 13:

[1783] The server uses an emotion engine to recognize the user's emotions. Specifically, the server analyzes data acquired from the smartphone's camera and microphone, and generates emotion data from the user's facial expressions and voice.

[1784] Step 14:

[1785] The server adjusts the generated menu based on the emotional data. Specifically, if the server determines that the user is under stress, it will prioritize suggesting ingredients that have a relaxing effect and easy-to-cook recipes.

[1786] Step 15:

[1787] The device customizes the information displayed on the refrigerator display or mobile device based on the emotional data. Specifically, if the device recognizes that the user is tired, it will prioritize easy-to-prepare recipes. If the user's emotional state is positive, it will suggest new recipes to try.

[1788] Example flow

[1789] For example, when a user opens a refrigerator, the display shows the following information:

[1790] High priority (red): Milk (expiration date 2 days later)

[1791] Medium priority (yellow): Apple (changing color)

[1792] Low priority (green): Eggs (expiration date one week later)

[1793] When a user smiles through the camera in front of the refrigerator, the emotion engine recognizes this information. The server determines that the user is relaxed and suggests an easy and enjoyable recipe for "French toast."

[1794] Additionally, your mobile device will display the following information:

[1795] French toast recipe (milk and eggs)

[1796] Missing Ingredient List: Butter

[1797] Consumption timing notification: Please consume the apples as soon as possible

[1798] In this way, users can manage the food in their refrigerators without waste and consume it in a planned manner. In addition, cooking suggestions are made based on the user's emotions, allowing for more personalized food management. This system will reduce food waste and household expenses.

[1799] Example 2

[1800] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1801] While conventional refrigerator food management systems can grasp the type and condition of ingredients, they have difficulty managing ingredients and suggesting menus that match the user's emotions and individual lifestyles. Furthermore, while they can predict the rate of deterioration and calculate usage priorities, they do not support adjusting menus or suggesting appropriate consumption times based on the user's emotions. As a result, user satisfaction is low, food waste is often generated, and unnecessary stress is often generated. To solve these issues, a system that can recognize the user's emotional state and customize services based on that state was needed.

[1802] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for periodically acquiring video data from a video input device installed in the refrigerator, means for analyzing the video data to identify the type, condition, and deterioration rate of ingredients, means for calculating the use priority of each ingredient based on the identified information, means for visualizing the use priority and displaying it on the refrigerator's display device or a mobile information terminal, means for generating a menu of dishes based on the use priority, means for identifying shortages of ingredients and displaying it as a shopping list, means for predicting appropriate consumption times based on the deterioration rate of ingredients and notifying the user, means for recognizing the user's emotions using an emotion engine, means for adjusting the menu based on the user's emotions, and means for customizing display information based on the user's emotions. This enables personalized ingredient management and menu suggestions based on the user's emotional state, thereby reducing ingredient waste and improving user satisfaction.

[1803] The "video input device installed inside the refrigerator" refers to a camera or other image capturing device that is installed inside the refrigerator and that captures video data of the food ingredients inside the refrigerator.

[1804] "Video data" is digital data that includes image information of food items in the refrigerator.

[1805] "Type of food" refers to the specific item names of food stored in the refrigerator, such as apples, milk, eggs, etc.

[1806] "Condition of ingredients" refers to the current quality and degree of deterioration of ingredients in the refrigerator.

[1807] "Deterioration rate" is an index that indicates how quickly each food ingredient stored in a refrigerator deteriorates.

[1808] "Usage priority" is calculated based on the priority of using food items in the refrigerator, taking into account factors such as the rate of deterioration and expiration date.

[1809] A "display device" is a display or monitor attached to a refrigerator and is used to visually display information about ingredients and menu items.

[1810] A "mobile information terminal" is an electronic device such as a smartphone or tablet that can be carried by a user and that displays ingredient information and menus.

[1811] "Cooking Menu" is a meal menu suggested based on the ingredients in your refrigerator.

[1812] A "shopping list" is a list of ingredients that need to be purchased based on the ingredients in the refrigerator and the menu.

[1813] "Consumption timing" refers to the predicted optimal time to consume a food item based on its deterioration rate.

[1814] An "emotion engine" is an analysis device or software that analyzes a user's emotional state, stores it as data, and uses it for subsequent processing.

[1815] "User emotion" means the user's psychological state analyzed through the emotion engine.

[1816] "Customizing display information" means adjusting the information displayed on the refrigerator display device or mobile information terminal based on the user's emotional state and priorities.

[1817] The system embodying this invention recognizes the user's emotional state by combining an emotion engine and customizes services based on this to effectively manage ingredients in a refrigerator. The system includes a video input device installed in the refrigerator, a server, a user's mobile terminal, and an emotion engine.

[1818] Hardware and Software Configuration

[1819] 1. Video input device (camera) inside the refrigerator

[1820] A camera is installed inside the refrigerator to periodically capture video data of the ingredients. The camera can take high-resolution images and send the data to a server every hour, for example.

[1821] 2. Server

[1822] The server acquires, analyzes, stores, and processes video data. Specific software used for image recognition is OpenCV and TensorFlow. The emotion engine is used to analyze user emotion data.

[1823] 3. Terminal (refrigerator display or mobile terminal)

[1824] The terminal displays the information sent from the server, visually conveying information such as the priority of ingredients, menus, shopping lists, and consumption timing to the user.

[1825] Data processing and calculation

[1826] 1. Acquiring video data

[1827] The server periodically acquires video data from the video input device (camera) inside the refrigerator, including the type and condition of the food stored in the refrigerator.

[1828] 2. Recognizing ingredients and understanding their condition

[1829] The server uses image recognition software (e.g., OpenCV or TensorFlow) to analyze the captured video data, thereby identifying the type and condition of the food (rate of deterioration and expiration date).

[1830] 3. Calculation of usage priority

[1831] The server calculates the usage priority of each ingredient based on the type and condition data of the ingredient, assigning a higher priority to ingredients that are deteriorating or approaching their expiration date.

[1832] 4. Visualization of usage priority

[1833] The device receives the usage priority information sent from the server and displays it on the refrigerator display or mobile device. The priority is indicated by a color, allowing the user to understand the importance at a glance.

[1834] 5. Automatic menu generation

[1835] The server automatically generates menus using a generative AI model (e.g., GPT-4) based on the ingredient usage priority data, and suggests recipes that include high-priority ingredients.

[1836] 6. Listing ingredients you are lacking and making shopping suggestions

[1837] The server compares the current ingredient information with the generated menu, lists any missing ingredients, and suggests a shopping list to the user.

[1838] 7. Prediction of deterioration rate and suggestion of consumption timing

[1839] The server monitors the rate at which ingredients deteriorate and notifies the user when it is time to consume them.

[1840] 8. User Emotion Recognition

[1841] The server uses an emotion engine to recognize the user's emotions, and this data is obtained from the smartphone's camera and other sensors.

[1842] 9. Emotionally-based menu adjustments

[1843] The server uses an AI model to adjust the menu based on the user's emotional data.

[1844] 10. Emotion-Based Information Display

[1845] The device customizes the information displayed on the refrigerator display and mobile device based on the user's emotional state.

[1846] Specific examples

[1847] For example, when a user opens a refrigerator, the display shows the following information:

[1848] High priority (red): Milk (expiration date 2 days later)

[1849] Medium priority (yellow): Apple (changing color)

[1850] Low priority (green): Eggs (expiration date one week later)

[1851] When the user smiles through the camera in front of the refrigerator, the emotion engine recognizes this information. The server determines that the user is relaxed and suggests a simple and enjoyable recipe for "French toast." The mobile device also displays the following information:

[1852] French toast recipe (milk and eggs)

[1853] Missing Ingredient List: Butter

[1854] Consumption timing notification: Please consume the apples as soon as possible

[1855] Example prompts to input to the generative AI model

[1856] "Suggest a simple recipe using the milk and eggs in the fridge. The user is currently relaxed."

[1857] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1858] Step 1:

[1859] The server periodically acquires video data from a video input device (camera) installed inside the refrigerator.

[1860] Input: Footage from inside a refrigerator.

[1861] Data processing / calculation: The camera takes high-resolution images every hour and sends them to the server as digital data.

[1862] Output: Video data of food in the refrigerator.

[1863] Step 2:

[1864] The server uses image recognition software (e.g., OpenCV or TensorFlow) to analyze the captured video data, thereby identifying the type and condition of the food ingredients (rate of deterioration and expiration date).

[1865] Input: Video data of food in a refrigerator.

[1866] Data processing / calculation: Detects food ingredients such as apples, milk, and eggs from the video and calculates the degree of deterioration and expiration date of each.

[1867] Output: Data on the type, condition and deterioration rate of identified ingredients.

[1868] Step 3:

[1869] The server calculates the use priority of each ingredient based on the type and state data of the ingredient.

[1870] Input: Data on the type, condition and deterioration rate of identified ingredients.

[1871] Data processing / calculation: A mathematical algorithm is used to assign a priority score based on the degree of deterioration and expiration date. Food items that are in an advanced state of deterioration or approaching their expiration date are given a higher priority.

[1872] Output: Usage priority data for each ingredient.

[1873] Step 4:

[1874] The terminal receives the information on the usage priority sent from the server and displays it on the refrigerator's display device or the mobile terminal.

[1875] Input: Usage priority data for each ingredient.

[1876] Data processing / calculation: Usage priority is displayed in color (e.g., high priority is red, medium priority is yellow, and low priority is green).

[1877] Output: Visually displayed ingredient usage priority information.

[1878] Step 5:

[1879] The server automatically generates menus using a generative AI model (e.g., GPT-4) based on ingredient usage priority data.

[1880] Input: Usage priority data for each ingredient.

[1881] Data processing / calculation: Send the prompt "Please suggest a simple recipe using milk and eggs. The user is currently in a relaxed state" to the generative AI model and receive the generated recipe.

[1882] Output: Automatically generated menu information.

[1883] Step 6:

[1884] The server compares the current ingredient information with the generated menu, lists any missing ingredients, and suggests a shopping list to the user.

[1885] Input: Current ingredient information, automatically generated menu information.

[1886] Data processing / calculation: Matching ingredient information with the menu, identifying missing ingredients, and generating a shopping list.

[1887] Output: Shopping list.

[1888] Step 7:

[1889] The server monitors the rate at which ingredients deteriorate and notifies the user when it is time to consume the ingredients.

[1890] Input: Food deterioration rate data.

[1891] Data processing / calculation: Analyzes deterioration rate data and calculates the appropriate consumption timing. For example, generates a notification saying "Please consume the apples as soon as possible."

[1892] Output: Consumption timing notification.

[1893] Step 8:

[1894] The server uses an emotion engine to recognize the user's emotions, and this data is obtained from the smartphone's camera and other sensors.

[1895] Input: Emotion data from smartphone cameras and sensors.

[1896] Data processing / calculation: Analyze the user's facial expressions and voice, and save the results as emotional data.

[1897] Output: User emotion data.

[1898] Step 9:

[1899] The server uses an AI model to adjust the menu based on the user's emotional data.

[1900] Input: User emotion data, automatically generated menu information.

[1901] Data processing / calculation: Emotional data is taken into account and menus are adjusted to be more relaxing (e.g., simple dishes when under stress).

[1902] Output: Menu information tailored based on emotions.

[1903] Step 10:

[1904] The terminal customizes the information displayed on the refrigerator display and the mobile terminal according to the user's emotional state.

[1905] Input: User emotion data.

[1906] Data processing / calculation: Based on emotional data, information to be displayed preferentially (e.g., easy recipes) is selected.

[1907] Output: Customized display information.

[1908] (Application example 2)

[1909] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1910] Conventional refrigerator food management systems focus on calculating the priority of ingredients based on their condition and deterioration rate and notifying the user of this priority. However, because information is provided unilaterally without considering the user's emotional state, there is a problem in that it is not possible to display optimal menu suggestions or product information for the user. The present invention aims to more effectively manage ingredients and suggest products in stores by recognizing the user's emotional state in real time and providing personalized services based on that.

[1911] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1912] In this invention, the server includes means for periodically acquiring video data from a video input device installed in the refrigerator, means for analyzing the video data to identify the type, condition, and deterioration rate of ingredients, means for calculating the use priority of each ingredient based on the identified information, means for visualizing the use priority and displaying it on the refrigerator display or a mobile terminal, means for automatically generating a recipe based on the use priority, means for identifying shortages of ingredients and displaying it as a shopping list, means for predicting appropriate consumption times based on the deterioration rate of ingredients and notifying the user, means including an emotion recognition engine for recognizing the user's emotional state, means for adjusting the recipe based on the user's emotional data, and means for displaying information about products in the store in real time. This enables optimal ingredient management and personalized product suggestions according to the user's emotional state.

[1913] The "video input device" is a device that is installed inside the refrigerator and periodically acquires video data of the food ingredients inside.

[1914] An "image recognition algorithm" is a calculation method for analyzing acquired video data and identifying the type, condition, and rate of deterioration of ingredients.

[1915] "Use priority" is an index that indicates the priority of consumption based on the type, condition, and deterioration rate of food ingredients.

[1916] A "display" is a device for visually displaying information, and may be installed on the surface of a refrigerator or on a mobile device.

[1917] A "mobile terminal" is a portable electronic device such as a smartphone or tablet that receives and displays information.

[1918] A "menu" is a plan of dishes using ingredients, and is automatically generated based on the priority of using ingredients.

[1919] A "shopping list" is a list of information that lists ingredients that are in short supply and is presented for purchase.

[1920] An "emotion recognition engine" is a software or hardware element that analyzes a user's facial expressions and voice and recognizes their emotional state.

[1921] "User emotion data" is data that indicates the temporary or persistent emotional state of the user, obtained through analysis by the emotion recognition engine.

[1922] "Product suggestion" is an act or system that presents product information in a store based on the user's emotional state.

[1923] "Real-time" is a concept that indicates that information is processed immediately and provided without delay.

[1924] In this invention, the following system is implemented to realize the management of ingredients in a refrigerator and the provision of personalized services based on the user's emotional state.

[1925] First, the server periodically acquires video data from a video input device installed inside the refrigerator. An image recognition algorithm is used to analyze this video data. The image recognition algorithm identifies the type, condition, and deterioration rate of ingredients, and calculates the usage priority of each ingredient based on this information. The usage priority is visualized on the refrigerator or mobile device display, allowing users to understand the status of ingredients at a glance.

[1926] Next, the server automatically generates a menu of dishes based on the calculated usage priority. This automatic generation uses a generative AI model. The generated menu is notified to the mobile device and displayed. The server also identifies any missing ingredients and displays them as a shopping list on the mobile device.

[1927] Furthermore, the server monitors the rate at which food items deteriorate and notifies the user of the appropriate time to consume them. For example, if an apple is deteriorating faster than usual, the server will notify the user, "Consume the apple sooner." This information encourages the user to consume food appropriately according to their lifestyle.

[1928] Next, an emotion recognition engine recognizes the user's emotional state. This engine captures and analyzes facial and voice data from the smartphone's camera and other sensors. Based on the user's emotional data, the system adjusts the menu accordingly. For example, if the user is tired, it will suggest simple, relaxing dishes.

[1929] Smart glasses and head-mounted displays (HMDs) are also used to display real-time product information in stores. When a user approaches or picks up a particular product, information about that product and recommended recipes are displayed.

[1930] As a concrete example, suppose a user wearing smart glasses is walking through a supermarket and stops in front of a bottle of milk. The camera in the smart glasses captures the user's smile, and the system determines that the user is relaxed and enjoying themselves. In this case, the smart glasses will display "Fresh milk. Recommended recipe: Panna cotta."

[1931] Prompt Sentence Examples

[1932] "A camera in smart glasses captures the user's facial expressions and suggests supermarket products to the user based on their emotional state. The system analyzes the user's emotional data in real time and displays appropriate product and recipe information based on the results."

[1933] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1934] Flow of the system program that realizes the application example

[1935] Step 1:

[1936] The server periodically acquires video data from a video input device installed inside the refrigerator.

[1937] Input: Video data from a video input device.

[1938] Data processing: Video data is converted to the appropriate resolution and saved as an image file on the server.

[1939] Output: Video data in image file format.

[1940] Step 2:

[1941] The server analyzes the captured video data and uses image recognition algorithms to identify the type, condition and rate of deterioration of the food.

[1942] Input: Stored video data.

[1943] Data processing: Apply image recognition algorithms to identify the type, condition, and rate of deterioration of ingredients.

[1944] Output: Identification data including type, condition and rate of deterioration of the food material.

[1945] Step 3:

[1946] The server calculates the use priority of each ingredient based on the identified information.

[1947] Input: Identification data for food type, condition, and rate of deterioration.

[1948] Data calculation: Calculates the usage priority score based on the deterioration rate and condition.

[1949] Output: Usage priority score for each ingredient.

[1950] Step 4:

[1951] The terminal receives the usage priority information sent from the server and displays it on the refrigerator display or mobile terminal.

[1952] Input: Usage priority score.

[1953] Data transformation: Converting usage priority scores into visual formats such as color coding.

[1954] Output: Visually enhanced usage priority information.

[1955] Step 5:

[1956] The server automatically generates a menu of dishes based on the priority of use.

[1957] Input: Usage priority score for each ingredient.

[1958] Data computation: Uses generative AI models to generate optimal menus.

[1959] Output: The generated menu data.

[1960] Step 6:

[1961] The terminal receives the generated menu data and notifies the mobile terminal.

[1962] Input: Generated menu data.

[1963] Data processing: Convert menu data into notification format.

[1964] Output: Notification message to mobile device.

[1965] Step 7:

[1966] The server compares the current ingredient information with the generated menu, identifies any missing ingredients, and displays them as a shopping list.

[1967] Input: Current ingredient information, generated menu data.

[1968] Data calculation: Compare the ingredients needed for the menu with the ingredients currently available.

[1969] Output: A list of missing ingredients.

[1970] Step 8:

[1971] The server monitors the rate at which ingredients deteriorate and notifies the user of the appropriate time to consume them.

[1972] Input: Food deterioration rate data.

[1973] Data calculation: Analyzes deterioration rate data and calculates the appropriate consumption timing.

[1974] Output: Consumption timing notification message.

[1975] Step 9:

[1976] The server uses an emotion recognition engine to recognize the emotional state of the user.

[1977] Input: Facial expression and voice data captured from a smartphone's camera and sensors.

[1978] Data calculation: Apply emotion recognition algorithms to analyze the user's emotional state.

[1979] Output: User emotion data.

[1980] Step 10:

[1981] The server adjusts the menu based on the user's emotional data.

[1982] Input: User emotion data.

[1983] Data calculation: Optimal menu adjustment based on emotional data.

[1984] Output: Adjusted menu data.

[1985] Step 11:

[1986] The server displays information about products in the store in real time.

[1987] Input: Camera image data from smart glasses or head-mounted displays, and user location information.

[1988] Data calculation: Analyzes camera footage and location information to display appropriate product information.

[1989] Output: Product information shown on the user's display.

[1990] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1991] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1992] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1993] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1994] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1995] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1996] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1997] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1998] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1999] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[2000] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[2001] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[2002] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[2003] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[2004] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[2005] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific ...

Claims

1. means for periodically acquiring video data from a video input device installed in the refrigerator; means for analyzing the video data to identify the type, state, and deterioration rate of food material; a means for calculating a use priority of each ingredient based on the identified information; a means for visualizing the usage priority and displaying it on a display of the refrigerator or a mobile terminal; means for automatically generating a menu of dishes based on the use priority; A means to identify shortages of ingredients and display them as a shopping list; A means for predicting the appropriate time to consume the food based on the rate of deterioration of the food and notifying the user of the time; A system including:

2. 10. The system of claim 1, wherein the system analyzes video data from inside the refrigerator and uses an image recognition algorithm to identify the type, condition and deterioration rate of ingredients.

3. The system according to claim 1, further comprising means for notifying a mobile terminal of a menu of dishes generated based on the usage priority and displaying the menu.

Images