System

Abstract

A system is provided.SOLUTION: A system, comprising: means for scanning ingredients in a refrigerator; means for matching the scanned ingredient information against a database; means for suggesting to a user recipes that can be made with ingredients on hand based on the matched information; and means for suggesting to a user ingredients that are missing based on the matched information.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In today's busy lifestyles, managing food in the refrigerator is difficult, especially when it comes to checking food stock and choosing appropriate recipes. This often leads to food waste and unhealthy eating habits. Furthermore, unclear inventory checks can lead to unnecessary purchases or forgotten purchases. There is a need for a system that can solve these issues and support efficient refrigerator management and healthy eating habits. [Means for solving the problem]

[0005] The present invention provides a system for efficiently managing ingredients in a refrigerator and supporting a user's diet. The system includes a means for scanning ingredients in the refrigerator, a means for comparing the scanned ingredient information with a database, a means for suggesting recipes to the user that can be made with ingredients on hand based on the compared information, and a means for suggesting ingredients that the user is lacking based on the compared information. This enables automatic identification of ingredients in the refrigerator and supports efficient shopping and recipe selection. Furthermore, by using an image capture device and an image recognition algorithm, ingredient information can be obtained accurately and quickly, reducing the user's effort. Furthermore, by adding a means for managing expiration dates based on ingredient information, food waste can be prevented, leading to a healthy and economical lifestyle.

[0006] A "refrigerator" is an electrical appliance that maintains low temperatures to preserve food.

[0007] "Scanning" is the process of electronically detecting and reading specific areas or objects.

[0008] A "database" is a structured collection of data for organizing, managing, and retrieving information.

[0009] "Verification" is the process of comparing acquired data with other known data to check for matches and mismatches.

[0010] A "recipe" is a guide that provides the steps and ingredients for making a particular dish.

[0011] A "proposal" is the act of presenting a particular solution or option to others.

[0012] "Ingredients" are the raw materials used to make dishes and food.

[0013] An "image capture device" is a device that electronically captures a particular image.

[0014] An "image recognition algorithm" is a computer program that analyzes image data and identifies specific objects or features.

[0015] "Best before" is the date by which a food product can be consumed in its best condition.

[0016] "Management" is the act of effectively operating and maintaining a particular process or system. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] The system of the present invention aims to efficiently manage food ingredients in a refrigerator and support the user's eating habits. A specific implementation method of this system will be described below.

[0039] Overall system overview

[0040] This system consists of a terminal installed inside the refrigerator, a server connected via the Internet, and an application or web interface used by the user. The terminal is equipped with a camera or sensor that scans ingredients in the refrigerator. The scanned data is sent to the server, where it is analyzed and managed. Based on the analysis results, the server provides the user with information about ingredients, suggests recipes, and notifies them of any missing ingredients.

[0041] Specific implementation methods

[0042] 1. How to scan ingredients

[0043] The device periodically scans each shelf and drawer in the refrigerator. For example, the device's camera automatically activates at 10:00 every morning and captures images of the refrigerator interior. The captured image data is then analyzed using a simple image recognition algorithm to identify the type and location of ingredients. This identification data is then sent to a server via the Internet.

[0044] 2. Data analysis on the server

[0045] The server then performs detailed image recognition processing based on the received scan data. Specifically, advanced image recognition algorithms on the server determine the type, quantity, and expiration date of ingredients. For example, it can identify milk, eggs, tomatoes, etc. from the captured image and store this data in a database. At the same time, information on newly added and consumed ingredients is also updated.

[0046] 3. Recipe suggestions

[0047] The server searches a database for recipes that the user can make based on the latest list of ingredients in the refrigerator. It then suggests multiple optimal recipes based on various criteria, such as the user's past preferences, reviews, and cooking time. These recipes are then sent to the user's application or web interface. For example, if milk and eggs are available, recipes such as "French toast" and "omelette" will be suggested.

[0048] 4. Notification of missing ingredients

[0049] The server identifies frequently used ingredients that are in short supply based on the refrigerator's food list and the user's preference data. The server then checks the database for any missing ingredients and notifies the user, for example, "There's only a little lettuce left, so we recommend you buy some." The user can then create a shopping list based on this information.

[0050] Specific examples

[0051] For example, if a user puts eggs, milk, and tomatoes in the refrigerator, the system operates as follows:

[0052] 1. Every morning at 10:00, the device scans the refrigerator and captures images of eggs, milk, and tomatoes.

[0053] 2. The server analyzes the received image in detail and identifies it as 6 eggs, 500ml of milk, and 3 tomatoes.

[0054] 3. The server registers this information in a database and updates the list of ingredients in the refrigerator.

[0055] 4. The server searches for recipes that the user can make based on the ingredient list and suggests dishes such as "omelette" or "pasta with tomato sauce."

[0056] 5. The server analyzes the user's past data and notifies them, "Since you tend to make salads frequently, we recommend that you buy more lettuce."

[0057] 6. The user reviews these suggestions through the app, selects a recipe, starts cooking, and purchases any additional ingredients needed.

[0058] In this way, the system efficiently supports the user's diet and allows for easy food management in the refrigerator.

[0059] The processing flow will be explained below.

[0060] Step 1: The device starts scanning the inside of the refrigerator.

[0061] The device activates the camera inside the refrigerator and scans each shelf and drawer in turn, capturing an image of each scanned section.

[0062] Step 2: The device performs initial image recognition processing.

[0063] The device performs simple local processing on the captured image to identify the type and location of ingredients, and generates identification data (such as the type, location, and quantity of ingredients).

[0064] Step 3: The terminal sends the identification data to the server.

[0065] The terminal sends the captured image data along with the initial recognition results to the server.

[0066] Step 4: The server receives the data.

[0067] The server receives the data sent from the device and prepares for detailed image analysis.

[0068] Step 5: The server performs detailed image analysis.

[0069] The server uses advanced image recognition algorithms to identify detailed information about each ingredient (type, quantity, expiration date).

[0070] Step 6: The server checks against the database.

[0071] The server compares the identified ingredient information with a database to confirm detailed information such as the name, quantity, and expiration date.

[0072] Step 7: The server creates the ingredient list.

[0073] Based on the detailed information, the server creates an up-to-date list of ingredients in the refrigerator and stores it in a database.

[0074] Step 8: The server looks up the recipe.

[0075] Based on the updated ingredient list, the server searches its database for recipes that can be made with the ingredients on hand.

[0076] Step 9: The server evaluates and selects the recipes.

[0077] The server extracts multiple recipe candidates and evaluates and selects the optimal recipe, taking into account conditions such as cooking time and user preferences.

[0078] Step 10: The server notifies the user of the recipe.

[0079] The server notifies the user's application or web interface of the selected recipe list.

[0080] Step 11: The server identifies the missing ingredients.

[0081] The server identifies the ingredients that are missing based on the list of ingredients in the refrigerator and the user's preference data.

[0082] Step 12: The server notifies the user of the missing ingredients.

[0083] The server creates a list of identified ingredients that are in short supply and sends a notification to the user recommending their purchase.

[0084] Step 13: The user receives the notification.

[0085] The user can review the received recipe suggestions and missing ingredient notifications through the application or web interface.

[0086] Step 14: The user selects a recipe and begins cooking.

[0087] The user selects a recipe from the ones notified and begins cooking using ingredients in the refrigerator.

[0088] Step 15: The user buys more ingredients.

[0089] The user creates a shopping list based on the ingredients they are notified of and purchases the missing ingredients in stores or online.

[0090] Example 1

[0091] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0092] Conventional refrigerator food management systems have difficulty accurately grasping the types and quantities of ingredients, making it difficult to properly suggest recipes that can be made with ingredients on hand or ingredients that are missing. Furthermore, they are not sufficient in suggesting recipes based on the user's preferences or in informing the user of missing ingredients. This has resulted in insufficient efficiency in the user's diet and in managing the food in the refrigerator.

[0093] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0094] In this invention, the server includes means for periodically scanning ingredients in the refrigerator, means for transmitting the scanned ingredient information to the server, means for performing detailed analysis based on the scan data to identify the type, amount, and expiration date of the ingredients and update the database, means for suggesting recipes that can be made with the current ingredients to the user based on the updated database information, and means for notifying the user of missing ingredients based on the updated database information. This allows the server to accurately grasp ingredient information in the refrigerator and suggest appropriate recipes to the user and notify them of missing ingredients.

[0095] "Means for periodically scanning ingredients" refers to a device or process that uses a camera or sensor installed inside the refrigerator to capture images and data of ingredients at regular intervals.

[0096] The "means for transmitting scanned ingredient information to a server" is a function or process for transmitting data on ingredients scanned in the refrigerator to a remote server via the Internet or a network.

[0097] "A means of performing detailed analysis based on the scanned data, recognizing the type, quantity, and expiration date of ingredients, and updating the database" refers to a function or process that analyzes the received image data of ingredients using an advanced image recognition algorithm, recognizes detailed information about the ingredients, and reflects this in the database.

[0098] "Means for suggesting recipes to the user that can be made with the current ingredients" is a function or process that searches a database for recipes that can be made using the ingredients in the refrigerator and displays or notifies the user.

[0099] The "means for notifying the user of ingredients that are running low" is a function or process that identifies ingredients that are running low or low in the refrigerator based on database information and notifies the user.

[0100] The "camera and initial image recognition algorithm" refers to a camera device for photographing ingredients in the refrigerator and a program for analyzing the photographed images at an early stage and identifying the general type and location of the ingredients.

[0101] "Means for referencing past preference data to suggest more suitable recipes and missing ingredients" refers to a function or process that analyzes the user's past cooking and ingredient usage history and suggests individually customized recipes and missing ingredients based on that data.

[0102] MODE FOR CARRYING OUT THE INVENTION

[0103] The system of the present invention aims to efficiently manage food ingredients in a refrigerator and support the user's dietary habits. This system consists of a terminal installed in the refrigerator, a server connected via the Internet, and an application or web interface used by the user.

[0104] The device is equipped with a camera and sensors that periodically scan the food items in the refrigerator. Specifically, the camera automatically starts up at 10:00 every morning and takes pictures of each shelf and drawer in the refrigerator in sequence. This collects image data of the food items. The collected data is then simply analyzed using a basic image recognition algorithm to identify the general type and location of the food items. This identification data is then sent to a server via the Internet.

[0105] The server performs detailed image analysis based on the scanned data it receives. Specifically, an advanced image recognition algorithm on the server identifies the type, quantity, and expiration date of ingredients, and stores each piece of data in a database. As a result of the analysis, information on newly added or consumed ingredients in the refrigerator is updated.

[0106] The server then searches for recipes that the user can create based on the updated ingredient list. The server selects the best recipe based on various criteria, such as the user's past preferences, reviews, cooking time, etc. The result is reported to the user's application or web interface.

[0107] Furthermore, the server identifies ingredients that are running low based on the information in the database. This allows users to effectively manage the ingredients in their refrigerator without forgetting to buy the ingredients they need. For example, if a user frequently makes salads, the server has a function that notifies them by saying, "You're running low on lettuce, so we recommend you buy some."

[0108] Specific examples

[0109] For example, if a user puts eggs, milk, and tomatoes in the refrigerator, the system operates as follows:

[0110] 1. Every morning at 10:00, the device scans the refrigerator and captures images of eggs, milk, and tomatoes.

[0111] 2. The server analyzes the received image in detail and identifies it as 6 eggs, 500ml of milk, and 3 tomatoes.

[0112] 3. The server registers this information in a database and updates the list of ingredients in the refrigerator.

[0113] 4. Based on the ingredient list, the server suggests recipes that the user can make, such as "omelette" or "pasta with tomato sauce."

[0114] 5. The server analyzes the user's past data and notifies them, "Since you tend to make salads frequently, we recommend that you buy more lettuce."

[0115] 6. The user reviews these suggestions through the app, selects a recipe, starts cooking, and purchases any additional ingredients needed.

[0116] This system allows users to efficiently manage the ingredients in their refrigerators and support their dietary habits. It is also expected that by knowing in advance which ingredients are running low, users will be able to reduce wasteful shopping and make more effective use of ingredients.

[0117] Prompt Sentence Examples

[0118] A camera and a sensor are installed in the refrigerator. They automatically wake up every morning at 10:00 and scan the ingredients in the refrigerator. The scanned data is sent to a server, which uses advanced image recognition algorithms to identify the ingredients and update its database. The server then uses this information to suggest recipes to the user and notify them of any missing ingredients. Please explain how such a system works.

[0119] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0120] Step 1:

[0121] Start scanning food in the refrigerator

[0122] The device automatically activates the camera at 10:00 every morning and takes pictures of each shelf and drawer in the refrigerator in sequence.

[0123] Input: Video data from a camera installed in a refrigerator.

[0124] Data processing: Image capture of each shelf and drawer.

[0125] Output: Captured image data of the inside of the refrigerator.

[0126] Step 2:

[0127] Initial image recognition processing and data transmission

[0128] The device analyzes the captured image data using a rudimentary image recognition algorithm to identify the general type and location of the food item.

[0129] Input: The captured image data.

[0130] Data calculation: Identifying the type and location of ingredients using an early image recognition algorithm.

[0131] Output: Simple ingredient data identified.

[0132] The terminal transmits the identification data to a server over the Internet.

[0133] Step 3:

[0134] Detailed image analysis

[0135] The server uses advanced image recognition algorithms to perform detailed analysis of the received scan data and identify the type, quantity, and expiration date of the ingredients.

[0136] Input: Simple ingredient data and captured image data sent from the device.

[0137] Data calculation: Analysis using detailed image recognition algorithms.

[0138] Output: Detailed information about the recognized ingredients (type, quantity, expiration date).

[0139] Step 4:

[0140] Database Update

[0141] The server stores the analyzed detailed data of ingredients in a database and updates the list of ingredients in the refrigerator.

[0142] Input: Detailed data of recognized ingredients.

[0143] Data calculation: Saving and updating data in a database.

[0144] Output: Updated ingredient list.

[0145] Step 5:

[0146] Recipe Suggestions

[0147] The server searches the database for recipes that the user can make based on the latest ingredient list.

[0148] Input: Latest ingredient list, user's past preference data, reviews, cooking time.

[0149] Data Computation: Recipe search and selection algorithms.

[0150] Output: A list of recipes suggested to the user.

[0151] The server posts these recipes to the user's application or web interface.

[0152] Step 6:

[0153] Notification of shortage of ingredients

[0154] The server refers to the list of ingredients in the refrigerator and the user's preference data to identify ingredients that may be in short supply.

[0155] Input: Latest ingredient list, user preference data.

[0156] Data calculation: Algorithm for identifying missing ingredients.

[0157] Output: Information about missing ingredients notified to the user.

[0158] The server notifies the user's application of the missing ingredients.

[0159] In this way, the system streamlines food ingredient management and supports the user's eating habits.

[0160] (Application example 1)

[0161] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0162] In recent years, systems aimed at managing food ingredients in refrigerators and improving dietary habits have been attracting attention. However, while these systems can manage food ingredients, they lack the ability to quickly replenish ingredients when they run out. This requires users to manually replenish ingredients, making it difficult to achieve an efficient diet. Furthermore, if food identification and expiration date management are not adequate, food waste can occur. This increases stress and burden on users' daily lives and leads to food waste.

[0163] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0164] In this invention, the server includes a means for scanning ingredients in the refrigerator, a means for comparing the scanned ingredient information with a database, a means for suggesting recipes to the user that can be made with ingredients on hand based on the compared information, a means for suggesting ingredients that are in short supply based on the compared information, and a means for automatically ordering the ingredients that are in short supply. This allows for efficient ingredient management and rapid replenishment. Furthermore, reducing food waste can enrich the user's diet.

[0165] "Means for scanning ingredients in the refrigerator" refers to devices or sensors installed inside the refrigerator to obtain information about ingredients.

[0166] "Scanned food ingredient information" is information including data such as the type, quantity, and expiration date of the food ingredients in the refrigerator.

[0167] "Means for database matching" refers to algorithms or systems that compare scanned ingredient information with existing databases to verify and update their contents.

[0168] "Means of suggesting recipes to users that can be made using ingredients on hand" refers to software or an interface that searches for recipes that can be made using ingredients currently in the refrigerator and presents them to the user.

[0169] "Means for suggesting ingredients that are in short supply to the user" refers to a function or system that identifies ingredients that are in short supply based on information about ingredients on hand and notifies the user of the list.

[0170] "Means for automatic ordering" refers to a system that automatically orders ingredients that are in short supply from external food delivery services, etc., to replenish them.

[0171] "Image capture device" refers to a device such as a camera that takes pictures of food in the refrigerator.

[0172] "Image recognition algorithm" refers to a computer program or technology used to analyze the type and quantity of ingredients from an image captured by an image capture device.

[0173] "Expiration date management" refers to software or systems that track the expiration dates of food ingredients and notify users when the expiration date is approaching.

[0174] The present invention provides a system for efficiently managing food ingredients in a refrigerator and supporting a user's dietary habits. A specific implementation method of this system is described below.

[0175] Overall system overview

[0176] This system consists of a terminal inside the refrigerator, a server connected via the Internet, and an application or web interface used by the user. The terminal is equipped with a camera or sensor that scans ingredients in the refrigerator. The scanned data is sent to the server, where it is analyzed and managed. Based on the analysis results, the server provides the user with ingredient information, suggests recipes, and notifies them of any ingredients they are lacking, and automatically orders the missing ingredients.

[0177] Specific implementation methods

[0178] 1. How to scan ingredients

[0179] The device periodically scans each shelf and drawer in the refrigerator. For example, the device's camera automatically activates at 10:00 every morning and captures images of the refrigerator's interior. The captured image data is then analyzed using a simple image recognition algorithm to identify the type and location of ingredients. This identification data is then sent to a server via the Internet.

[0180] 2. Data analysis on the server

[0181] The server then performs detailed image recognition processing based on the received scan data. Specifically, advanced image recognition algorithms on the server determine the type, quantity, and expiration date of ingredients. For example, it can identify milk, eggs, tomatoes, etc. from the captured image and store this data in a database. At the same time, information on newly added and consumed ingredients is also updated.

[0182] 3. Recipe suggestions

[0183] The server searches a database for recipes that the user can make based on the latest list of ingredients in the refrigerator. It then suggests multiple optimal recipes based on various criteria, such as the user's past preferences, reviews, and cooking time. These recipes are then sent to the user's application or web interface. For example, if milk and eggs are available, recipes such as "French toast" and "omelette" will be suggested.

[0184] 4. Notification of shortage of ingredients and automatic ordering

[0185] The server identifies frequently used ingredients that are in short supply based on the list of ingredients in the refrigerator and the user's preference data. The missing ingredients are checked again against the database, and the user is notified, for example, "There is little lettuce left, so we recommend you buy some." In addition, a function is provided to automatically order these ingredients from a food delivery service to replenish them. This saves users the trouble of shopping and enables efficient food management.

[0186] Specific examples

[0187] For example, if a user puts eggs, milk, and tomatoes in the refrigerator, the system operates as follows:

[0188] 1. Every morning at 10:00, the device scans the refrigerator and captures images of eggs, milk, and tomatoes.

[0189] 2. The server analyzes the received image in detail and identifies it as 6 eggs, 500ml of milk, and 3 tomatoes.

[0190] 3. The server registers this information in a database and updates the list of ingredients in the refrigerator.

[0191] 4. The server searches for recipes that the user can make based on the ingredient list and suggests dishes such as "omelette" or "pasta with tomato sauce."

[0192] 5. The server analyzes the user's past data and notifies them that "Since you tend to make salads frequently, we recommend that you buy more lettuce."

[0193] 6. The server automatically orders the missing ingredients from a food delivery service to replenish them.

[0194] Prompt Sentence Examples

[0195] Example prompt:

[0196] Based on the current list of ingredients in your fridge, suggest recipes you can make and order any missing ingredients from a food delivery service. For example, if your current list of ingredients includes 500ml of eggs, 3 bottles of milk, and 2 tomatoes, suggest recipes you can make and what ingredients you're missing and automatically order them with the appropriate food delivery service.

[0197] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0198] Step 1:

[0199] The device scans the inside of the refrigerator

[0200] Input: Every morning at 10am, the camera inside the refrigerator will automatically turn on.

[0201] Data processing / data calculation: A camera captures images of the ingredients in the refrigerator.

[0202] Output: Image data is generated.

[0203] How it works: The device's built-in camera captures multiple still images or videos, which are then pre-processed using a basic image processing algorithm.

[0204] Step 2:

[0205] The device sends the scanned data to the server.

[0206] Input: Image data acquired in step 1.

[0207] Data processing / data calculation: Image data is compressed and sent to a cloud server via the Internet.

[0208] Output: Image data received by the server.

[0209] Specific operation: The device uses Wi-Fi or other Internet connection to upload image data to a designated storage area on the cloud server.

[0210] Step 3:

[0211] The server analyzes the data using image recognition algorithms

[0212] Input: Image data sent to the server.

[0213] Data processing / data calculation: Using advanced image recognition algorithms (e.g., TensorFlow or OpenCV), the type, quantity, and expiration date of ingredients are identified.

[0214] Output: Parsed food ingredient information database.

[0215] Specific operation: Image recognition software on the server analyzes the image and records information about the identified ingredients in a database.

[0216] Step 4:

[0217] The server suggests recipes to the user

[0218] Input: Ingredient information parsed in step 3.

[0219] Data processing / data calculation: Based on the ingredient information, a matching recipe is searched for from the recipe information in the database.

[0220] Output: A list of recipes suggested to the user.

[0221] Specific operation: The server selects a suitable recipe based on the user's past preferences, reviews, cooking time, etc., and notifies the user's application or web interface.

[0222] Step 5:

[0223] The server notifies the user of any ingredients that are missing.

[0224] Input: Ingredient information analyzed in step 3 and user preference data.

[0225] Data processing / data calculation: Compare the ingredient list with user preference data to identify missing ingredients.

[0226] Output: Notification of missing ingredients.

[0227] Specific operation: The server sends a notification to the user about a shortage of ingredients, such as "Lettuce is low, so we recommend you buy some."

[0228] Step 6:

[0229] The server automatically orders ingredients that are in short supply.

[0230] Input: Shortage ingredients identified in step 5.

[0231] Data processing / data calculation: Send order data for missing ingredients to the designated food delivery service.

[0232] Output: Automatic ordering to food delivery service.

[0233] Specific operation: The server uses the food delivery service's API to automatically order missing ingredients and replenish them in the user's refrigerator.

[0234] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0235] The present invention relates to a system for efficiently managing food items in a refrigerator and supporting the emotions and eating habits of a user. A specific implementation method of this system will be described below.

[0236] Overall system overview

[0237] This system consists of a terminal installed in the user's refrigerator, a server connected via the Internet, and an application or web interface used by the user. It also incorporates an emotion engine that recognizes the user's emotions. The terminal is equipped with a camera and sensors to scan ingredients in the refrigerator. The scanned data is sent to the server for analysis and management. The emotion engine analyzes the user's facial expressions, voice, and text input, and based on this, it makes recipe suggestions and notifies users of missing ingredients.

[0238] Specific implementation methods

[0239] 1. How to scan ingredients

[0240] The device periodically scans each shelf and drawer in the refrigerator. For example, the device's camera automatically activates at 10:00 every morning and captures images of the refrigerator interior. The captured image data is then analyzed using a simple image recognition algorithm to identify the type and location of ingredients. This identification data is then sent to a server via the Internet.

[0241] 2. Data analysis on the server

[0242] The server then performs detailed image recognition processing based on the received scan data. Specifically, advanced image recognition algorithms on the server determine the type, quantity, and expiration date of ingredients. For example, it can identify milk, eggs, tomatoes, etc. from the captured image and store this data in a database. At the same time, information on newly added and consumed ingredients is also updated.

[0243] 3. User Emotion Recognition

[0244] The emotion engine analyzes the user's facial expressions, voice, text input, etc. to recognize the user's emotions. For example, when a user launches the app to add ingredients, the camera captures the user's facial expressions and the voice input is analyzed by the emotion engine. This allows the emotion engine to identify the user's current emotional state (joy, anger, sadness, stress, etc.).

[0245] 4. Linking Recipe Suggestions with Emotions

[0246] The server selects recipes that the user is likely to like based on the emotional data obtained from the emotion engine. For example, if the user is feeling stressed, it will suggest recipes such as "herbal tea" or "spinach smoothie," which have a relaxing effect. If the user is happy, it will suggest recipes such as "chocolate cake," which is easy to make and fun to make.

[0247] 5. Notification of missing ingredients

[0248] The server identifies ingredients that are running low based on the list of ingredients in the refrigerator and data from the emotion engine. For example, if a user is running low on an ingredient needed for a dish they frequently cook, the server will send a notification saying, "You're running low on cabbage, so we recommend you buy some."

[0249] Specific examples

[0250] For example, if a user puts eggs, milk, and tomatoes in the refrigerator, the system operates as follows:

[0251] 1. Every morning at 10:00, the device scans the refrigerator and captures images of eggs, milk, and tomatoes.

[0252] 2. The server analyzes the received image in detail and identifies it as 6 eggs, 500ml of milk, and 3 tomatoes.

[0253] 3. The server registers this information in a database and updates the list of ingredients in the refrigerator.

[0254] 4. The emotion engine analyzes the user's facial expressions and voice and identifies their current emotion as "stress."

[0255] 5. The server will suggest a relaxing recipe, such as a "spinach smoothie," based on the ingredient list and emotional data.

[0256] 6. The server analyzes the user's past data and notifies them, "Since you tend to make salads frequently, we recommend that you buy more lettuce."

[0257] 7. The user reviews these suggestions through the application and plans to make a "spinach smoothie" and buy some lettuce.

[0258] In this way, the system efficiently supports the user's diet and makes suggestions tailored to their emotions, helping them achieve a more satisfying life.

[0259] The processing flow will be explained below.

[0260] Step 1:

[0261] The device will begin scanning the inside of the refrigerator.

[0262] The device activates the camera inside the refrigerator and scans each shelf and drawer in turn, capturing an image of each scanned section.

[0263] Step 2:

[0264] The device performs initial image recognition processing.

[0265] The device performs simple local processing on the captured image to identify the type and location of ingredients, and generates identification data (such as the type, location, and quantity of ingredients).

[0266] Step 3:

[0267] The terminal transmits the identification data to the server.

[0268] The terminal sends the captured image data along with the initial recognition results to the server.

[0269] Step 4:

[0270] The server receives the data.

[0271] The server receives the data sent from the device and prepares for detailed image analysis.

[0272] Step 5:

[0273] The server performs detailed image analysis.

[0274] The server uses advanced image recognition algorithms to identify detailed information about each ingredient (type, quantity, expiration date).

[0275] Step 6:

[0276] The server checks it against its database.

[0277] The server compares the identified ingredient information with a database to confirm detailed information such as the name, quantity, and expiration date.

[0278] Step 7:

[0279] The server creates an ingredient list.

[0280] Based on the detailed information, the server creates an up-to-date list of ingredients in the refrigerator and stores it in a database.

[0281] Step 8:

[0282] The emotion engine recognizes the user's emotions.

[0283] The emotion engine captures and analyzes data such as a user's facial expressions, voice, and text input to identify the user's emotional state.

[0284] Step 9:

[0285] The server searches for the recipe.

[0286] The server searches the database for recipes that can be made with the ingredients on hand based on the updated ingredient list and the emotional data obtained from the emotion engine.

[0287] Step 10:

[0288] The server evaluates and selects recipes.

[0289] The server extracts multiple recipe candidates and evaluates and selects the optimal recipe taking into account factors such as the user's emotional state, cooking time, and past preferences.

[0290] Step 11:

[0291] The server notifies the user of the recipe.

[0292] The server notifies the user's application or web interface of the selected recipe list.

[0293] Step 12:

[0294] The server identifies the missing ingredients.

[0295] The server identifies the ingredients that are missing based on the list of ingredients in the refrigerator and the user's preference data.

[0296] Step 13:

[0297] The server notifies the user of the missing ingredients.

[0298] The server creates a list of identified ingredients that are in short supply and sends a notification to the user recommending their purchase.

[0299] Step 14:

[0300] The user receives a notification.

[0301] The user can review the received recipe suggestions and missing ingredient notifications through the application or web interface.

[0302] Step 15:

[0303] The user selects a recipe and begins cooking.

[0304] The user selects a recipe from the ones notified and begins cooking using ingredients in the refrigerator.

[0305] Step 16:

[0306] The user buys more ingredients.

[0307] The user creates a shopping list based on the ingredients they are notified of and purchases the missing ingredients in stores or online.

[0308] With this specific processing flow, the system efficiently manages the user's ingredients and suggests recipes according to their emotional state, supporting a more satisfying diet.

[0309] Example 2

[0310] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0311] In modern life, there is a demand for efficient management of food in the refrigerator and support for dietary habits that are tailored to the user's mood. Conventional systems require manual management of food inventory and recipe suggestions, which poses a challenge in that they are unable to suggest recipes tailored to the user's mood or notify users when ingredients are in short supply. This can cause unnecessary stress for users. Furthermore, users must also manually manage expiration dates, which can lead to food waste.

[0312] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0313] In this invention, the server includes a means for scanning ingredients in the refrigerator, a means for comparing the scanned ingredient information with a database, and a means for analyzing the user's emotional state. This automates the management of ingredients in the refrigerator, making it possible to suggest recipes tailored to the user's emotions and notify the user of any missing ingredients. Furthermore, the server automatically manages expiration dates, reducing food waste and improving the user's quality of life.

[0314] "Means for scanning food ingredients in a refrigerator" is a general term for devices or mechanisms that use cameras or sensors installed in the refrigerator to acquire images and data of food ingredients in the refrigerator periodically or as needed.

[0315] "Means for comparing scanned ingredient information with a database" is a general term for systems and algorithms that analyze the images and data of ingredients obtained by scanning and compare that information with information in an existing database.

[0316] "Means for suggesting recipes to users that can be made using ingredients on hand" is a general term for algorithms and procedures for presenting users with recipes that can be made using ingredients currently in the refrigerator, based on ingredient information in the database.

[0317] "Means for suggesting ingredients that are in short supply to users" is a general term for systems and algorithms that analyze ingredient inventory based on ingredient information stored in a database and notify users of ingredients that are in short supply.

[0318] "Means for analyzing a user's emotional state" is a general term for systems or algorithms that analyze a user's facial expressions, voice, or text input to identify the user's current emotional state.

[0319] "Means for suggesting appropriate recipes to users based on their emotional state" is a general term for algorithms and systems that take into account the user's emotional state and suggest to the user the recipe that is best suited to that situation.

[0320] "Means for managing expiration dates based on food ingredient information" is a general term for systems and algorithms that automatically manage the expiration dates of each ingredient based on scanned food ingredient information and notify users.

[0321] "Means for notifying users of shortages of ingredients" is a general term for systems and algorithms that notify users when they are running low on ingredients that they frequently use or need, based on the inventory status of ingredients.

[0322] This invention relates to a system that efficiently manages food ingredients in a refrigerator and supports a dietary lifestyle that matches the user's emotions. Specifically, it is composed of a terminal installed in the refrigerator, a server connected via the Internet, and an application or web interface used by the user. It also incorporates an emotion engine that recognizes the user's emotions.

[0323] Program processing

[0324] Food scanning

[0325] The device periodically scans each shelf and drawer in the refrigerator. Specifically, the device's built-in camera (e.g., a high-resolution camera) automatically starts up at 10:00 every morning and captures images of the inside of the refrigerator. This image data is temporarily stored in the device's internal memory. The encrypted image data is then sent to a server via the Internet via a Wi-Fi module (e.g., ESP8266).

[0326] Data analysis

[0327] The server analyzes the received image data. Specifically, it uses an image recognition algorithm (e.g., TensorFlow) on the server to identify the type, quantity, and location of ingredients. The results of this analysis are registered in a database, and the list of ingredients in the refrigerator is automatically updated. At the same time, it also determines expiration dates and identifies any missing ingredients.

[0328] emotion recognition

[0329] The emotion engine analyzes the user's facial expressions, voice, and text input. For example, when a user launches an app, the device's camera and microphone capture the user's facial expressions and voice. For analysis, OpenFace and Google Cloud Speech-to-Text API are used, for example. Based on this, the emotion engine identifies the user's emotional state, and this information is sent to the server.

[0330] Recipe suggestions and missing ingredient notifications

[0331] The server suggests appropriate recipes to the user based on the data obtained from the emotion engine and the scanned ingredient information. For example, if the user is feeling stressed, it will suggest a relaxing recipe such as a "spinach smoothie." If it is determined that an ingredient is in short supply, it will generate a notification such as "We are running low on cabbage, so we recommend you buy some," and send it to the user's application.

[0332] Specific examples

[0333] For example, if a user puts eggs, milk, and tomatoes in the refrigerator, the system will do the following:

[0334] 1. Every morning at 10:00, the device scans the refrigerator and captures images of eggs, milk, and tomatoes.

[0335] 2. The server analyzes the received image in detail and identifies it as 6 eggs, 500ml of milk, and 3 tomatoes.

[0336] 3. The server registers this information in a database and updates the list of ingredients in the refrigerator.

[0337] 4. The emotion engine analyzes the user's facial expressions and voice and identifies their current emotion as "stress."

[0338] 5. Based on the ingredient list and emotional data, the server suggests a relaxing recipe: "Spinach Smoothie."

[0339] 6. The server analyzes the user's past data and notifies them, "Since you tend to make salads frequently, we recommend that you buy more lettuce."

[0340] 7. The user reviews these suggestions through the application and plans to make a "spinach smoothie" and buy some lettuce.

[0341] In this way, the system efficiently supports the user's diet and makes suggestions tailored to their emotions, helping them achieve a more satisfying life.

[0342] Prompt Sentence Examples

[0343] "I have eggs, milk, and tomatoes in my fridge, but I've been feeling stressed lately. Can you suggest a relaxing recipe using these ingredients?"

[0344] "Also, check out the other ingredients you need for salads you make frequently."

[0345] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0346] Step 1:

[0347] The device scans each shelf and drawer in the refrigerator.

[0348] Input: The device's camera captures images of the inside of the refrigerator according to the user's schedule (e.g., every morning at 10:00).

[0349] Data processing: The captured images are stored in the device's internal memory.

[0350] Specific operation: The camera operates to take pictures of the shelves inside the refrigerator in sequence and generate image data.

[0351] Output: Image data of scanned ingredients.

[0352] Step 2:

[0353] The device encrypts the scanned image data and sends it to a server via the Internet.

[0354] Input: The image data generated in step 1.

[0355] Data processing: Image data is encrypted and transmitted through the Wi-Fi module.

[0356] Specific operation: The device uses Wi-Fi to send encrypted image data to the server.

[0357] Output: Encrypted image data sent to the server.

[0358] Step 3:

[0359] The server analyzes the received image data.

[0360] Input: Encrypted image data.

[0361] Data processing: Using an image recognition algorithm (e.g., TensorFlow) on the server, the type, quantity, and location of ingredients are identified, and this information is saved and updated in the database.

[0362] Specific operation: The server runs an image recognition algorithm and stores the identified ingredient data in a database.

[0363] Output: Parsed ingredient data, updated database.

[0364] Step 4:

[0365] An emotion engine analyzes the user's emotional state.

[0366] Input: User facial expressions, voice, and text input.

[0367] Data Processing: The emotion engine analyzes this data and identifies the user's emotional state (e.g., stress, joy, sadness) using tools (e.g., OpenFace, Google Cloud Speech-to-Text API).

[0368] Specific operation: The camera captures the user's facial expressions, the microphone records audio, and the emotion engine analyzes it.

[0369] Output: Parsed emotional state data.

[0370] Step 5:

[0371] The server suggests appropriate recipes based on emotional state data and scanned ingredient data.

[0372] Input: Emotional state data, food ingredient data.

[0373] Data processing: The server searches the database for recipes that match the emotion and suggests them to the user.

[0374] Specific operation: The server performs a database search and selects recipes that match the emotion.

[0375] Output: The proposed recipe.

[0376] Step 6:

[0377] The server identifies the missing ingredients based on the scanned ingredient data and notifies the user.

[0378] Input: Ingredient data, shortage conditions.

[0379] Data processing: The server checks the ingredient list and identifies if any frequently used or necessary ingredients are missing.

[0380] Specific operation: The server analyzes the ingredient list and extracts information about missing ingredients.

[0381] Output: Identification of missing ingredients and recommended purchase notifications.

[0382] Step 7:

[0383] The server sends the suggested recipe and missing ingredient notification to the user's application.

[0384] Input: Suggested recipes, notifications of missing ingredients.

[0385] Data processing: Generation and transmission of notification data.

[0386] Specific operation: The server generates a notification and sends it to the user's smartphone.

[0387] Output: Notification to the user's smartphone.

[0388] Step 8:

[0389] The user checks the notification through the application and makes a plan to create the suggested recipe or purchase the missing ingredients.

[0390] Enter: notifications on your smartphone.

[0391] Data processing: The user decides what to do based on the notification.

[0392] Specific action: The user opens the application on their smartphone and checks the notification.

[0393] Output: Recipe creation or ingredient shopping planning.

[0394] (Application example 2)

[0395] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0396] Currently, refrigerator food management systems are limited to managing ingredients and suggesting recipes. However, they are unable to respond to the user's emotional state and are unable to provide more personalized services based on the user's emotions and eating habits. Therefore, a new system is needed that can provide comprehensive support for the user's eating habits by suggesting cooking methods based on emotions and suggesting the purchase of missing ingredients.

[0397] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for scanning ingredients in the refrigerator, means for comparing the scanned ingredient information with a database, means for suggesting to the user recipes that can be made with ingredients on hand based on the compared information, means for suggesting to the user ingredients that are missing, emotion analysis means for recognizing the user's emotional state, means for suggesting recipes appropriate for the user's emotional state based on the emotion data recognized by the emotion analysis means, and means for suggesting the purchase of ingredients that are missing. This makes it possible to suggest dietary habits that take the user's emotional state into consideration, thereby improving user satisfaction and achieving efficient ingredient management.

[0398] "Means for scanning food items in a refrigerator" refers to technology that uses image capture devices such as cameras and sensors and image recognition algorithms to detect the type, quantity, and location of food items in a refrigerator.

[0399] The "means for comparing the scanned ingredient information with a database" is a technology that compares the acquired ingredient information in the refrigerator with a pre-registered ingredient database to identify matching ingredients and new ingredients that have been added.

[0400] "Means for suggesting recipes to users that can be made using ingredients on hand" refers to technology that provides users with recipes that can be made using ingredients in the refrigerator, based on recipe information stored in a database.

[0401] The "means of suggesting missing ingredients to the user" is a technology that compares the list of ingredients in the refrigerator with recipe information and notifies the user if any ingredients needed to make a specific dish are missing.

[0402] "Emotion analysis means for recognizing the user's emotional state" is a technology that analyzes data such as the user's facial expressions, voice, and text input to identify the user's emotional state.

[0403] "Means for suggesting recipes appropriate to the emotional state based on the emotional data recognized by the emotion analysis means" refers to a technology that suggests the best dishes and drinks for a user based on the user's emotional state (e.g., stress, joy).

[0404] The "means for suggesting the purchase of ingredients that are in short supply" is a technology that notifies the user to purchase ingredients that are in short supply and supports the purchase based on the ingredient data in the refrigerator and the user's emotional state.

[0405] The following describes in detail an embodiment of the present invention. The system comprises a terminal installed in a refrigerator, a server connected via the Internet, and an application or web interface used by a user.

[0406] Food scanning method

[0407] The device is equipped with an image capture device such as a camera or sensor to scan the food items in the refrigerator. This periodically captures images of the food items in the refrigerator. The image data is analyzed using an image recognition algorithm to identify the type and quantity of food items and their expiration dates. This process uses the open source libraries OpenCV and Keras / TensorFlow.

[0408] Database matching method

[0409] The scanned food information is sent to a server via the Internet. The server compares the received data with a database to manage which food ingredients are stored in the refrigerator and in what quantities. This comparison means updates and registers new ingredients to the database.

[0410] Recipe suggestion method

[0411] The server is equipped with a means for suggesting possible recipes based on the user's available ingredients. The server analyzes the recipe information in the database and notifies the user of recipes that can be made using the ingredients in the refrigerator. The server also takes into account the user's past preference data.

[0412] Means for suggesting insufficient ingredients

[0413] The server analyzes the information to identify ingredients that are in short supply and notifies the user, allowing them to know what ingredients they need to buy. Furthermore, the server also provides a function to link with delivery services and purchase the suggested ingredients online.

[0414] Emotion analysis means

[0415] The system also includes an emotion analysis mechanism to recognize the user's emotional state by analyzing the user's facial expressions, voice, and text input data to identify emotions, using a pre-trained emotion recognition model (Keras / TensorFlow).

[0416] Emotion-based recipe suggestion method

[0417] Based on the emotional data recognized by the emotion analysis means, the server suggests cooking recipes suited to the user's emotional state, for example, if the user is under stress, it suggests dishes and drinks that have a relaxing effect.

[0418] Proposal method for purchasing ingredients that are in short supply

[0419] The server notifies the user to purchase ingredients that are in short supply based on the food data in the refrigerator and the user's emotional state, and also provides an interface for purchasing ingredients online.

[0420] Specific examples

[0421] For example, if a user comes home tired from work, the system scans the refrigerator to obtain food ingredient data, and if it detects a "stressed state" from the user's facial expression, it will suggest recipes such as relaxing herbal tea or spinach smoothie. If any ingredients are missing, the system will notify the user so that they can purchase them directly online.

[0422] Prompt Sentence Examples

[0423] Scan images of your refrigerator to suggest recipes based on current ingredients and your emotions, and let you know if you're missing any ingredients.

[0424] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0425] Step 1:

[0426] The device scans the ingredients in the refrigerator.

[0427] Input: Image data of the inside of the refrigerator.

[0428] Output: Image data of the acquired ingredients.

[0429] Specific operation: The camera installed in the refrigerator will automatically start up at a set time (for example, 10:00 every morning) and take pictures of each shelf inside the refrigerator.

[0430] Step 2:

[0431] The terminal transmits the acquired image data to the server.

[0432] Input: Image data of the acquired ingredients.

[0433] Output: Image data sent to the server.

[0434] How it works: After capturing an image, it automatically sends the image data to a server over the Internet. Encryption is used to ensure security.

[0435] Step 3:

[0436] The server analyzes the image data and extracts ingredient information.

[0437] Input: The submitted image data.

[0438] Output: Parsed ingredient information (e.g. type, quantity, expiration date).

[0439] Specific operation: The server runs an image recognition algorithm using Keras / TensorFlow to identify the type and quantity of ingredients from the image, and also determine the expiration date of each ingredient.

[0440] Step 4:

[0441] The server compares the extracted ingredient information with the database.

[0442] Input: Parsed ingredient information.

[0443] Output: Updated ingredient information in the database.

[0444] Specific operation: The server updates the ingredient list by updating the database with new ingredients and information on ingredients that have been consumed.

[0445] Step 5:

[0446] A user uses the sentiment analysis means through an application.

[0447] Input: User's facial expression, voice, and text data.

[0448] Output: The perceived emotional state of the user.

[0449] How it works: The user launches the app and inputs facial and voice data via the camera and microphone into the emotion recognition model, which analyzes this data to identify the user's current emotional state (e.g., stress, joy).

[0450] Step 6:

[0451] The server suggests cooking methods that suit the emotional state based on emotion analysis data.

[0452] Input: User's emotional state, food ingredient information from the database.

[0453] Output: A list of suggested dishes appropriate for the emotional state.

[0454] Specific operation: Based on the emotion recognition results, the server selects the recipe that best suits the user's current emotion (e.g., "spinach smoothie" if stressed) and generates a list of suggestions.

[0455] Step 7:

[0456] The server identifies the ingredients that are in short supply and notifies the user to purchase them.

[0457] Input: Updated database of ingredients, suggested recipes.

[0458] Output: A list of missing ingredients with suggested purchases.

[0459] What happens: The server checks if the ingredients needed for the proposed recipe are in its database, and if any ingredients are missing (e.g., lettuce), it lists them and generates a notification.

[0460] Step 8:

[0461] The user checks the suggested recipe and missing ingredients and uses the delivery service.

[0462] Input: Suggested recipe, list of missing ingredients suggested for purchase.

[0463] Output: Decision to purchase or use delivery service.

[0464] What it does: The user uses the app to review suggested recipes and a list of missing ingredients, and then purchases ingredients online or orders food delivery as needed.

[0465] Through these steps, the system can provide personalized dietary support tailored to the user's emotional state.

[0466] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0467] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0468] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0469] [Second embodiment]

[0470] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0471] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0472] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0473] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0474] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0475] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0476] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0477] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0478] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0479] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0480] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0481] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0482] The system of the present invention aims to efficiently manage food ingredients in a refrigerator and support the user's eating habits. A specific implementation method of this system will be described below.

[0483] Overall system overview

[0484] This system consists of a terminal installed inside the refrigerator, a server connected via the Internet, and an application or web interface used by the user. The terminal is equipped with a camera or sensor that scans ingredients in the refrigerator. The scanned data is sent to the server, where it is analyzed and managed. Based on the analysis results, the server provides the user with information about ingredients, suggests recipes, and notifies them of any missing ingredients.

[0485] Specific implementation methods

[0486] 1. How to scan ingredients

[0487] The device periodically scans each shelf and drawer in the refrigerator. For example, the device's camera automatically activates at 10:00 every morning and captures images of the refrigerator interior. The captured image data is then analyzed using a simple image recognition algorithm to identify the type and location of ingredients. This identification data is then sent to a server via the Internet.

[0488] 2. Data analysis on the server

[0489] The server then performs detailed image recognition processing based on the received scan data. Specifically, advanced image recognition algorithms on the server determine the type, quantity, and expiration date of ingredients. For example, it can identify milk, eggs, tomatoes, etc. from the captured image and store this data in a database. At the same time, information on newly added and consumed ingredients is also updated.

[0490] 3. Recipe suggestions

[0491] The server searches a database for recipes that the user can make based on the latest list of ingredients in the refrigerator. It then suggests multiple optimal recipes based on various criteria, such as the user's past preferences, reviews, and cooking time. These recipes are then sent to the user's application or web interface. For example, if milk and eggs are available, recipes such as "French toast" and "omelette" will be suggested.

[0492] 4. Notification of missing ingredients

[0493] The server identifies frequently used ingredients that are in short supply based on the refrigerator's food list and the user's preference data. The server then checks the database for any missing ingredients and notifies the user, for example, "There's only a little lettuce left, so we recommend you buy some." The user can then create a shopping list based on this information.

[0494] Specific examples

[0495] For example, if a user puts eggs, milk, and tomatoes in the refrigerator, the system operates as follows:

[0496] 1. Every morning at 10:00, the device scans the refrigerator and captures images of eggs, milk, and tomatoes.

[0497] 2. The server analyzes the received image in detail and identifies it as 6 eggs, 500ml of milk, and 3 tomatoes.

[0498] 3. The server registers this information in a database and updates the list of ingredients in the refrigerator.

[0499] 4. The server searches for recipes that the user can make based on the ingredient list and suggests dishes such as "omelette" or "pasta with tomato sauce."

[0500] 5. The server analyzes the user's past data and notifies them, "Since you tend to make salads frequently, we recommend that you buy more lettuce."

[0501] 6. The user reviews these suggestions through the app, selects a recipe, starts cooking, and purchases any additional ingredients needed.

[0502] In this way, the system efficiently supports the user's diet and allows for easy food management in the refrigerator.

[0503] The processing flow will be explained below.

[0504] Step 1: The device starts scanning the inside of the refrigerator.

[0505] The device activates the camera inside the refrigerator and scans each shelf and drawer in turn, capturing an image of each scanned section.

[0506] Step 2: The device performs initial image recognition processing.

[0507] The device performs simple local processing on the captured image to identify the type and location of ingredients, and generates identification data (such as the type, location, and quantity of ingredients).

[0508] Step 3: The terminal sends the identification data to the server.

[0509] The terminal sends the captured image data along with the initial recognition results to the server.

[0510] Step 4: The server receives the data.

[0511] The server receives the data sent from the device and prepares for detailed image analysis.

[0512] Step 5: The server performs detailed image analysis.

[0513] The server uses advanced image recognition algorithms to identify detailed information about each ingredient (type, quantity, expiration date).

[0514] Step 6: The server checks against the database.

[0515] The server compares the identified ingredient information with a database to confirm detailed information such as the name, quantity, and expiration date.

[0516] Step 7: The server creates the ingredient list.

[0517] Based on the detailed information, the server creates an up-to-date list of ingredients in the refrigerator and stores it in a database.

[0518] Step 8: The server looks up the recipe.

[0519] Based on the updated ingredient list, the server searches its database for recipes that can be made with the ingredients on hand.

[0520] Step 9: The server evaluates and selects the recipes.

[0521] The server extracts multiple recipe candidates and evaluates and selects the optimal recipe, taking into account conditions such as cooking time and user preferences.

[0522] Step 10: The server notifies the user of the recipe.

[0523] The server notifies the user's application or web interface of the selected recipe list.

[0524] Step 11: The server identifies the missing ingredients.

[0525] The server identifies the ingredients that are missing based on the list of ingredients in the refrigerator and the user's preference data.

[0526] Step 12: The server notifies the user of the missing ingredients.

[0527] The server creates a list of identified ingredients that are in short supply and sends a notification to the user recommending their purchase.

[0528] Step 13: The user receives the notification.

[0529] The user can review the received recipe suggestions and missing ingredient notifications through the application or web interface.

[0530] Step 14: The user selects a recipe and begins cooking.

[0531] The user selects a recipe from the ones notified and begins cooking using ingredients in the refrigerator.

[0532] Step 15: The user buys more ingredients.

[0533] The user creates a shopping list based on the ingredients they are notified of and purchases the missing ingredients in stores or online.

[0534] Example 1

[0535] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0536] Conventional refrigerator food management systems have difficulty accurately grasping the types and quantities of ingredients, making it difficult to properly suggest recipes that can be made with ingredients on hand or ingredients that are missing. Furthermore, they are not sufficient in suggesting recipes based on the user's preferences or in informing the user of missing ingredients. This has resulted in insufficient efficiency in the user's diet and in managing the food in the refrigerator.

[0537] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0538] In this invention, the server includes means for periodically scanning ingredients in the refrigerator, means for transmitting the scanned ingredient information to the server, means for performing detailed analysis based on the scan data to identify the type, amount, and expiration date of the ingredients and update the database, means for suggesting recipes that can be made with the current ingredients to the user based on the updated database information, and means for notifying the user of missing ingredients based on the updated database information. This allows the server to accurately grasp ingredient information in the refrigerator and suggest appropriate recipes to the user and notify them of missing ingredients.

[0539] "Means for periodically scanning ingredients" refers to a device or process that uses a camera or sensor installed inside the refrigerator to capture images and data of ingredients at regular intervals.

[0540] The "means for transmitting scanned ingredient information to a server" is a function or process for transmitting data on ingredients scanned in the refrigerator to a remote server via the Internet or a network.

[0541] "A means of performing detailed analysis based on the scanned data, recognizing the type, quantity, and expiration date of ingredients, and updating the database" refers to a function or process that analyzes the received image data of ingredients using an advanced image recognition algorithm, recognizes detailed information about the ingredients, and reflects this in the database.

[0542] "Means for suggesting recipes to the user that can be made with the current ingredients" is a function or process that searches a database for recipes that can be made using the ingredients in the refrigerator and displays or notifies the user.

[0543] The "means for notifying the user of ingredients that are running low" is a function or process that identifies ingredients that are running low or low in the refrigerator based on database information and notifies the user.

[0544] The "camera and initial image recognition algorithm" refers to a camera device for photographing ingredients in the refrigerator and a program for analyzing the photographed images at an early stage and identifying the general type and location of the ingredients.

[0545] "Means for referencing past preference data to suggest more suitable recipes and missing ingredients" refers to a function or process that analyzes the user's past cooking and ingredient usage history and suggests individually customized recipes and missing ingredients based on that data.

[0546] MODE FOR CARRYING OUT THE INVENTION

[0547] The system of the present invention aims to efficiently manage food ingredients in a refrigerator and support the user's dietary habits. This system consists of a terminal installed in the refrigerator, a server connected via the Internet, and an application or web interface used by the user.

[0548] The device is equipped with a camera and sensors that periodically scan the food items in the refrigerator. Specifically, the camera automatically starts up at 10:00 every morning and takes pictures of each shelf and drawer in the refrigerator in sequence. This collects image data of the food items. The collected data is then simply analyzed using a basic image recognition algorithm to identify the general type and location of the food items. This identification data is then sent to a server via the Internet.

[0549] The server performs detailed image analysis based on the scanned data it receives. Specifically, an advanced image recognition algorithm on the server identifies the type, quantity, and expiration date of ingredients, and stores each piece of data in a database. As a result of the analysis, information on newly added or consumed ingredients in the refrigerator is updated.

[0550] The server then searches for recipes that the user can create based on the updated ingredient list. The server selects the best recipe based on various criteria, such as the user's past preferences, reviews, cooking time, etc. The result is reported to the user's application or web interface.

[0551] Furthermore, the server identifies ingredients that are running low based on the information in the database. This allows users to effectively manage the ingredients in their refrigerator without forgetting to buy the ingredients they need. For example, if a user frequently makes salads, the server has a function that notifies them by saying, "You're running low on lettuce, so we recommend you buy some."

[0552] Specific examples

[0553] For example, if a user puts eggs, milk, and tomatoes in the refrigerator, the system operates as follows:

[0554] 1. Every morning at 10:00, the device scans the refrigerator and captures images of eggs, milk, and tomatoes.

[0555] 2. The server analyzes the received image in detail and identifies it as 6 eggs, 500ml of milk, and 3 tomatoes.

[0556] 3. The server registers this information in a database and updates the list of ingredients in the refrigerator.

[0557] 4. Based on the ingredient list, the server suggests recipes that the user can make, such as "omelette" or "pasta with tomato sauce."

[0558] 5. The server analyzes the user's past data and notifies them, "Since you tend to make salads frequently, we recommend that you buy more lettuce."

[0559] 6. The user reviews these suggestions through the app, selects a recipe, starts cooking, and purchases any additional ingredients needed.

[0560] This system allows users to efficiently manage the ingredients in their refrigerators and support their dietary habits. It is also expected that by knowing in advance which ingredients are running low, users will be able to reduce wasteful shopping and make more effective use of ingredients.

[0561] Prompt Sentence Examples

[0562] A camera and a sensor are installed in the refrigerator. They automatically wake up every morning at 10:00 and scan the ingredients in the refrigerator. The scanned data is sent to a server, which uses advanced image recognition algorithms to identify the ingredients and update its database. The server then uses this information to suggest recipes to the user and notify them of any missing ingredients. Please explain how such a system works.

[0563] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0564] Step 1:

[0565] Start scanning food in the refrigerator

[0566] The device automatically activates the camera at 10:00 every morning and takes pictures of each shelf and drawer in the refrigerator in sequence.

[0567] Input: Video data from a camera installed in a refrigerator.

[0568] Data processing: Image capture of each shelf and drawer.

[0569] Output: Captured image data of the inside of the refrigerator.

[0570] Step 2:

[0571] Initial image recognition processing and data transmission

[0572] The device analyzes the captured image data using a rudimentary image recognition algorithm to identify the general type and location of the food item.

[0573] Input: The captured image data.

[0574] Data calculation: Identifying the type and location of ingredients using an early image recognition algorithm.

[0575] Output: Simple ingredient data identified.

[0576] The terminal transmits the identification data to a server over the Internet.

[0577] Step 3:

[0578] Detailed image analysis

[0579] The server uses advanced image recognition algorithms to perform detailed analysis of the received scan data and identify the type, quantity, and expiration date of the ingredients.

[0580] Input: Simple ingredient data and captured image data sent from the device.

[0581] Data calculation: Analysis using detailed image recognition algorithms.

[0582] Output: Detailed information about the recognized ingredients (type, quantity, expiration date).

[0583] Step 4:

[0584] Database Update

[0585] The server stores the analyzed detailed data of ingredients in a database and updates the list of ingredients in the refrigerator.

[0586] Input: Detailed data of recognized ingredients.

[0587] Data calculation: Saving and updating data in a database.

[0588] Output: Updated ingredient list.

[0589] Step 5:

[0590] Recipe Suggestions

[0591] The server searches the database for recipes that the user can make based on the latest ingredient list.

[0592] Input: Latest ingredient list, user's past preference data, reviews, cooking time.

[0593] Data Computation: Recipe search and selection algorithms.

[0594] Output: A list of recipes suggested to the user.

[0595] The server posts these recipes to the user's application or web interface.

[0596] Step 6:

[0597] Notification of shortage of ingredients

[0598] The server refers to the list of ingredients in the refrigerator and the user's preference data to identify ingredients that may be in short supply.

[0599] Input: Latest ingredient list, user preference data.

[0600] Data calculation: Algorithm for identifying missing ingredients.

[0601] Output: Information about missing ingredients notified to the user.

[0602] The server notifies the user's application of the missing ingredients.

[0603] In this way, the system streamlines food ingredient management and supports the user's eating habits.

[0604] (Application example 1)

[0605] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0606] In recent years, systems aimed at managing food ingredients in refrigerators and improving dietary habits have been attracting attention. However, while these systems can manage food ingredients, they lack the ability to quickly replenish ingredients when they run out. This requires users to manually replenish ingredients, making it difficult to achieve an efficient diet. Furthermore, if food identification and expiration date management are not adequate, food waste can occur. This increases stress and burden on users' daily lives and leads to food waste.

[0607] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0608] In this invention, the server includes a means for scanning ingredients in the refrigerator, a means for comparing the scanned ingredient information with a database, a means for suggesting recipes to the user that can be made with ingredients on hand based on the compared information, a means for suggesting ingredients that are in short supply based on the compared information, and a means for automatically ordering the ingredients that are in short supply. This allows for efficient ingredient management and rapid replenishment. Furthermore, reducing food waste can enrich the user's diet.

[0609] "Means for scanning ingredients in the refrigerator" refers to devices or sensors installed inside the refrigerator to obtain information about ingredients.

[0610] "Scanned food ingredient information" is information including data such as the type, quantity, and expiration date of the food ingredients in the refrigerator.

[0611] "Means for database matching" refers to algorithms or systems that compare scanned ingredient information with existing databases to verify and update their contents.

[0612] "Means of suggesting recipes to users that can be made using ingredients on hand" refers to software or an interface that searches for recipes that can be made using ingredients currently in the refrigerator and presents them to the user.

[0613] "Means for suggesting ingredients that are in short supply to the user" refers to a function or system that identifies ingredients that are in short supply based on information about ingredients on hand and notifies the user of the list.

[0614] "Means for automatic ordering" refers to a system that automatically orders ingredients that are in short supply from external food delivery services, etc., to replenish them.

[0615] "Image capture device" refers to a device such as a camera that takes pictures of food in the refrigerator.

[0616] "Image recognition algorithm" refers to a computer program or technology used to analyze the type and quantity of ingredients from an image captured by an image capture device.

[0617] "Expiration date management" refers to software or systems that track the expiration dates of food ingredients and notify users when the expiration date is approaching.

[0618] The present invention provides a system for efficiently managing food ingredients in a refrigerator and supporting a user's dietary habits. A specific implementation method of this system is described below.

[0619] Overall system overview

[0620] This system consists of a terminal inside the refrigerator, a server connected via the Internet, and an application or web interface used by the user. The terminal is equipped with a camera or sensor that scans ingredients in the refrigerator. The scanned data is sent to the server, where it is analyzed and managed. Based on the analysis results, the server provides the user with ingredient information, suggests recipes, and notifies them of any ingredients they are lacking, and automatically orders the missing ingredients.

[0621] Specific implementation methods

[0622] 1. How to scan ingredients

[0623] The device periodically scans each shelf and drawer in the refrigerator. For example, the device's camera automatically activates at 10:00 every morning and captures images of the refrigerator's interior. The captured image data is then analyzed using a simple image recognition algorithm to identify the type and location of ingredients. This identification data is then sent to a server via the Internet.

[0624] 2. Data analysis on the server

[0625] The server then performs detailed image recognition processing based on the received scan data. Specifically, advanced image recognition algorithms on the server determine the type, quantity, and expiration date of ingredients. For example, it can identify milk, eggs, tomatoes, etc. from the captured image and store this data in a database. At the same time, information on newly added and consumed ingredients is also updated.

[0626] 3. Recipe suggestions

[0627] The server searches a database for recipes that the user can make based on the latest list of ingredients in the refrigerator. It then suggests multiple optimal recipes based on various criteria, such as the user's past preferences, reviews, and cooking time. These recipes are then sent to the user's application or web interface. For example, if milk and eggs are available, recipes such as "French toast" and "omelette" will be suggested.

[0628] 4. Notification of shortage of ingredients and automatic ordering

[0629] The server identifies frequently used ingredients that are in short supply based on the list of ingredients in the refrigerator and the user's preference data. The missing ingredients are checked again against the database, and the user is notified, for example, "There is little lettuce left, so we recommend you buy some." In addition, a function is provided to automatically order these ingredients from a food delivery service to replenish them. This saves users the trouble of shopping and enables efficient food management.

[0630] Specific examples

[0631] For example, if a user puts eggs, milk, and tomatoes in the refrigerator, the system operates as follows:

[0632] 1. Every morning at 10:00, the device scans the refrigerator and captures images of eggs, milk, and tomatoes.

[0633] 2. The server analyzes the received image in detail and identifies it as 6 eggs, 500ml of milk, and 3 tomatoes.

[0634] 3. The server registers this information in a database and updates the list of ingredients in the refrigerator.

[0635] 4. The server searches for recipes that the user can make based on the ingredient list and suggests dishes such as "omelette" or "pasta with tomato sauce."

[0636] 5. The server analyzes the user's past data and notifies them that "Since you tend to make salads frequently, we recommend that you buy more lettuce."

[0637] 6. The server automatically orders the missing ingredients from a food delivery service to replenish them.

[0638] Prompt Sentence Examples

[0639] Example prompt:

[0640] Based on the current list of ingredients in your fridge, suggest recipes you can make and order any missing ingredients from a food delivery service. For example, if your current list of ingredients includes 500ml of eggs, 3 bottles of milk, and 2 tomatoes, suggest recipes you can make and what ingredients you're missing and automatically order them with the appropriate food delivery service.

[0641] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0642] Step 1:

[0643] The device scans the inside of the refrigerator

[0644] Input: Every morning at 10am, the camera inside the refrigerator will automatically turn on.

[0645] Data processing / data calculation: A camera captures images of the ingredients in the refrigerator.

[0646] Output: Image data is generated.

[0647] How it works: The device's built-in camera captures multiple still images or videos, which are then pre-processed using a basic image processing algorithm.

[0648] Step 2:

[0649] The device sends the scanned data to the server.

[0650] Input: Image data acquired in step 1.

[0651] Data processing / data calculation: Image data is compressed and sent to a cloud server via the Internet.

[0652] Output: Image data received by the server.

[0653] Specific operation: The device uses Wi-Fi or other Internet connection to upload image data to a designated storage area on the cloud server.

[0654] Step 3:

[0655] The server analyzes the data using image recognition algorithms

[0656] Input: Image data sent to the server.

[0657] Data processing / data calculation: Using advanced image recognition algorithms (e.g., TensorFlow or OpenCV), the type, quantity, and expiration date of ingredients are identified.

[0658] Output: Parsed food ingredient information database.

[0659] Specific operation: Image recognition software on the server analyzes the image and records information about the identified ingredients in a database.

[0660] Step 4:

[0661] The server suggests recipes to the user

[0662] Input: Ingredient information parsed in step 3.

[0663] Data processing / data calculation: Based on the ingredient information, a matching recipe is searched for from the recipe information in the database.

[0664] Output: A list of recipes suggested to the user.

[0665] Specific operation: The server selects a suitable recipe based on the user's past preferences, reviews, cooking time, etc., and notifies the user's application or web interface.

[0666] Step 5:

[0667] The server notifies the user of any ingredients that are missing.

[0668] Input: Ingredient information analyzed in step 3 and user preference data.

[0669] Data processing / data calculation: Compare the ingredient list with user preference data to identify missing ingredients.

[0670] Output: Notification of missing ingredients.

[0671] Specific operation: The server sends a notification to the user about a shortage of ingredients, such as "Lettuce is low, so we recommend you buy some."

[0672] Step 6:

[0673] The server automatically orders ingredients that are in short supply.

[0674] Input: Shortage ingredients identified in step 5.

[0675] Data processing / data calculation: Send order data for missing ingredients to the designated food delivery service.

[0676] Output: Automatic ordering to food delivery service.

[0677] Specific operation: The server uses the food delivery service's API to automatically order missing ingredients and replenish them in the user's refrigerator.

[0678] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0679] The present invention relates to a system for efficiently managing food items in a refrigerator and supporting the emotions and eating habits of a user. A specific implementation method of this system will be described below.

[0680] Overall system overview

[0681] This system consists of a terminal installed in the user's refrigerator, a server connected via the Internet, and an application or web interface used by the user. It also incorporates an emotion engine that recognizes the user's emotions. The terminal is equipped with a camera and sensors to scan ingredients in the refrigerator. The scanned data is sent to the server for analysis and management. The emotion engine analyzes the user's facial expressions, voice, and text input, and based on this, it makes recipe suggestions and notifies users of missing ingredients.

[0682] Specific implementation methods

[0683] 1. How to scan ingredients

[0684] The device periodically scans each shelf and drawer in the refrigerator. For example, the device's camera automatically activates at 10:00 every morning and captures images of the refrigerator interior. The captured image data is then analyzed using a simple image recognition algorithm to identify the type and location of ingredients. This identification data is then sent to a server via the Internet.

[0685] 2. Data analysis on the server

[0686] The server then performs detailed image recognition processing based on the received scan data. Specifically, advanced image recognition algorithms on the server determine the type, quantity, and expiration date of ingredients. For example, it can identify milk, eggs, tomatoes, etc. from the captured image and store this data in a database. At the same time, information on newly added and consumed ingredients is also updated.

[0687] 3. User Emotion Recognition

[0688] The emotion engine analyzes the user's facial expressions, voice, text input, etc. to recognize the user's emotions. For example, when a user launches the app to add ingredients, the camera captures the user's facial expressions and the voice input is analyzed by the emotion engine. This allows the emotion engine to identify the user's current emotional state (joy, anger, sadness, stress, etc.).

[0689] 4. Linking Recipe Suggestions with Emotions

[0690] The server selects recipes that the user is likely to like based on the emotional data obtained from the emotion engine. For example, if the user is feeling stressed, it will suggest recipes such as "herbal tea" or "spinach smoothie," which have a relaxing effect. If the user is happy, it will suggest recipes such as "chocolate cake," which is easy to make and fun to make.

[0691] 5. Notification of missing ingredients

[0692] The server identifies ingredients that are running low based on the list of ingredients in the refrigerator and data from the emotion engine. For example, if a user is running low on an ingredient needed for a dish they frequently cook, the server will send a notification saying, "You're running low on cabbage, so we recommend you buy some."

[0693] Specific examples

[0694] For example, if a user puts eggs, milk, and tomatoes in the refrigerator, the system operates as follows:

[0695] 1. Every morning at 10:00, the device scans the refrigerator and captures images of eggs, milk, and tomatoes.

[0696] 2. The server analyzes the received image in detail and identifies it as 6 eggs, 500ml of milk, and 3 tomatoes.

[0697] 3. The server registers this information in a database and updates the list of ingredients in the refrigerator.

[0698] 4. The emotion engine analyzes the user's facial expressions and voice and identifies their current emotion as "stress."

[0699] 5. The server will suggest a relaxing recipe, such as a "spinach smoothie," based on the ingredient list and emotional data.

[0700] 6. The server analyzes the user's past data and notifies them, "Since you tend to make salads frequently, we recommend that you buy more lettuce."

[0701] 7. The user reviews these suggestions through the application and plans to make a "spinach smoothie" and buy some lettuce.

[0702] In this way, the system efficiently supports the user's diet and makes suggestions tailored to their emotions, helping them achieve a more satisfying life.

[0703] The processing flow will be explained below.

[0704] Step 1:

[0705] The device will begin scanning the inside of the refrigerator.

[0706] The device activates the camera inside the refrigerator and scans each shelf and drawer in turn, capturing an image of each scanned section.

[0707] Step 2:

[0708] The device performs initial image recognition processing.

[0709] The device performs simple local processing on the captured image to identify the type and location of ingredients, and generates identification data (such as the type, location, and quantity of ingredients).

[0710] Step 3:

[0711] The terminal transmits the identification data to the server.

[0712] The terminal sends the captured image data along with the initial recognition results to the server.

[0713] Step 4:

[0714] The server receives the data.

[0715] The server receives the data sent from the device and prepares for detailed image analysis.

[0716] Step 5:

[0717] The server performs detailed image analysis.

[0718] The server uses advanced image recognition algorithms to identify detailed information about each ingredient (type, quantity, expiration date).

[0719] Step 6:

[0720] The server checks it against its database.

[0721] The server compares the identified ingredient information with a database to confirm detailed information such as the name, quantity, and expiration date.

[0722] Step 7:

[0723] The server creates an ingredient list.

[0724] Based on the detailed information, the server creates an up-to-date list of ingredients in the refrigerator and stores it in a database.

[0725] Step 8:

[0726] The emotion engine recognizes the user's emotions.

[0727] The emotion engine captures and analyzes data such as a user's facial expressions, voice, and text input to identify the user's emotional state.

[0728] Step 9:

[0729] The server searches for the recipe.

[0730] The server searches the database for recipes that can be made with the ingredients on hand based on the updated ingredient list and the emotional data obtained from the emotion engine.

[0731] Step 10:

[0732] The server evaluates and selects recipes.

[0733] The server extracts multiple recipe candidates and evaluates and selects the optimal recipe taking into account factors such as the user's emotional state, cooking time, and past preferences.

[0734] Step 11:

[0735] The server notifies the user of the recipe.

[0736] The server notifies the user's application or web interface of the selected recipe list.

[0737] Step 12:

[0738] The server identifies the missing ingredients.

[0739] The server identifies the ingredients that are missing based on the list of ingredients in the refrigerator and the user's preference data.

[0740] Step 13:

[0741] The server notifies the user of the missing ingredients.

[0742] The server creates a list of identified ingredients that are in short supply and sends a notification to the user recommending their purchase.

[0743] Step 14:

[0744] The user receives a notification.

[0745] The user can review the received recipe suggestions and missing ingredient notifications through the application or web interface.

[0746] Step 15:

[0747] The user selects a recipe and begins cooking.

[0748] The user selects a recipe from the ones notified and begins cooking using ingredients in the refrigerator.

[0749] Step 16:

[0750] The user buys more ingredients.

[0751] The user creates a shopping list based on the ingredients they are notified of and purchases the missing ingredients in stores or online.

[0752] With this specific processing flow, the system efficiently manages the user's ingredients and suggests recipes according to their emotional state, supporting a more satisfying diet.

[0753] Example 2

[0754] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0755] In modern life, there is a demand for efficient management of food in the refrigerator and support for dietary habits that are tailored to the user's mood. Conventional systems require manual management of food inventory and recipe suggestions, which poses a challenge in that they are unable to suggest recipes tailored to the user's mood or notify users when ingredients are in short supply. This can cause unnecessary stress for users. Furthermore, users must also manually manage expiration dates, which can lead to food waste.

[0756] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0757] In this invention, the server includes a means for scanning ingredients in the refrigerator, a means for comparing the scanned ingredient information with a database, and a means for analyzing the user's emotional state. This automates the management of ingredients in the refrigerator, making it possible to suggest recipes tailored to the user's emotions and notify the user of any missing ingredients. Furthermore, the server automatically manages expiration dates, reducing food waste and improving the user's quality of life.

[0758] "Means for scanning food items in a refrigerator" is a general term for devices or mechanisms that use cameras or sensors installed in the refrigerator to acquire images and data of food items in the refrigerator periodically or as needed.

[0759] "Means for comparing scanned ingredient information with a database" is a general term for systems and algorithms that analyze the images and data of ingredients obtained by scanning and compare that information with information in an existing database.

[0760] "Means for suggesting recipes to users that can be made using ingredients on hand" is a general term for algorithms and procedures for presenting users with recipes that can be made using ingredients currently in the refrigerator, based on ingredient information in the database.

[0761] "Means for suggesting ingredients that are in short supply to users" is a general term for systems and algorithms that analyze ingredient inventory based on ingredient information stored in a database and notify users of ingredients that are in short supply.

[0762] "Means for analyzing a user's emotional state" is a general term for systems or algorithms that analyze a user's facial expressions, voice, or text input to identify the user's current emotional state.

[0763] "Means for suggesting appropriate recipes to users based on their emotional state" is a general term for algorithms and systems that take into account the user's emotional state and suggest to the user the recipe that is best suited to that situation.

[0764] "Means for managing expiration dates based on food ingredient information" is a general term for systems and algorithms that automatically manage the expiration dates of each ingredient based on scanned food ingredient information and notify users.

[0765] "Means for notifying users of shortages of ingredients" is a general term for systems and algorithms that notify users when they are running low on ingredients that they frequently use or need, based on the inventory status of ingredients.

[0766] This invention relates to a system that efficiently manages food ingredients in a refrigerator and supports a dietary lifestyle that matches the user's emotions. Specifically, it is composed of a terminal installed in the refrigerator, a server connected via the Internet, and an application or web interface used by the user. It also incorporates an emotion engine that recognizes the user's emotions.

[0767] Program processing

[0768] Food scanning

[0769] The device periodically scans each shelf and drawer in the refrigerator. Specifically, the device's built-in camera (e.g., a high-resolution camera) automatically starts up at 10:00 every morning and captures images of the inside of the refrigerator. This image data is temporarily stored in the device's internal memory. The encrypted image data is then sent to a server via the Internet via a Wi-Fi module (e.g., ESP8266).

[0770] Data analysis

[0771] The server analyzes the received image data. Specifically, it uses an image recognition algorithm (e.g., TensorFlow) on the server to identify the type, quantity, and location of ingredients. The results of this analysis are registered in a database, and the list of ingredients in the refrigerator is automatically updated. At the same time, it also determines expiration dates and identifies any missing ingredients.

[0772] emotion recognition

[0773] The emotion engine analyzes the user's facial expressions, voice, and text input. For example, when a user launches an app, the device's camera and microphone capture the user's facial expressions and voice. For analysis, OpenFace and Google Cloud Speech-to-Text API are used, for example. Based on this, the emotion engine identifies the user's emotional state, and this information is sent to the server.

[0774] Recipe suggestions and missing ingredient notifications

[0775] The server suggests appropriate recipes to the user based on the data obtained from the emotion engine and the scanned ingredient information. For example, if the user is feeling stressed, it will suggest a relaxing recipe such as a "spinach smoothie." If it is determined that an ingredient is in short supply, it will generate a notification such as "We are running low on cabbage, so we recommend you buy some," and send it to the user's application.

[0776] Specific examples

[0777] For example, if a user puts eggs, milk, and tomatoes in the refrigerator, the system will do the following:

[0778] 1. Every morning at 10:00, the device scans the refrigerator and captures images of eggs, milk, and tomatoes.

[0779] 2. The server analyzes the received image in detail and identifies it as 6 eggs, 500ml of milk, and 3 tomatoes.

[0780] 3. The server registers this information in a database and updates the list of ingredients in the refrigerator.

[0781] 4. The emotion engine analyzes the user's facial expressions and voice and identifies their current emotion as "stress."

[0782] 5. Based on the ingredient list and emotional data, the server suggests a relaxing recipe: "Spinach Smoothie."

[0783] 6. The server analyzes the user's past data and notifies them, "Since you tend to make salads frequently, we recommend that you buy more lettuce."

[0784] 7. The user reviews these suggestions through the application and plans to make a "spinach smoothie" and buy some lettuce.

[0785] In this way, the system efficiently supports the user's diet and makes suggestions tailored to their emotions, helping them achieve a more satisfying life.

[0786] Prompt Sentence Examples

[0787] "I have eggs, milk, and tomatoes in my fridge, but I've been feeling stressed lately. Can you suggest a relaxing recipe using these ingredients?"

[0788] "Also, check out the other ingredients you need for salads you make frequently."

[0789] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0790] Step 1:

[0791] The device scans each shelf and drawer in the refrigerator.

[0792] Input: The device's camera captures images of the inside of the refrigerator according to the user's schedule (e.g., every morning at 10:00).

[0793] Data processing: The captured images are stored in the device's internal memory.

[0794] Specific operation: The camera operates to take pictures of the shelves inside the refrigerator in sequence and generate image data.

[0795] Output: Image data of scanned ingredients.

[0796] Step 2:

[0797] The device encrypts the scanned image data and sends it to a server via the Internet.

[0798] Input: The image data generated in step 1.

[0799] Data processing: Image data is encrypted and transmitted through the Wi-Fi module.

[0800] Specific operation: The device uses Wi-Fi to send encrypted image data to the server.

[0801] Output: Encrypted image data sent to the server.

[0802] Step 3:

[0803] The server analyzes the received image data.

[0804] Input: Encrypted image data.

[0805] Data processing: Using an image recognition algorithm (e.g., TensorFlow) on the server, the type, quantity, and location of ingredients are identified, and this information is saved and updated in the database.

[0806] Specific operation: The server runs an image recognition algorithm and stores the identified ingredient data in a database.

[0807] Output: Parsed ingredient data, updated database.

[0808] Step 4:

[0809] An emotion engine analyzes the user's emotional state.

[0810] Input: User facial expressions, voice, and text input.

[0811] Data Processing: The emotion engine analyzes this data and identifies the user's emotional state (e.g., stress, joy, sadness) using tools (e.g., OpenFace, Google Cloud Speech-to-Text API).

[0812] Specific operation: The camera captures the user's facial expressions, the microphone records audio, and the emotion engine analyzes it.

[0813] Output: Parsed emotional state data.

[0814] Step 5:

[0815] The server suggests appropriate recipes based on emotional state data and scanned ingredient data.

[0816] Input: Emotional state data, food ingredient data.

[0817] Data processing: The server searches the database for recipes that match the emotion and suggests them to the user.

[0818] Specific operation: The server performs a database search and selects recipes that match the emotion.

[0819] Output: The proposed recipe.

[0820] Step 6:

[0821] The server identifies the missing ingredients based on the scanned ingredient data and notifies the user.

[0822] Input: Ingredient data, shortage conditions.

[0823] Data processing: The server checks the ingredient list and identifies if any frequently used or necessary ingredients are missing.

[0824] Specific operation: The server analyzes the ingredient list and extracts information about missing ingredients.

[0825] Output: Identification of missing ingredients and recommended purchase notifications.

[0826] Step 7:

[0827] The server sends the suggested recipe and missing ingredient notification to the user's application.

[0828] Input: Suggested recipes, notifications of missing ingredients.

[0829] Data processing: Generation and transmission of notification data.

[0830] Specific operation: The server generates a notification and sends it to the user's smartphone.

[0831] Output: Notification to the user's smartphone.

[0832] Step 8:

[0833] The user checks the notification through the application and makes a plan to create the suggested recipe or purchase the missing ingredients.

[0834] Enter: notifications on your smartphone.

[0835] Data processing: The user decides what to do based on the notification.

[0836] Specific action: The user opens the application on their smartphone and checks the notification.

[0837] Output: Recipe creation or ingredient shopping planning.

[0838] (Application example 2)

[0839] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0840] Currently, refrigerator food management systems are limited to managing ingredients and suggesting recipes. However, they are unable to respond to the user's emotional state and are unable to provide more personalized services based on the user's emotions and eating habits. Therefore, a new system is needed that can provide comprehensive support for the user's eating habits by suggesting cooking methods based on emotions and suggesting the purchase of missing ingredients.

[0841] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for scanning ingredients in the refrigerator, means for comparing the scanned ingredient information with a database, means for suggesting to the user recipes that can be made with ingredients on hand based on the compared information, means for suggesting to the user ingredients that are missing, emotion analysis means for recognizing the user's emotional state, means for suggesting recipes appropriate for the user's emotional state based on the emotion data recognized by the emotion analysis means, and means for suggesting the purchase of ingredients that are missing. This makes it possible to suggest dietary habits that take the user's emotional state into consideration, thereby improving user satisfaction and achieving efficient ingredient management.

[0842] "Means for scanning food items in a refrigerator" refers to technology that uses image capture devices such as cameras and sensors and image recognition algorithms to detect the type, quantity, and location of food items in a refrigerator.

[0843] The "means for comparing the scanned ingredient information with a database" is a technology that compares the acquired ingredient information in the refrigerator with a pre-registered ingredient database to identify matching ingredients and new ingredients that have been added.

[0844] "Means for suggesting recipes to users that can be made using ingredients on hand" refers to technology that provides users with recipes that can be made using ingredients in the refrigerator, based on recipe information stored in a database.

[0845] The "means of suggesting missing ingredients to the user" is a technology that compares the list of ingredients in the refrigerator with recipe information and notifies the user if any ingredients needed to make a specific dish are missing.

[0846] "Emotion analysis means for recognizing the user's emotional state" is a technology that analyzes data such as the user's facial expressions, voice, and text input to identify the user's emotional state.

[0847] "Means for suggesting recipes appropriate to the emotional state based on the emotional data recognized by the emotion analysis means" refers to a technology that suggests the best dishes and drinks for a user based on the user's emotional state (e.g., stress, joy).

[0848] The "means for suggesting the purchase of ingredients that are in short supply" is a technology that notifies the user to purchase ingredients that are in short supply and supports the purchase based on the ingredient data in the refrigerator and the user's emotional state.

[0849] The following describes in detail an embodiment of the present invention. The system comprises a terminal installed in a refrigerator, a server connected via the Internet, and an application or web interface used by a user.

[0850] Food scanning method

[0851] The device is equipped with an image capture device such as a camera or sensor to scan the food items in the refrigerator. This periodically captures images of the food items in the refrigerator. The image data is analyzed using an image recognition algorithm to identify the type and quantity of food items and their expiration dates. This process uses the open source libraries OpenCV and Keras / TensorFlow.

[0852] Database matching method

[0853] The scanned food information is sent to a server via the Internet. The server compares the received data with a database to manage which food ingredients are stored in the refrigerator and in what quantities. This comparison means updates and registers new ingredients to the database.

[0854] Recipe suggestion method

[0855] The server is equipped with a means for suggesting possible recipes based on the user's available ingredients. The server analyzes the recipe information in the database and notifies the user of recipes that can be made using the ingredients in the refrigerator. The server also takes into account the user's past preference data.

[0856] Means for suggesting insufficient ingredients

[0857] The server analyzes the information to identify ingredients that are in short supply and notifies the user, allowing them to know what ingredients they need to buy. Furthermore, the server also provides a function to link with delivery services and purchase the suggested ingredients online.

[0858] Emotion analysis means

[0859] The system also includes an emotion analysis mechanism to recognize the user's emotional state by analyzing the user's facial expressions, voice, and text input data to identify emotions, using a pre-trained emotion recognition model (Keras / TensorFlow).

[0860] Emotion-based recipe suggestion method

[0861] Based on the emotional data recognized by the emotion analysis means, the server suggests cooking recipes suited to the user's emotional state, for example, if the user is under stress, it suggests dishes and drinks that have a relaxing effect.

[0862] Proposal method for purchasing ingredients that are in short supply

[0863] The server notifies the user to purchase ingredients that are in short supply based on the food data in the refrigerator and the user's emotional state, and also provides an interface for purchasing ingredients online.

[0864] Specific examples

[0865] For example, if a user comes home tired from work, the system scans the refrigerator to obtain food ingredient data, and if it detects a "stressed state" from the user's facial expression, it will suggest recipes such as relaxing herbal tea or spinach smoothie. If any ingredients are missing, the system will notify the user so that they can purchase them directly online.

[0866] Prompt Sentence Examples

[0867] Scan images of your refrigerator to suggest recipes based on current ingredients and your emotions, and let you know if you're missing any ingredients.

[0868] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0869] Step 1:

[0870] The device scans the ingredients in the refrigerator.

[0871] Input: Image data of the inside of the refrigerator.

[0872] Output: Image data of the acquired ingredients.

[0873] Specific operation: The camera installed in the refrigerator will automatically start up at a set time (for example, 10:00 every morning) and take pictures of each shelf inside the refrigerator.

[0874] Step 2:

[0875] The terminal transmits the acquired image data to the server.

[0876] Input: Image data of the acquired ingredients.

[0877] Output: Image data sent to the server.

[0878] How it works: After capturing an image, it automatically sends the image data to a server over the Internet. Encryption is used to ensure security.

[0879] Step 3:

[0880] The server analyzes the image data and extracts ingredient information.

[0881] Input: The submitted image data.

[0882] Output: Parsed ingredient information (e.g. type, quantity, expiration date).

[0883] Specific operation: The server runs an image recognition algorithm using Keras / TensorFlow to identify the type and quantity of ingredients from the image, and also determine the expiration date of each ingredient.

[0884] Step 4:

[0885] The server compares the extracted ingredient information with the database.

[0886] Input: Parsed ingredient information.

[0887] Output: Updated ingredient information in the database.

[0888] Specific operation: The server updates the ingredient list by updating the database with new ingredients and information on ingredients that have been consumed.

[0889] Step 5:

[0890] A user uses the sentiment analysis means through an application.

[0891] Input: User's facial expression, voice, and text data.

[0892] Output: The perceived emotional state of the user.

[0893] How it works: The user launches the app and inputs facial and voice data via the camera and microphone into the emotion recognition model, which analyzes this data to identify the user's current emotional state (e.g., stress, joy).

[0894] Step 6:

[0895] The server suggests cooking methods that suit the emotional state based on emotion analysis data.

[0896] Input: User's emotional state, food ingredient information from the database.

[0897] Output: A list of suggested dishes appropriate for the emotional state.

[0898] Specific operation: Based on the emotion recognition results, the server selects the recipe that best suits the user's current emotion (e.g., "spinach smoothie" if stressed) and generates a list of suggestions.

[0899] Step 7:

[0900] The server identifies the ingredients that are in short supply and notifies the user to purchase them.

[0901] Input: Updated database of ingredients, suggested recipes.

[0902] Output: A list of missing ingredients with suggested purchases.

[0903] What happens: The server checks if the ingredients needed for the proposed recipe are in its database, and if any ingredients are missing (e.g., lettuce), it lists them and generates a notification.

[0904] Step 8:

[0905] The user checks the suggested recipe and missing ingredients and uses the delivery service.

[0906] Input: Suggested recipe, list of missing ingredients suggested for purchase.

[0907] Output: Decision to purchase or use delivery service.

[0908] What it does: The user uses the app to review suggested recipes and a list of missing ingredients, and then purchases ingredients online or orders food delivery as needed.

[0909] Through these steps, the system can provide personalized dietary support tailored to the user's emotional state.

[0910] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0911] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0912] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0913] [Third embodiment]

[0914] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0915] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0916] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0917] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0918] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0919] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0920] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0921] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0922] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0923] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0924] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0925] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0926] The system of the present invention aims to efficiently manage food ingredients in a refrigerator and support the user's eating habits. A specific implementation method of this system will be described below.

[0927] Overall system overview

[0928] This system consists of a terminal installed inside the refrigerator, a server connected via the Internet, and an application or web interface used by the user. The terminal is equipped with a camera or sensor that scans ingredients in the refrigerator. The scanned data is sent to the server, where it is analyzed and managed. Based on the analysis results, the server provides the user with information about ingredients, suggests recipes, and notifies them of any missing ingredients.

[0929] Specific implementation methods

[0930] 1. How to scan ingredients

[0931] The device periodically scans each shelf and drawer in the refrigerator. For example, the device's camera automatically activates at 10:00 every morning and captures images of the refrigerator interior. The captured image data is then analyzed using a simple image recognition algorithm to identify the type and location of ingredients. This identification data is then sent to a server via the Internet.

[0932] 2. Data analysis on the server

[0933] The server then performs detailed image recognition processing based on the received scan data. Specifically, advanced image recognition algorithms on the server determine the type, quantity, and expiration date of ingredients. For example, it can identify milk, eggs, tomatoes, etc. from the captured image and store this data in a database. At the same time, information on newly added and consumed ingredients is also updated.

[0934] 3. Recipe suggestions

[0935] The server searches a database for recipes that the user can make based on the latest list of ingredients in the refrigerator. It then suggests multiple optimal recipes based on various criteria, such as the user's past preferences, reviews, and cooking time. These recipes are then sent to the user's application or web interface. For example, if milk and eggs are available, recipes such as "French toast" and "omelette" will be suggested.

[0936] 4. Notification of missing ingredients

[0937] The server identifies frequently used ingredients that are in short supply based on the refrigerator's food list and the user's preference data. The server then checks the database for any missing ingredients and notifies the user, for example, "There's only a little lettuce left, so we recommend you buy some." The user can then create a shopping list based on this information.

[0938] Specific examples

[0939] For example, if a user puts eggs, milk, and tomatoes in the refrigerator, the system operates as follows:

[0940] 1. Every morning at 10:00, the device scans the refrigerator and captures images of eggs, milk, and tomatoes.

[0941] 2. The server analyzes the received image in detail and identifies it as 6 eggs, 500ml of milk, and 3 tomatoes.

[0942] 3. The server registers this information in a database and updates the list of ingredients in the refrigerator.

[0943] 4. The server searches for recipes that the user can make based on the ingredient list and suggests dishes such as "omelette" or "pasta with tomato sauce."

[0944] 5. The server analyzes the user's past data and notifies them, "Since you tend to make salads frequently, we recommend that you buy more lettuce."

[0945] 6. The user reviews these suggestions through the app, selects a recipe, starts cooking, and purchases any additional ingredients needed.

[0946] In this way, the system efficiently supports the user's diet and allows for easy food management in the refrigerator.

[0947] The processing flow will be explained below.

[0948] Step 1: The device starts scanning the inside of the refrigerator.

[0949] The device activates the camera inside the refrigerator and scans each shelf and drawer in turn, capturing an image of each scanned section.

[0950] Step 2: The device performs initial image recognition processing.

[0951] The device performs simple local processing on the captured image to identify the type and location of ingredients, and generates identification data (such as the type, location, and quantity of ingredients).

[0952] Step 3: The terminal sends the identification data to the server.

[0953] The terminal sends the captured image data along with the initial recognition results to the server.

[0954] Step 4: The server receives the data.

[0955] The server receives the data sent from the device and prepares for detailed image analysis.

[0956] Step 5: The server performs detailed image analysis.

[0957] The server uses advanced image recognition algorithms to identify detailed information about each ingredient (type, quantity, expiration date).

[0958] Step 6: The server checks against the database.

[0959] The server compares the identified ingredient information with a database to confirm detailed information such as the name, quantity, and expiration date.

[0960] Step 7: The server creates the ingredient list.

[0961] Based on the detailed information, the server creates an up-to-date list of ingredients in the refrigerator and stores it in a database.

[0962] Step 8: The server looks up the recipe.

[0963] Based on the updated ingredient list, the server searches its database for recipes that can be made with the ingredients on hand.

[0964] Step 9: The server evaluates and selects the recipes.

[0965] The server extracts multiple recipe candidates and evaluates and selects the optimal recipe, taking into account conditions such as cooking time and user preferences.

[0966] Step 10: The server notifies the user of the recipe.

[0967] The server notifies the user's application or web interface of the selected recipe list.

[0968] Step 11: The server identifies the missing ingredients.

[0969] The server identifies the ingredients that are missing based on the list of ingredients in the refrigerator and the user's preference data.

[0970] Step 12: The server notifies the user of the missing ingredients.

[0971] The server creates a list of identified ingredients that are in short supply and sends a notification to the user recommending their purchase.

[0972] Step 13: The user receives the notification.

[0973] The user can review the received recipe suggestions and missing ingredient notifications through the application or web interface.

[0974] Step 14: The user selects a recipe and begins cooking.

[0975] The user selects a recipe from the ones notified and begins cooking using ingredients in the refrigerator.

[0976] Step 15: The user buys more ingredients.

[0977] The user creates a shopping list based on the ingredients they are notified of and purchases the missing ingredients in stores or online.

[0978] Example 1

[0979] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0980] Conventional refrigerator food management systems have difficulty accurately grasping the types and quantities of ingredients, making it difficult to properly suggest recipes that can be made with ingredients on hand or ingredients that are missing. Furthermore, they are not sufficient in suggesting recipes based on the user's preferences or in informing the user of missing ingredients. This has resulted in insufficient efficiency in the user's diet and in managing the food in the refrigerator.

[0981] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0982] In this invention, the server includes means for periodically scanning ingredients in the refrigerator, means for transmitting the scanned ingredient information to the server, means for performing detailed analysis based on the scan data to identify the type, amount, and expiration date of the ingredients and update the database, means for suggesting recipes that can be made with the current ingredients to the user based on the updated database information, and means for notifying the user of missing ingredients based on the updated database information. This allows the server to accurately grasp ingredient information in the refrigerator and suggest appropriate recipes to the user and notify them of missing ingredients.

[0983] "Means for periodically scanning ingredients" refers to a device or process that uses a camera or sensor installed inside the refrigerator to capture images and data of ingredients at regular intervals.

[0984] The "means for transmitting scanned ingredient information to a server" is a function or process for transmitting data on ingredients scanned in the refrigerator to a remote server via the Internet or a network.

[0985] "A means of performing detailed analysis based on the scanned data, recognizing the type, quantity, and expiration date of ingredients, and updating the database" refers to a function or process that analyzes the received image data of ingredients using an advanced image recognition algorithm, recognizes detailed information about the ingredients, and reflects this in the database.

[0986] "Means for suggesting recipes to the user that can be made with the current ingredients" is a function or process that searches a database for recipes that can be made using the ingredients in the refrigerator and displays or notifies the user.

[0987] The "means for notifying the user of ingredients that are running low" is a function or process that identifies ingredients that are running low or low in the refrigerator based on database information and notifies the user.

[0988] The "camera and initial image recognition algorithm" refers to a camera device for photographing ingredients in the refrigerator and a program for analyzing the photographed images at an early stage and identifying the general type and location of the ingredients.

[0989] "Means for referencing past preference data to suggest more suitable recipes and missing ingredients" refers to a function or process that analyzes the user's past cooking and ingredient usage history and suggests individually customized recipes and missing ingredients based on that data.

[0990] MODE FOR CARRYING OUT THE INVENTION

[0991] The system of the present invention aims to efficiently manage food ingredients in a refrigerator and support the user's dietary habits. This system consists of a terminal installed in the refrigerator, a server connected via the Internet, and an application or web interface used by the user.

[0992] The device is equipped with a camera and sensors that periodically scan the food items in the refrigerator. Specifically, the camera automatically starts up at 10:00 every morning and takes pictures of each shelf and drawer in the refrigerator in sequence. This collects image data of the food items. The collected data is then simply analyzed using a basic image recognition algorithm to identify the general type and location of the food items. This identification data is then sent to a server via the Internet.

[0993] The server performs detailed image analysis based on the scanned data it receives. Specifically, an advanced image recognition algorithm on the server identifies the type, quantity, and expiration date of ingredients, and stores each piece of data in a database. As a result of the analysis, information on newly added or consumed ingredients in the refrigerator is updated.

[0994] The server then searches for recipes that the user can create based on the updated ingredient list. The server selects the best recipe based on various criteria, such as the user's past preferences, reviews, cooking time, etc. The result is reported to the user's application or web interface.

[0995] Furthermore, the server identifies ingredients that are running low based on the information in the database. This allows users to effectively manage the ingredients in their refrigerator without forgetting to buy the ingredients they need. For example, if a user frequently makes salads, the server has a function that notifies them by saying, "You're running low on lettuce, so we recommend you buy some."

[0996] Specific examples

[0997] For example, if a user puts eggs, milk, and tomatoes in the refrigerator, the system operates as follows:

[0998] 1. Every morning at 10:00, the device scans the refrigerator and captures images of eggs, milk, and tomatoes.

[0999] 2. The server analyzes the received image in detail and identifies it as 6 eggs, 500ml of milk, and 3 tomatoes.

[1000] 3. The server registers this information in a database and updates the list of ingredients in the refrigerator.

[1001] 4. Based on the ingredient list, the server suggests recipes that the user can make, such as "omelette" or "pasta with tomato sauce."

[1002] 5. The server analyzes the user's past data and notifies them, "Since you tend to make salads frequently, we recommend that you buy more lettuce."

[1003] 6. The user reviews these suggestions through the app, selects a recipe, starts cooking, and purchases any additional ingredients needed.

[1004] This system allows users to efficiently manage the ingredients in their refrigerators and support their dietary habits. It is also expected that by knowing in advance which ingredients are running low, users will be able to reduce wasteful shopping and make more effective use of ingredients.

[1005] Prompt Sentence Examples

[1006] A camera and a sensor are installed in the refrigerator. They automatically wake up every morning at 10:00 and scan the ingredients in the refrigerator. The scanned data is sent to a server, which uses advanced image recognition algorithms to identify the ingredients and update its database. The server then uses this information to suggest recipes to the user and notify them of any missing ingredients. Please explain how such a system works.

[1007] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1008] Step 1:

[1009] Start scanning food in the refrigerator

[1010] The device automatically activates the camera at 10:00 every morning and takes pictures of each shelf and drawer in the refrigerator in sequence.

[1011] Input: Video data from a camera installed in a refrigerator.

[1012] Data processing: Image capture of each shelf and drawer.

[1013] Output: Captured image data of the inside of the refrigerator.

[1014] Step 2:

[1015] Initial image recognition processing and data transmission

[1016] The device analyzes the captured image data using a rudimentary image recognition algorithm to identify the general type and location of the food item.

[1017] Input: The captured image data.

[1018] Data calculation: Identifying the type and location of ingredients using an early image recognition algorithm.

[1019] Output: Simple ingredient data identified.

[1020] The terminal transmits the identification data to a server over the Internet.

[1021] Step 3:

[1022] Detailed image analysis

[1023] The server uses advanced image recognition algorithms to perform detailed analysis of the received scan data and identify the type, quantity, and expiration date of the ingredients.

[1024] Input: Simple ingredient data and captured image data sent from the device.

[1025] Data calculation: Analysis using detailed image recognition algorithms.

[1026] Output: Detailed information about the recognized ingredients (type, quantity, expiration date).

[1027] Step 4:

[1028] Database Update

[1029] The server stores the analyzed detailed data of ingredients in a database and updates the list of ingredients in the refrigerator.

[1030] Input: Detailed data of recognized ingredients.

[1031] Data calculation: Saving and updating data in a database.

[1032] Output: Updated ingredient list.

[1033] Step 5:

[1034] Recipe Suggestions

[1035] The server searches the database for recipes that the user can make based on the latest ingredient list.

[1036] Input: Latest ingredient list, user's past preference data, reviews, cooking time.

[1037] Data Computation: Recipe search and selection algorithms.

[1038] Output: A list of recipes suggested to the user.

[1039] The server posts these recipes to the user's application or web interface.

[1040] Step 6:

[1041] Notification of shortage of ingredients

[1042] The server refers to the list of ingredients in the refrigerator and the user's preference data to identify ingredients that may be in short supply.

[1043] Input: Latest ingredient list, user preference data.

[1044] Data calculation: Algorithm for identifying missing ingredients.

[1045] Output: Information about missing ingredients notified to the user.

[1046] The server notifies the user's application of the missing ingredients.

[1047] In this way, the system streamlines food ingredient management and supports the user's eating habits.

[1048] (Application example 1)

[1049] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1050] In recent years, systems aimed at managing food ingredients in refrigerators and improving dietary habits have been attracting attention. However, while these systems can manage food ingredients, they lack the ability to quickly replenish ingredients when they run out. This requires users to manually replenish ingredients, making it difficult to achieve an efficient diet. Furthermore, if food identification and expiration date management are not adequate, food waste can occur. This increases stress and burden on users' daily lives and leads to food waste.

[1051] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1052] In this invention, the server includes a means for scanning ingredients in the refrigerator, a means for comparing the scanned ingredient information with a database, a means for suggesting recipes to the user that can be made with ingredients on hand based on the compared information, a means for suggesting ingredients that are in short supply based on the compared information, and a means for automatically ordering the ingredients that are in short supply. This allows for efficient ingredient management and rapid replenishment. Furthermore, reducing food waste can enrich the user's diet.

[1053] "Means for scanning ingredients in the refrigerator" refers to devices or sensors installed inside the refrigerator to obtain information about ingredients.

[1054] "Scanned food ingredient information" is information including data such as the type, quantity, and expiration date of the food ingredients in the refrigerator.

[1055] "Means for database matching" refers to algorithms or systems that compare scanned ingredient information with existing databases to verify and update their contents.

[1056] "Means of suggesting recipes to users that can be made using ingredients on hand" refers to software or an interface that searches for recipes that can be made using ingredients currently in the refrigerator and presents them to the user.

[1057] "Means for suggesting ingredients that are in short supply to the user" refers to a function or system that identifies ingredients that are in short supply based on information about ingredients on hand and notifies the user of the list.

[1058] "Means for automatic ordering" refers to a system that automatically orders ingredients that are in short supply from external food delivery services, etc., to replenish them.

[1059] "Image capture device" refers to a device such as a camera that takes pictures of food in the refrigerator.

[1060] "Image recognition algorithm" refers to a computer program or technology used to analyze the type and quantity of ingredients from an image captured by an image capture device.

[1061] "Expiration date management" refers to software or systems that track the expiration dates of food ingredients and notify users when the expiration date is approaching.

[1062] The present invention provides a system for efficiently managing food ingredients in a refrigerator and supporting a user's dietary habits. A specific implementation method of this system is described below.

[1063] Overall system overview

[1064] This system consists of a terminal inside the refrigerator, a server connected via the Internet, and an application or web interface used by the user. The terminal is equipped with a camera or sensor that scans ingredients in the refrigerator. The scanned data is sent to the server, where it is analyzed and managed. Based on the analysis results, the server provides the user with ingredient information, suggests recipes, and notifies them of any ingredients they are lacking, and automatically orders the missing ingredients.

[1065] Specific implementation methods

[1066] 1. How to scan ingredients

[1067] The device periodically scans each shelf and drawer in the refrigerator. For example, the device's camera automatically activates at 10:00 every morning and captures images of the refrigerator's interior. The captured image data is then analyzed using a simple image recognition algorithm to identify the type and location of ingredients. This identification data is then sent to a server via the Internet.

[1068] 2. Data analysis on the server

[1069] The server then performs detailed image recognition processing based on the received scan data. Specifically, advanced image recognition algorithms on the server determine the type, quantity, and expiration date of ingredients. For example, it can identify milk, eggs, tomatoes, etc. from the captured image and store this data in a database. At the same time, information on newly added and consumed ingredients is also updated.

[1070] 3. Recipe suggestions

[1071] The server searches a database for recipes that the user can make based on the latest list of ingredients in the refrigerator. It then suggests multiple optimal recipes based on various criteria, such as the user's past preferences, reviews, and cooking time. These recipes are then sent to the user's application or web interface. For example, if milk and eggs are available, recipes such as "French toast" and "omelette" will be suggested.

[1072] 4. Notification of shortage of ingredients and automatic ordering

[1073] The server identifies frequently used ingredients that are in short supply based on the list of ingredients in the refrigerator and the user's preference data. The missing ingredients are checked again against the database, and the user is notified, for example, "There is little lettuce left, so we recommend you buy some." In addition, a function is provided to automatically order these ingredients from a food delivery service to replenish them. This saves users the trouble of shopping and enables efficient food management.

[1074] Specific examples

[1075] For example, if a user puts eggs, milk, and tomatoes in the refrigerator, the system operates as follows:

[1076] 1. Every morning at 10:00, the device scans the refrigerator and captures images of eggs, milk, and tomatoes.

[1077] 2. The server analyzes the received image in detail and identifies it as 6 eggs, 500ml of milk, and 3 tomatoes.

[1078] 3. The server registers this information in a database and updates the list of ingredients in the refrigerator.

[1079] 4. The server searches for recipes that the user can make based on the ingredient list and suggests dishes such as "omelette" or "pasta with tomato sauce."

[1080] 5. The server analyzes the user's past data and notifies them that "Since you tend to make salads frequently, we recommend that you buy more lettuce."

[1081] 6. The server automatically orders the missing ingredients from a food delivery service to replenish them.

[1082] Prompt Sentence Examples

[1083] Example prompt:

[1084] Based on the current list of ingredients in your fridge, suggest recipes you can make and order any missing ingredients from a food delivery service. For example, if your current list of ingredients includes 500ml of eggs, 3 bottles of milk, and 2 tomatoes, suggest recipes you can make and what ingredients you're missing and automatically order them with the appropriate food delivery service.

[1085] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1086] Step 1:

[1087] The device scans the inside of the refrigerator

[1088] Input: Every morning at 10am, the camera inside the refrigerator will automatically turn on.

[1089] Data processing / data calculation: A camera captures images of the ingredients in the refrigerator.

[1090] Output: Image data is generated.

[1091] How it works: The device's built-in camera captures multiple still images or videos, which are then pre-processed using a basic image processing algorithm.

[1092] Step 2:

[1093] The device sends the scanned data to the server.

[1094] Input: Image data acquired in step 1.

[1095] Data processing / data calculation: Image data is compressed and sent to a cloud server via the Internet.

[1096] Output: Image data received by the server.

[1097] Specific operation: The device uses Wi-Fi or other Internet connection to upload image data to a designated storage area on the cloud server.

[1098] Step 3:

[1099] The server analyzes the data using image recognition algorithms

[1100] Input: Image data sent to the server.

[1101] Data processing / data calculation: Using advanced image recognition algorithms (e.g., TensorFlow or OpenCV), the type, quantity, and expiration date of ingredients are identified.

[1102] Output: Parsed food ingredient information database.

[1103] Specific operation: Image recognition software on the server analyzes the image and records information about the identified ingredients in a database.

[1104] Step 4:

[1105] The server suggests recipes to the user

[1106] Input: Ingredient information parsed in step 3.

[1107] Data processing / data calculation: Based on the ingredient information, a matching recipe is searched for from the recipe information in the database.

[1108] Output: A list of recipes suggested to the user.

[1109] Specific operation: The server selects a suitable recipe based on the user's past preferences, reviews, cooking time, etc., and notifies the user's application or web interface.

[1110] Step 5:

[1111] The server notifies the user of any ingredients that are missing.

[1112] Input: Ingredient information analyzed in step 3 and user preference data.

[1113] Data processing / data calculation: Compare the ingredient list with user preference data to identify missing ingredients.

[1114] Output: Notification of missing ingredients.

[1115] Specific operation: The server sends a notification to the user about a shortage of ingredients, such as "Lettuce is low, so we recommend you buy some."

[1116] Step 6:

[1117] The server automatically orders ingredients that are in short supply.

[1118] Input: Shortage ingredients identified in step 5.

[1119] Data processing / data calculation: Send order data for missing ingredients to the designated food delivery service.

[1120] Output: Automatic ordering to food delivery service.

[1121] Specific operation: The server uses the food delivery service's API to automatically order missing ingredients and replenish them in the user's refrigerator.

[1122] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1123] The present invention relates to a system for efficiently managing food items in a refrigerator and supporting the emotions and eating habits of a user. A specific implementation method of this system will be described below.

[1124] Overall system overview

[1125] This system consists of a terminal installed in the user's refrigerator, a server connected via the Internet, and an application or web interface used by the user. It also incorporates an emotion engine that recognizes the user's emotions. The terminal is equipped with a camera and sensors to scan ingredients in the refrigerator. The scanned data is sent to the server for analysis and management. The emotion engine analyzes the user's facial expressions, voice, and text input, and based on this, it makes recipe suggestions and notifies users of missing ingredients.

[1126] Specific implementation methods

[1127] 1. How to scan ingredients

[1128] The device periodically scans each shelf and drawer in the refrigerator. For example, the device's camera automatically activates at 10:00 every morning and captures images of the refrigerator interior. The captured image data is then analyzed using a simple image recognition algorithm to identify the type and location of ingredients. This identification data is then sent to a server via the Internet.

[1129] 2. Data analysis on the server

[1130] The server then performs detailed image recognition processing based on the received scan data. Specifically, advanced image recognition algorithms on the server determine the type, quantity, and expiration date of ingredients. For example, it can identify milk, eggs, tomatoes, etc. from the captured image and store this data in a database. At the same time, information on newly added and consumed ingredients is also updated.

[1131] 3. User Emotion Recognition

[1132] The emotion engine analyzes the user's facial expressions, voice, text input, etc. to recognize the user's emotions. For example, when a user launches the app to add ingredients, the camera captures the user's facial expressions and the voice input is analyzed by the emotion engine. This allows the emotion engine to identify the user's current emotional state (joy, anger, sadness, stress, etc.).

[1133] 4. Linking Recipe Suggestions with Emotions

[1134] The server selects recipes that the user is likely to like based on the emotional data obtained from the emotion engine. For example, if the user is feeling stressed, it will suggest recipes such as "herbal tea" or "spinach smoothie," which have a relaxing effect. If the user is happy, it will suggest recipes such as "chocolate cake," which is easy to make and fun to make.

[1135] 5. Notification of missing ingredients

[1136] The server identifies ingredients that are running low based on the list of ingredients in the refrigerator and data from the emotion engine. For example, if a user is running low on an ingredient needed for a dish they frequently cook, the server will send a notification saying, "You're running low on cabbage, so we recommend you buy some."

[1137] Specific examples

[1138] For example, if a user puts eggs, milk, and tomatoes in the refrigerator, the system operates as follows:

[1139] 1. Every morning at 10:00, the device scans the refrigerator and captures images of eggs, milk, and tomatoes.

[1140] 2. The server analyzes the received image in detail and identifies it as 6 eggs, 500ml of milk, and 3 tomatoes.

[1141] 3. The server registers this information in a database and updates the list of ingredients in the refrigerator.

[1142] 4. The emotion engine analyzes the user's facial expressions and voice and identifies their current emotion as "stress."

[1143] 5. The server will suggest a relaxing recipe, such as a "spinach smoothie," based on the ingredient list and emotional data.

[1144] 6. The server analyzes the user's past data and notifies them, "Since you tend to make salads frequently, we recommend that you buy more lettuce."

[1145] 7. The user reviews these suggestions through the application and plans to make a "spinach smoothie" and buy some lettuce.

[1146] In this way, the system efficiently supports the user's diet and makes suggestions tailored to their emotions, helping them achieve a more satisfying life.

[1147] The processing flow will be explained below.

[1148] Step 1:

[1149] The device will begin scanning the inside of the refrigerator.

[1150] The device activates the camera inside the refrigerator and scans each shelf and drawer in turn, capturing an image of each scanned section.

[1151] Step 2:

[1152] The device performs initial image recognition processing.

[1153] The device performs simple local processing on the captured image to identify the type and location of ingredients, and generates identification data (such as the type, location, and quantity of ingredients).

[1154] Step 3:

[1155] The terminal transmits the identification data to the server.

[1156] The terminal sends the captured image data along with the initial recognition results to the server.

[1157] Step 4:

[1158] The server receives the data.

[1159] The server receives the data sent from the device and prepares for detailed image analysis.

[1160] Step 5:

[1161] The server performs detailed image analysis.

[1162] The server uses advanced image recognition algorithms to identify detailed information about each ingredient (type, quantity, expiration date).

[1163] Step 6:

[1164] The server checks it against its database.

[1165] The server compares the identified ingredient information with a database to confirm detailed information such as the name, quantity, and expiration date.

[1166] Step 7:

[1167] The server creates an ingredient list.

[1168] Based on the detailed information, the server creates an up-to-date list of ingredients in the refrigerator and stores it in a database.

[1169] Step 8:

[1170] The emotion engine recognizes the user's emotions.

[1171] The emotion engine captures and analyzes data such as a user's facial expressions, voice, and text input to identify the user's emotional state.

[1172] Step 9:

[1173] The server searches for the recipe.

[1174] The server searches the database for recipes that can be made with the ingredients on hand based on the updated ingredient list and the emotional data obtained from the emotion engine.

[1175] Step 10:

[1176] The server evaluates and selects recipes.

[1177] The server extracts multiple recipe candidates and evaluates and selects the optimal recipe taking into account factors such as the user's emotional state, cooking time, and past preferences.

[1178] Step 11:

[1179] The server notifies the user of the recipe.

[1180] The server notifies the user's application or web interface of the selected recipe list.

[1181] Step 12:

[1182] The server identifies the missing ingredients.

[1183] The server identifies the ingredients that are missing based on the list of ingredients in the refrigerator and the user's preference data.

[1184] Step 13:

[1185] The server notifies the user of the missing ingredients.

[1186] The server creates a list of identified ingredients that are in short supply and sends a notification to the user recommending their purchase.

[1187] Step 14:

[1188] The user receives a notification.

[1189] The user can review the received recipe suggestions and missing ingredient notifications through the application or web interface.

[1190] Step 15:

[1191] The user selects a recipe and begins cooking.

[1192] The user selects a recipe from the ones notified and begins cooking using ingredients in the refrigerator.

[1193] Step 16:

[1194] The user buys more ingredients.

[1195] The user creates a shopping list based on the ingredients they are notified of and purchases the missing ingredients in stores or online.

[1196] With this specific processing flow, the system efficiently manages the user's ingredients and suggests recipes according to their emotional state, supporting a more satisfying diet.

[1197] Example 2

[1198] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1199] In modern life, there is a demand for efficient management of food in the refrigerator and support for dietary habits that are tailored to the user's mood. Conventional systems require manual management of food inventory and recipe suggestions, which poses a challenge in that they are unable to suggest recipes tailored to the user's mood or notify users when ingredients are in short supply. This can cause unnecessary stress for users. Furthermore, users must also manually manage expiration dates, which can lead to food waste.

[1200] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1201] In this invention, the server includes a means for scanning ingredients in the refrigerator, a means for comparing the scanned ingredient information with a database, and a means for analyzing the user's emotional state. This automates the management of ingredients in the refrigerator, making it possible to suggest recipes tailored to the user's emotions and notify the user of any missing ingredients. Furthermore, the server automatically manages expiration dates, reducing food waste and improving the user's quality of life.

[1202] "Means for scanning food ingredients in a refrigerator" is a general term for devices or mechanisms that use cameras or sensors installed in the refrigerator to acquire images and data of food ingredients in the refrigerator periodically or as needed.

[1203] "Means for comparing scanned ingredient information with a database" is a general term for systems and algorithms that analyze the images and data of ingredients obtained by scanning and compare that information with information in an existing database.

[1204] "Means for suggesting recipes to users that can be made using ingredients on hand" is a general term for algorithms and procedures for presenting users with recipes that can be made using ingredients currently in the refrigerator, based on ingredient information in the database.

[1205] "Means for suggesting ingredients that are in short supply to users" is a general term for systems and algorithms that analyze ingredient inventory based on ingredient information stored in a database and notify users of ingredients that are in short supply.

[1206] "Means for analyzing a user's emotional state" is a general term for systems or algorithms that analyze a user's facial expressions, voice, or text input to identify the user's current emotional state.

[1207] "Means for suggesting appropriate recipes to users based on their emotional state" is a general term for algorithms and systems that take into account the user's emotional state and suggest to the user the recipe that is best suited to that situation.

[1208] "Means for managing expiration dates based on food ingredient information" is a general term for systems and algorithms that automatically manage the expiration dates of each ingredient based on scanned food ingredient information and notify users.

[1209] "Means for notifying users of shortages of ingredients" is a general term for systems and algorithms that notify users when they are running low on ingredients that they frequently use or need, based on the inventory status of ingredients.

[1210] This invention relates to a system that efficiently manages food ingredients in a refrigerator and supports a dietary lifestyle that matches the user's emotions. Specifically, it is composed of a terminal installed in the refrigerator, a server connected via the Internet, and an application or web interface used by the user. It also incorporates an emotion engine that recognizes the user's emotions.

[1211] Program processing

[1212] Food scanning

[1213] The device periodically scans each shelf and drawer in the refrigerator. Specifically, the device's built-in camera (e.g., a high-resolution camera) automatically starts up at 10:00 every morning and captures images of the inside of the refrigerator. This image data is temporarily stored in the device's internal memory. The encrypted image data is then sent to a server via the Internet via a Wi-Fi module (e.g., ESP8266).

[1214] Data analysis

[1215] The server analyzes the received image data. Specifically, it uses an image recognition algorithm (e.g., TensorFlow) on the server to identify the type, quantity, and location of ingredients. The results of this analysis are registered in a database, and the list of ingredients in the refrigerator is automatically updated. At the same time, it also determines expiration dates and identifies any missing ingredients.

[1216] emotion recognition

[1217] The emotion engine analyzes the user's facial expressions, voice, and text input. For example, when a user launches an app, the device's camera and microphone capture the user's facial expressions and voice. For analysis, OpenFace and Google Cloud Speech-to-Text API are used, for example. Based on this, the emotion engine identifies the user's emotional state, and this information is sent to the server.

[1218] Recipe suggestions and missing ingredient notifications

[1219] The server suggests appropriate recipes to the user based on the data obtained from the emotion engine and the scanned ingredient information. For example, if the user is feeling stressed, it will suggest a relaxing recipe such as a "spinach smoothie." If it is determined that an ingredient is in short supply, it will generate a notification such as "We are running low on cabbage, so we recommend you buy some," and send it to the user's application.

[1220] Specific examples

[1221] For example, if a user puts eggs, milk, and tomatoes in the refrigerator, the system will do the following:

[1222] 1. Every morning at 10:00, the device scans the refrigerator and captures images of eggs, milk, and tomatoes.

[1223] 2. The server analyzes the received image in detail and identifies it as 6 eggs, 500ml of milk, and 3 tomatoes.

[1224] 3. The server registers this information in a database and updates the list of ingredients in the refrigerator.

[1225] 4. The emotion engine analyzes the user's facial expressions and voice and identifies their current emotion as "stress."

[1226] 5. Based on the ingredient list and emotional data, the server suggests a relaxing recipe: "Spinach Smoothie."

[1227] 6. The server analyzes the user's past data and notifies them, "Since you tend to make salads frequently, we recommend that you buy more lettuce."

[1228] 7. The user reviews these suggestions through the application and plans to make a "spinach smoothie" and buy some lettuce.

[1229] In this way, the system efficiently supports the user's diet and makes suggestions tailored to their emotions, helping them achieve a more satisfying life.

[1230] Prompt Sentence Examples

[1231] "I have eggs, milk, and tomatoes in my fridge, but I've been feeling stressed lately. Can you suggest a relaxing recipe using these ingredients?"

[1232] "Also, check out the other ingredients you need for salads you make frequently."

[1233] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1234] Step 1:

[1235] The device scans each shelf and drawer in the refrigerator.

[1236] Input: The device's camera captures images of the inside of the refrigerator according to the user's schedule (e.g., every morning at 10:00).

[1237] Data processing: The captured images are stored in the device's internal memory.

[1238] Specific operation: The camera operates to take pictures of the shelves inside the refrigerator in sequence and generate image data.

[1239] Output: Image data of scanned ingredients.

[1240] Step 2:

[1241] The device encrypts the scanned image data and sends it to a server via the Internet.

[1242] Input: The image data generated in step 1.

[1243] Data processing: Image data is encrypted and transmitted through the Wi-Fi module.

[1244] Specific operation: The device uses Wi-Fi to send encrypted image data to the server.

[1245] Output: Encrypted image data sent to the server.

[1246] Step 3:

[1247] The server analyzes the received image data.

[1248] Input: Encrypted image data.

[1249] Data processing: Using an image recognition algorithm (e.g., TensorFlow) on the server, the type, quantity, and location of ingredients are identified, and this information is saved and updated in the database.

[1250] Specific operation: The server runs an image recognition algorithm and stores the identified ingredient data in a database.

[1251] Output: Parsed ingredient data, updated database.

[1252] Step 4:

[1253] An emotion engine analyzes the user's emotional state.

[1254] Input: User facial expressions, voice, and text input.

[1255] Data Processing: The emotion engine analyzes this data and identifies the user's emotional state (e.g., stress, joy, sadness) using tools (e.g., OpenFace, Google Cloud Speech-to-Text API).

[1256] Specific operation: The camera captures the user's facial expressions, the microphone records audio, and the emotion engine analyzes it.

[1257] Output: Parsed emotional state data.

[1258] Step 5:

[1259] The server suggests appropriate recipes based on emotional state data and scanned ingredient data.

[1260] Input: Emotional state data, food ingredient data.

[1261] Data processing: The server searches the database for recipes that match the emotion and suggests them to the user.

[1262] Specific operation: The server performs a database search and selects recipes that match the emotion.

[1263] Output: The proposed recipe.

[1264] Step 6:

[1265] The server identifies the missing ingredients based on the scanned ingredient data and notifies the user.

[1266] Input: Ingredient data, shortage conditions.

[1267] Data processing: The server checks the ingredient list and identifies if any frequently used or necessary ingredients are missing.

[1268] Specific operation: The server analyzes the ingredient list and extracts information about missing ingredients.

[1269] Output: Identification of missing ingredients and recommended purchase notifications.

[1270] Step 7:

[1271] The server sends the suggested recipe and missing ingredient notification to the user's application.

[1272] Input: Suggested recipes, notifications of missing ingredients.

[1273] Data processing: Generation and transmission of notification data.

[1274] Specific operation: The server generates a notification and sends it to the user's smartphone.

[1275] Output: Notification to the user's smartphone.

[1276] Step 8:

[1277] The user checks the notification through the application and makes a plan to create the suggested recipe or purchase the missing ingredients.

[1278] Enter: notifications on your smartphone.

[1279] Data processing: The user decides what to do based on the notification.

[1280] Specific action: The user opens the application on their smartphone and checks the notification.

[1281] Output: Recipe creation or ingredient shopping planning.

[1282] (Application example 2)

[1283] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1284] Currently, refrigerator food management systems are limited to managing ingredients and suggesting recipes. However, they are unable to respond to the user's emotional state and are unable to provide more personalized services based on the user's emotions and eating habits. Therefore, a new system is needed that can provide comprehensive support for the user's eating habits by suggesting cooking methods based on emotions and suggesting the purchase of missing ingredients.

[1285] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for scanning ingredients in the refrigerator, means for comparing the scanned ingredient information with a database, means for suggesting to the user recipes that can be made with ingredients on hand based on the compared information, means for suggesting to the user ingredients that are missing, emotion analysis means for recognizing the user's emotional state, means for suggesting recipes appropriate for the user's emotional state based on the emotion data recognized by the emotion analysis means, and means for suggesting the purchase of ingredients that are missing. This makes it possible to suggest dietary habits that take the user's emotional state into consideration, thereby improving user satisfaction and achieving efficient ingredient management.

[1286] "Means for scanning food items in a refrigerator" refers to technology that uses image capture devices such as cameras and sensors and image recognition algorithms to detect the type, quantity, and location of food items in a refrigerator.

[1287] The "means for comparing the scanned ingredient information with a database" is a technology that compares the acquired ingredient information in the refrigerator with a pre-registered ingredient database to identify matching ingredients and new ingredients that have been added.

[1288] "Means for suggesting recipes to users that can be made using ingredients on hand" refers to technology that provides users with recipes that can be made using ingredients in the refrigerator, based on recipe information stored in a database.

[1289] The "means of suggesting missing ingredients to the user" is a technology that compares the list of ingredients in the refrigerator with recipe information and notifies the user if any ingredients needed to make a specific dish are missing.

[1290] "Emotion analysis means for recognizing the user's emotional state" is a technology that analyzes data such as the user's facial expressions, voice, and text input to identify the user's emotional state.

[1291] "Means for suggesting recipes appropriate to the emotional state based on the emotional data recognized by the emotion analysis means" refers to a technology that suggests the best dishes and drinks for a user based on the user's emotional state (e.g., stress, joy).

[1292] The "means for suggesting the purchase of ingredients that are in short supply" is a technology that notifies the user to purchase ingredients that are in short supply and supports the purchase based on the ingredient data in the refrigerator and the user's emotional state.

[1293] The following describes in detail an embodiment of the present invention. The system comprises a terminal installed in a refrigerator, a server connected via the Internet, and an application or web interface used by a user.

[1294] Food scanning method

[1295] The device is equipped with an image capture device such as a camera or sensor to scan the food items in the refrigerator. This periodically captures images of the food items in the refrigerator. The image data is analyzed using an image recognition algorithm to identify the type and quantity of food items and their expiration dates. This process uses the open source libraries OpenCV and Keras / TensorFlow.

[1296] Database matching method

[1297] The scanned food information is sent to a server via the Internet. The server compares the received data with a database to manage which food ingredients are stored in the refrigerator and in what quantities. This comparison means updates and registers new ingredients to the database.

[1298] Recipe suggestion method

[1299] The server is equipped with a means for suggesting possible recipes based on the user's available ingredients. The server analyzes the recipe information in the database and notifies the user of recipes that can be made using the ingredients in the refrigerator. The server also takes into account the user's past preference data.

[1300] Means for suggesting insufficient ingredients

[1301] The server analyzes the information to identify ingredients that are in short supply and notifies the user, allowing them to know what ingredients they need to buy. Furthermore, the server also provides a function to link with delivery services and purchase the suggested ingredients online.

[1302] Emotion analysis means

[1303] The system also includes an emotion analysis mechanism to recognize the user's emotional state by analyzing the user's facial expressions, voice, and text input data to identify emotions, using a pre-trained emotion recognition model (Keras / TensorFlow).

[1304] Emotion-based recipe suggestion method

[1305] Based on the emotional data recognized by the emotion analysis means, the server suggests cooking recipes suited to the user's emotional state, for example, if the user is under stress, it suggests dishes and drinks that have a relaxing effect.

[1306] Proposal method for purchasing ingredients that are in short supply

[1307] The server notifies the user to purchase ingredients that are in short supply based on the food data in the refrigerator and the user's emotional state, and also provides an interface for purchasing ingredients online.

[1308] Specific examples

[1309] For example, if a user comes home tired from work, the system scans the refrigerator to obtain food ingredient data, and if it detects a "stressed state" from the user's facial expression, it will suggest recipes such as relaxing herbal tea or spinach smoothie. If any ingredients are missing, the system will notify the user so that they can purchase them directly online.

[1310] Prompt Sentence Examples

[1311] Scan images of your refrigerator to suggest recipes based on current ingredients and your emotions, and let you know if you're missing any ingredients.

[1312] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1313] Step 1:

[1314] The device scans the ingredients in the refrigerator.

[1315] Input: Image data of the inside of the refrigerator.

[1316] Output: Image data of the acquired ingredients.

[1317] Specific operation: The camera installed in the refrigerator will automatically start up at a set time (for example, 10:00 every morning) and take pictures of each shelf inside the refrigerator.

[1318] Step 2:

[1319] The terminal transmits the acquired image data to the server.

[1320] Input: Image data of the acquired ingredients.

[1321] Output: Image data sent to the server.

[1322] How it works: After capturing an image, it automatically sends the image data to a server over the Internet. Encryption is used to ensure security.

[1323] Step 3:

[1324] The server analyzes the image data and extracts ingredient information.

[1325] Input: The submitted image data.

[1326] Output: Parsed ingredient information (e.g. type, quantity, expiration date).

[1327] Specific operation: The server runs an image recognition algorithm using Keras / TensorFlow to identify the type and quantity of ingredients from the image, and also determine the expiration date of each ingredient.

[1328] Step 4:

[1329] The server compares the extracted ingredient information with the database.

[1330] Input: Parsed ingredient information.

[1331] Output: Updated ingredient information in the database.

[1332] Specific operation: The server updates the ingredient list by updating the database with new ingredients and information on ingredients that have been consumed.

[1333] Step 5:

[1334] A user uses the sentiment analysis means through an application.

[1335] Input: User's facial expression, voice, and text data.

[1336] Output: The perceived emotional state of the user.

[1337] How it works: The user launches the app and inputs facial and voice data via the camera and microphone into the emotion recognition model, which analyzes this data to identify the user's current emotional state (e.g., stress, joy).

[1338] Step 6:

[1339] The server suggests cooking methods that suit the emotional state based on emotion analysis data.

[1340] Input: User's emotional state, food ingredient information from the database.

[1341] Output: A list of suggested dishes appropriate for the emotional state.

[1342] Specific operation: Based on the emotion recognition results, the server selects the recipe that best suits the user's current emotion (e.g., "spinach smoothie" if stressed) and generates a list of suggestions.

[1343] Step 7:

[1344] The server identifies the ingredients that are in short supply and notifies the user to purchase them.

[1345] Input: Updated database of ingredients, suggested recipes.

[1346] Output: A list of missing ingredients with suggested purchases.

[1347] What happens: The server checks if the ingredients needed for the proposed recipe are in its database, and if any ingredients are missing (e.g., lettuce), it lists them and generates a notification.

[1348] Step 8:

[1349] The user checks the suggested recipe and missing ingredients and uses the delivery service.

[1350] Input: Suggested recipe, list of missing ingredients suggested for purchase.

[1351] Output: Decision to purchase or use delivery service.

[1352] What it does: The user uses the app to review suggested recipes and a list of missing ingredients, and then purchases ingredients online or orders food delivery as needed.

[1353] Through these steps, the system can provide personalized dietary support tailored to the user's emotional state.

[1354] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1355] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1356] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1357] [Fourth embodiment]

[1358] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1359] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1360] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1361] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1362] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1363] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1364] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1365] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1366] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1367] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1368] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1369] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1370] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1371] The system of the present invention aims to efficiently manage food ingredients in a refrigerator and support the user's eating habits. A specific implementation method of this system will be described below.

[1372] Overall system overview

[1373] This system consists of a terminal installed inside the refrigerator, a server connected via the Internet, and an application or web interface used by the user. The terminal is equipped with a camera or sensor that scans ingredients in the refrigerator. The scanned data is sent to the server, where it is analyzed and managed. Based on the analysis results, the server provides the user with information about ingredients, suggests recipes, and notifies them of any missing ingredients.

[1374] Specific implementation methods

[1375] 1. How to scan ingredients

[1376] The device periodically scans each shelf and drawer in the refrigerator. For example, the device's camera automatically activates at 10:00 every morning and captures images of the refrigerator interior. The captured image data is then analyzed using a simple image recognition algorithm to identify the type and location of ingredients. This identification data is then sent to a server via the Internet.

[1377] 2. Data analysis on the server

[1378] The server then performs detailed image recognition processing based on the received scan data. Specifically, advanced image recognition algorithms on the server determine the type, quantity, and expiration date of ingredients. For example, it can identify milk, eggs, tomatoes, etc. from the captured image and store this data in a database. At the same time, information on newly added and consumed ingredients is also updated.

[1379] 3. Recipe suggestions

[1380] The server searches a database for recipes that the user can make based on the latest list of ingredients in the refrigerator. It then suggests multiple optimal recipes based on various criteria, such as the user's past preferences, reviews, and cooking time. These recipes are then sent to the user's application or web interface. For example, if milk and eggs are available, recipes such as "French toast" and "omelette" will be suggested.

[1381] 4. Notification of missing ingredients

[1382] The server identifies frequently used ingredients that are in short supply based on the refrigerator's food list and the user's preference data. The server then checks the database for any missing ingredients and notifies the user, for example, "There's only a little lettuce left, so we recommend you buy some." The user can then create a shopping list based on this information.

[1383] Specific examples

[1384] For example, if a user puts eggs, milk, and tomatoes in the refrigerator, the system operates as follows:

[1385] 1. Every morning at 10:00, the device scans the refrigerator and captures images of eggs, milk, and tomatoes.

[1386] 2. The server analyzes the received image in detail and identifies it as 6 eggs, 500ml of milk, and 3 tomatoes.

[1387] 3. The server registers this information in a database and updates the list of ingredients in the refrigerator.

[1388] 4. The server searches for recipes that the user can make based on the ingredient list and suggests dishes such as "omelette" or "pasta with tomato sauce."

[1389] 5. The server analyzes the user's past data and notifies them, "Since you tend to make salads frequently, we recommend that you buy more lettuce."

[1390] 6. The user reviews these suggestions through the app, selects a recipe, starts cooking, and purchases any additional ingredients needed.

[1391] In this way, the system efficiently supports the user's diet and allows for easy food management in the refrigerator.

[1392] The processing flow will be explained below.

[1393] Step 1: The device starts scanning the inside of the refrigerator.

[1394] The device activates the camera inside the refrigerator and scans each shelf and drawer in turn, capturing an image of each scanned section.

[1395] Step 2: The device performs initial image recognition processing.

[1396] The device performs simple local processing on the captured image to identify the type and location of ingredients, and generates identification data (such as the type, location, and quantity of ingredients).

[1397] Step 3: The terminal sends the identification data to the server.

[1398] The terminal sends the captured image data along with the initial recognition results to the server.

[1399] Step 4: The server receives the data.

[1400] The server receives the data sent from the device and prepares for detailed image analysis.

[1401] Step 5: The server performs detailed image analysis.

[1402] The server uses advanced image recognition algorithms to identify detailed information about each ingredient (type, quantity, expiration date).

[1403] Step 6: The server checks against the database.

[1404] The server compares the identified ingredient information with a database to confirm detailed information such as the name, quantity, and expiration date.

[1405] Step 7: The server creates the ingredient list.

[1406] Based on the detailed information, the server creates an up-to-date list of ingredients in the refrigerator and stores it in a database.

[1407] Step 8: The server looks up the recipe.

[1408] Based on the updated ingredient list, the server searches its database for recipes that can be made with the ingredients on hand.

[1409] Step 9: The server evaluates and selects the recipes.

[1410] The server extracts multiple recipe candidates and evaluates and selects the optimal recipe, taking into account conditions such as cooking time and user preferences.

[1411] Step 10: The server notifies the user of the recipe.

[1412] The server notifies the user's application or web interface of the selected recipe list.

[1413] Step 11: The server identifies the missing ingredients.

[1414] The server identifies the ingredients that are missing based on the list of ingredients in the refrigerator and the user's preference data.

[1415] Step 12: The server notifies the user of the missing ingredients.

[1416] The server creates a list of identified ingredients that are in short supply and sends a notification to the user recommending their purchase.

[1417] Step 13: The user receives the notification.

[1418] The user can review the received recipe suggestions and missing ingredient notifications through the application or web interface.

[1419] Step 14: The user selects a recipe and begins cooking.

[1420] The user selects a recipe from the ones notified and begins cooking using ingredients in the refrigerator.

[1421] Step 15: The user buys more ingredients.

[1422] The user creates a shopping list based on the ingredients they are notified of and purchases the missing ingredients in stores or online.

[1423] Example 1

[1424] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1425] Conventional refrigerator food management systems have difficulty accurately grasping the types and quantities of ingredients, making it difficult to properly suggest recipes that can be made with ingredients on hand or ingredients that are missing. Furthermore, they are not sufficient in suggesting recipes based on the user's preferences or in informing the user of missing ingredients. This has resulted in insufficient efficiency in the user's diet and in managing the food in the refrigerator.

[1426] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1427] In this invention, the server includes means for periodically scanning ingredients in the refrigerator, means for transmitting the scanned ingredient information to the server, means for performing detailed analysis based on the scan data to identify the type, amount, and expiration date of the ingredients and update the database, means for suggesting recipes that can be made with the current ingredients to the user based on the updated database information, and means for notifying the user of missing ingredients based on the updated database information. This allows the server to accurately grasp ingredient information in the refrigerator and suggest appropriate recipes to the user and notify them of missing ingredients.

[1428] "Means for periodically scanning ingredients" refers to a device or process that uses a camera or sensor installed inside the refrigerator to capture images and data of ingredients at regular intervals.

[1429] The "means for transmitting scanned ingredient information to a server" is a function or process for transmitting data on ingredients scanned in the refrigerator to a remote server via the Internet or a network.

[1430] "A means of performing detailed analysis based on the scanned data, recognizing the type, quantity, and expiration date of ingredients, and updating the database" refers to a function or process that analyzes the received image data of ingredients using an advanced image recognition algorithm, recognizes detailed information about the ingredients, and reflects this in the database.

[1431] "Means for suggesting recipes to the user that can be made with the current ingredients" is a function or process that searches a database for recipes that can be made using the ingredients in the refrigerator and displays or notifies the user.

[1432] The "means for notifying the user of ingredients that are running low" is a function or process that identifies ingredients that are running low or low in the refrigerator based on database information and notifies the user.

[1433] The "camera and initial image recognition algorithm" refers to a camera device for photographing ingredients in the refrigerator and a program for analyzing the photographed images at an early stage and identifying the general type and location of the ingredients.

[1434] "Means for referencing past preference data to suggest more suitable recipes and missing ingredients" refers to a function or process that analyzes the user's past cooking and ingredient usage history and suggests individually customized recipes and missing ingredients based on that data.

[1435] MODE FOR CARRYING OUT THE INVENTION

[1436] The system of the present invention aims to efficiently manage food ingredients in a refrigerator and support the user's dietary habits. This system consists of a terminal installed in the refrigerator, a server connected via the Internet, and an application or web interface used by the user.

[1437] The device is equipped with a camera and sensors that periodically scan the food items in the refrigerator. Specifically, the camera automatically starts up at 10:00 every morning and takes pictures of each shelf and drawer in the refrigerator in sequence. This collects image data of the food items. The collected data is then simply analyzed using a basic image recognition algorithm to identify the general type and location of the food items. This identification data is then sent to a server via the Internet.

[1438] The server performs detailed image analysis based on the scanned data it receives. Specifically, an advanced image recognition algorithm on the server identifies the type, quantity, and expiration date of ingredients, and stores each piece of data in a database. As a result of the analysis, information on newly added or consumed ingredients in the refrigerator is updated.

[1439] The server then searches for recipes that the user can create based on the updated ingredient list. The server selects the best recipe based on various criteria, such as the user's past preferences, reviews, cooking time, etc. The result is reported to the user's application or web interface.

[1440] Furthermore, the server identifies ingredients that are running low based on the information in the database. This allows users to effectively manage the ingredients in their refrigerator without forgetting to buy the ingredients they need. For example, if a user frequently makes salads, the server has a function that notifies them by saying, "You're running low on lettuce, so we recommend you buy some."

[1441] Specific examples

[1442] For example, if a user puts eggs, milk, and tomatoes in the refrigerator, the system operates as follows:

[1443] 1. Every morning at 10:00, the device scans the refrigerator and captures images of eggs, milk, and tomatoes.

[1444] 2. The server analyzes the received image in detail and identifies it as 6 eggs, 500ml of milk, and 3 tomatoes.

[1445] 3. The server registers this information in a database and updates the list of ingredients in the refrigerator.

[1446] 4. Based on the ingredient list, the server suggests recipes that the user can make, such as "omelette" or "pasta with tomato sauce."

[1447] 5. The server analyzes the user's past data and notifies them, "Since you tend to make salads frequently, we recommend that you buy more lettuce."

[1448] 6. The user reviews these suggestions through the app, selects a recipe, starts cooking, and purchases any additional ingredients needed.

[1449] This system allows users to efficiently manage the ingredients in their refrigerators and support their dietary habits. It is also expected that by knowing in advance which ingredients are running low, users will be able to reduce wasteful shopping and make more effective use of ingredients.

[1450] Prompt Sentence Examples

[1451] A camera and a sensor are installed in the refrigerator. They automatically wake up every morning at 10:00 and scan the ingredients in the refrigerator. The scanned data is sent to a server, which uses advanced image recognition algorithms to identify the ingredients and update its database. The server then uses this information to suggest recipes to the user and notify them of any missing ingredients. Please explain how such a system works.

[1452] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1453] Step 1:

[1454] Start scanning food in the refrigerator

[1455] The device automatically activates the camera at 10:00 every morning and takes pictures of each shelf and drawer in the refrigerator in sequence.

[1456] Input: Video data from a camera installed in a refrigerator.

[1457] Data processing: Image capture of each shelf and drawer.

[1458] Output: Captured image data of the inside of the refrigerator.

[1459] Step 2:

[1460] Initial image recognition processing and data transmission

[1461] The device analyzes the captured image data using a rudimentary image recognition algorithm to identify the general type and location of the food item.

[1462] Input: The captured image data.

[1463] Data calculation: Identifying the type and location of ingredients using an early image recognition algorithm.

[1464] Output: Simple ingredient data identified.

[1465] The terminal transmits the identification data to a server over the Internet.

[1466] Step 3:

[1467] Detailed image analysis

[1468] The server uses advanced image recognition algorithms to perform detailed analysis of the received scan data and identify the type, quantity, and expiration date of the ingredients.

[1469] Input: Simple ingredient data and captured image data sent from the device.

[1470] Data calculation: Analysis using detailed image recognition algorithms.

[1471] Output: Detailed information about the recognized ingredients (type, quantity, expiration date).

[1472] Step 4:

[1473] Database Update

[1474] The server stores the analyzed detailed data of ingredients in a database and updates the list of ingredients in the refrigerator.

[1475] Input: Detailed data of recognized ingredients.

[1476] Data calculation: Saving and updating data in a database.

[1477] Output: Updated ingredient list.

[1478] Step 5:

[1479] Recipe Suggestions

[1480] The server searches the database for recipes that the user can make based on the latest ingredient list.

[1481] Input: Latest ingredient list, user's past preference data, reviews, cooking time.

[1482] Data Computation: Recipe search and selection algorithms.

[1483] Output: A list of recipes suggested to the user.

[1484] The server posts these recipes to the user's application or web interface.

[1485] Step 6:

[1486] Notification of shortage of ingredients

[1487] The server refers to the list of ingredients in the refrigerator and the user's preference data to identify ingredients that may be in short supply.

[1488] Input: Latest ingredient list, user preference data.

[1489] Data calculation: Algorithm for identifying missing ingredients.

[1490] Output: Information about missing ingredients notified to the user.

[1491] The server notifies the user's application of the missing ingredients.

[1492] In this way, the system streamlines food ingredient management and supports the user's eating habits.

[1493] (Application example 1)

[1494] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1495] In recent years, systems aimed at managing food ingredients in refrigerators and improving dietary habits have been attracting attention. However, while these systems can manage food ingredients, they lack the ability to quickly replenish ingredients when they run out. This requires users to manually replenish ingredients, making it difficult to achieve an efficient diet. Furthermore, if food identification and expiration date management are not adequate, food waste can occur. This increases stress and burden on users' daily lives and leads to food waste.

[1496] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1497] In this invention, the server includes a means for scanning ingredients in the refrigerator, a means for comparing the scanned ingredient information with a database, a means for suggesting recipes to the user that can be made with ingredients on hand based on the compared information, a means for suggesting ingredients that are in short supply based on the compared information, and a means for automatically ordering the ingredients that are in short supply. This allows for efficient ingredient management and rapid replenishment. Furthermore, reducing food waste can enrich the user's diet.

[1498] "Means for scanning ingredients in the refrigerator" refers to devices or sensors installed inside the refrigerator to obtain information about ingredients.

[1499] "Scanned food ingredient information" is information including data such as the type, quantity, and expiration date of the food ingredients in the refrigerator.

[1500] "Means for database matching" refers to algorithms or systems that compare scanned ingredient information with existing databases to verify and update their contents.

[1501] "Means of suggesting recipes to users that can be made using ingredients on hand" refers to software or an interface that searches for recipes that can be made using ingredients currently in the refrigerator and presents them to the user.

[1502] "Means for suggesting ingredients that are in short supply to the user" refers to a function or system that identifies ingredients that are in short supply based on information about ingredients on hand and notifies the user of the list.

[1503] "Means for automatic ordering" refers to a system that automatically orders ingredients that are in short supply from external food delivery services, etc., to replenish them.

[1504] "Image capture device" refers to a device such as a camera that takes pictures of food in the refrigerator.

[1505] "Image recognition algorithm" refers to a computer program or technology used to analyze the type and quantity of ingredients from an image captured by an image capture device.

[1506] "Expiration date management" refers to software or systems that track the expiration dates of food ingredients and notify users when the expiration date is approaching.

[1507] The present invention provides a system for efficiently managing food ingredients in a refrigerator and supporting a user's dietary habits. A specific implementation method of this system is described below.

[1508] Overall system overview

[1509] This system consists of a terminal inside the refrigerator, a server connected via the Internet, and an application or web interface used by the user. The terminal is equipped with a camera or sensor that scans ingredients in the refrigerator. The scanned data is sent to the server, where it is analyzed and managed. Based on the analysis results, the server provides the user with ingredient information, suggests recipes, and notifies them of any ingredients they are lacking, and automatically orders the missing ingredients.

[1510] Specific implementation methods

[1511] 1. How to scan ingredients

[1512] The device periodically scans each shelf and drawer in the refrigerator. For example, the device's camera automatically activates at 10:00 every morning and captures images of the refrigerator's interior. The captured image data is then analyzed using a simple image recognition algorithm to identify the type and location of ingredients. This identification data is then sent to a server via the Internet.

[1513] 2. Data analysis on the server

[1514] The server then performs detailed image recognition processing based on the received scan data. Specifically, advanced image recognition algorithms on the server determine the type, quantity, and expiration date of ingredients. For example, it can identify milk, eggs, tomatoes, etc. from the captured image and store this data in a database. At the same time, information on newly added and consumed ingredients is also updated.

[1515] 3. Recipe suggestions

[1516] The server searches a database for recipes that the user can make based on the latest list of ingredients in the refrigerator. It then suggests multiple optimal recipes based on various criteria, such as the user's past preferences, reviews, and cooking time. These recipes are then sent to the user's application or web interface. For example, if milk and eggs are available, recipes such as "French toast" and "omelette" will be suggested.

[1517] 4. Notification of shortage of ingredients and automatic ordering

[1518] The server identifies frequently used ingredients that are in short supply based on the list of ingredients in the refrigerator and the user's preference data. The missing ingredients are checked again against the database, and the user is notified, for example, "There is little lettuce left, so we recommend you buy some." In addition, a function is provided to automatically order these ingredients from a food delivery service to replenish them. This saves users the trouble of shopping and enables efficient food management.

[1519] Specific examples

[1520] For example, if a user puts eggs, milk, and tomatoes in the refrigerator, the system operates as follows:

[1521] 1. Every morning at 10:00, the device scans the refrigerator and captures images of eggs, milk, and tomatoes.

[1522] 2. The server analyzes the received image in detail and identifies it as 6 eggs, 500ml of milk, and 3 tomatoes.

[1523] 3. The server registers this information in a database and updates the list of ingredients in the refrigerator.

[1524] 4. The server searches for recipes that the user can make based on the ingredient list and suggests dishes such as "omelette" or "pasta with tomato sauce."

[1525] 5. The server analyzes the user's past data and notifies them that "Since you tend to make salads frequently, we recommend that you buy more lettuce."

[1526] 6. The server automatically orders the missing ingredients from a food delivery service to replenish them.

[1527] Prompt Sentence Examples

[1528] Example prompt:

[1529] Based on the current list of ingredients in your fridge, suggest recipes you can make and order any missing ingredients from a food delivery service. For example, if your current list of ingredients includes 500ml of eggs, 3 bottles of milk, and 2 tomatoes, suggest recipes you can make and what ingredients you're missing and automatically order them with the appropriate food delivery service.

[1530] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1531] Step 1:

[1532] The device scans the inside of the refrigerator

[1533] Input: Every morning at 10am, the camera inside the refrigerator will automatically turn on.

[1534] Data processing / data calculation: A camera captures images of the ingredients in the refrigerator.

[1535] Output: Image data is generated.

[1536] How it works: The device's built-in camera captures multiple still images or videos, which are then pre-processed using a basic image processing algorithm.

[1537] Step 2:

[1538] The device sends the scanned data to the server.

[1539] Input: Image data acquired in step 1.

[1540] Data processing / data calculation: Image data is compressed and sent to a cloud server via the Internet.

[1541] Output: Image data received by the server.

[1542] Specific operation: The device uses Wi-Fi or other Internet connection to upload image data to a designated storage area on the cloud server.

[1543] Step 3:

[1544] The server analyzes the data using image recognition algorithms

[1545] Input: Image data sent to the server.

[1546] Data processing / data calculation: Using advanced image recognition algorithms (e.g., TensorFlow or OpenCV), the type, quantity, and expiration date of ingredients are identified.

[1547] Output: Parsed food ingredient information database.

[1548] Specific operation: Image recognition software on the server analyzes the image and records information about the identified ingredients in a database.

[1549] Step 4:

[1550] The server suggests recipes to the user

[1551] Input: Ingredient information parsed in step 3.

[1552] Data processing / data calculation: Based on the ingredient information, a matching recipe is searched for from the recipe information in the database.

[1553] Output: A list of recipes suggested to the user.

[1554] Specific operation: The server selects a suitable recipe based on the user's past preferences, reviews, cooking time, etc., and notifies the user's application or web interface.

[1555] Step 5:

[1556] The server notifies the user of any ingredients that are missing.

[1557] Input: Ingredient information analyzed in step 3 and user preference data.

[1558] Data processing / data calculation: Compare the ingredient list with user preference data to identify missing ingredients.

[1559] Output: Notification of missing ingredients.

[1560] Specific operation: The server sends a notification to the user about a shortage of ingredients, such as "Lettuce is low, so we recommend you buy some."

[1561] Step 6:

[1562] The server automatically orders ingredients that are in short supply.

[1563] Input: Shortage ingredients identified in step 5.

[1564] Data processing / data calculation: Send order data for missing ingredients to the designated food delivery service.

[1565] Output: Automatic ordering to food delivery service.

[1566] Specific operation: The server uses the food delivery service's API to automatically order missing ingredients and replenish them in the user's refrigerator.

[1567] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1568] The present invention relates to a system for efficiently managing food items in a refrigerator and supporting the emotions and eating habits of a user. A specific implementation method of this system will be described below.

[1569] Overall system overview

[1570] This system consists of a terminal installed in the user's refrigerator, a server connected via the Internet, and an application or web interface used by the user. It also incorporates an emotion engine that recognizes the user's emotions. The terminal is equipped with a camera and sensors to scan ingredients in the refrigerator. The scanned data is sent to the server for analysis and management. The emotion engine analyzes the user's facial expressions, voice, and text input, and based on this, it makes recipe suggestions and notifies users of missing ingredients.

[1571] Specific implementation methods

[1572] 1. How to scan ingredients

[1573] The device periodically scans each shelf and drawer in the refrigerator. For example, the device's camera automatically activates at 10:00 every morning and captures images of the refrigerator interior. The captured image data is then analyzed using a simple image recognition algorithm to identify the type and location of ingredients. This identification data is then sent to a server via the Internet.

[1574] 2. Data analysis on the server

[1575] The server then performs detailed image recognition processing based on the received scan data. Specifically, advanced image recognition algorithms on the server determine the type, quantity, and expiration date of ingredients. For example, it can identify milk, eggs, tomatoes, etc. from the captured image and store this data in a database. At the same time, information on newly added and consumed ingredients is also updated.

[1576] 3. User Emotion Recognition

[1577] The emotion engine analyzes the user's facial expressions, voice, text input, etc. to recognize the user's emotions. For example, when a user launches the app to add ingredients, the camera captures the user's facial expressions and the voice input is analyzed by the emotion engine. This allows the emotion engine to identify the user's current emotional state (joy, anger, sadness, stress, etc.).

[1578] 4. Linking Recipe Suggestions with Emotions

[1579] The server selects recipes that the user is likely to like based on the emotional data obtained from the emotion engine. For example, if the user is feeling stressed, it will suggest recipes such as "herbal tea" or "spinach smoothie," which have a relaxing effect. If the user is happy, it will suggest recipes such as "chocolate cake," which is easy to make and fun to make.

[1580] 5. Notification of missing ingredients

[1581] The server identifies ingredients that are running low based on the list of ingredients in the refrigerator and data from the emotion engine. For example, if a user is running low on an ingredient needed for a dish they frequently cook, the server will send a notification saying, "You're running low on cabbage, so we recommend you buy some."

[1582] Specific examples

[1583] For example, if a user puts eggs, milk, and tomatoes in the refrigerator, the system operates as follows:

[1584] 1. Every morning at 10:00, the device scans the refrigerator and captures images of eggs, milk, and tomatoes.

[1585] 2. The server analyzes the received image in detail and identifies it as 6 eggs, 500ml of milk, and 3 tomatoes.

[1586] 3. The server registers this information in a database and updates the list of ingredients in the refrigerator.

[1587] 4. The emotion engine analyzes the user's facial expressions and voice and identifies their current emotion as "stress."

[1588] 5. The server will suggest a relaxing recipe, such as a "spinach smoothie," based on the ingredient list and emotional data.

[1589] 6. The server analyzes the user's past data and notifies them, "Since you tend to make salads frequently, we recommend that you buy more lettuce."

[1590] 7. The user reviews these suggestions through the application and plans to make a "spinach smoothie" and buy some lettuce.

[1591] In this way, the system efficiently supports the user's diet and makes suggestions tailored to their emotions, helping them achieve a more satisfying life.

[1592] The processing flow will be explained below.

[1593] Step 1:

[1594] The device will begin scanning the inside of the refrigerator.

[1595] The device activates the camera inside the refrigerator and scans each shelf and drawer in turn, capturing an image of each scanned section.

[1596] Step 2:

[1597] The device performs initial image recognition processing.

[1598] The device performs simple local processing on the captured image to identify the type and location of ingredients, and generates identification data (such as the type, location, and quantity of ingredients).

[1599] Step 3:

[1600] The terminal transmits the identification data to the server.

[1601] The terminal sends the captured image data along with the initial recognition results to the server.

[1602] Step 4:

[1603] The server receives the data.

[1604] The server receives the data sent from the device and prepares for detailed image analysis.

[1605] Step 5:

[1606] The server performs detailed image analysis.

[1607] The server uses advanced image recognition algorithms to identify detailed information about each ingredient (type, quantity, expiration date).

[1608] Step 6:

[1609] The server checks it against its database.

[1610] The server compares the identified ingredient information with a database to confirm detailed information such as the name, quantity, and expiration date.

[1611] Step 7:

[1612] The server creates an ingredient list.

[1613] Based on the detailed information, the server creates an up-to-date list of ingredients in the refrigerator and stores it in a database.

[1614] Step 8:

[1615] The emotion engine recognizes the user's emotions.

[1616] The emotion engine captures and analyzes data such as a user's facial expressions, voice, and text input to identify the user's emotional state.

[1617] Step 9:

[1618] The server searches for the recipe.

[1619] The server searches the database for recipes that can be made with the ingredients on hand based on the updated ingredient list and the emotional data obtained from the emotion engine.

[1620] Step 10:

[1621] The server evaluates and selects recipes.

[1622] The server extracts multiple recipe candidates and evaluates and selects the optimal recipe taking into account factors such as the user's emotional state, cooking time, and past preferences.

[1623] Step 11:

[1624] The server notifies the user of the recipe.

[1625] The server notifies the user's application or web interface of the selected recipe list.

[1626] Step 12:

[1627] The server identifies the missing ingredients.

[1628] The server identifies the ingredients that are missing based on the list of ingredients in the refrigerator and the user's preference data.

[1629] Step 13:

[1630] The server notifies the user of the missing ingredients.

[1631] The server creates a list of identified ingredients that are in short supply and sends a notification to the user recommending their purchase.

[1632] Step 14:

[1633] The user receives a notification.

[1634] The user can review the received recipe suggestions and missing ingredient notifications through the application or web interface.

[1635] Step 15:

[1636] The user selects a recipe and begins cooking.

[1637] The user selects a recipe from the ones notified and begins cooking using ingredients in the refrigerator.

[1638] Step 16:

[1639] The user buys more ingredients.

[1640] The user creates a shopping list based on the ingredients they are notified of and purchases the missing ingredients in stores or online.

[1641] With this specific processing flow, the system efficiently manages the user's ingredients and suggests recipes according to their emotional state, supporting a more satisfying diet.

[1642] Example 2

[1643] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1644] In modern life, there is a demand for efficient management of food in the refrigerator and support for dietary habits that are tailored to the user's mood. Conventional systems require manual management of food inventory and recipe suggestions, which poses a challenge in that they are unable to suggest recipes tailored to the user's mood or notify users when ingredients are in short supply. This can cause unnecessary stress for users. Furthermore, users must also manually manage expiration dates, which can lead to food waste.

[1645] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1646] In this invention, the server includes a means for scanning ingredients in the refrigerator, a means for comparing the scanned ingredient information with a database, and a means for analyzing the user's emotional state. This automates the management of ingredients in the refrigerator, making it possible to suggest recipes tailored to the user's emotions and notify the user of any missing ingredients. Furthermore, the server automatically manages expiration dates, reducing food waste and improving the user's quality of life.

[1647] "Means for scanning food ingredients in a refrigerator" is a general term for devices or mechanisms that use cameras or sensors installed in the refrigerator to acquire images and data of food ingredients in the refrigerator periodically or as needed.

[1648] "Means for comparing scanned ingredient information with a database" is a general term for systems and algorithms that analyze the images and data of ingredients obtained by scanning and compare that information with information in an existing database.

[1649] "Means for suggesting recipes to users that can be made using ingredients on hand" is a general term for algorithms and procedures for presenting users with recipes that can be made using ingredients currently in the refrigerator, based on ingredient information in the database.

[1650] "Means for suggesting ingredients that are in short supply to users" is a general term for systems and algorithms that analyze ingredient inventory based on ingredient information stored in a database and notify users of ingredients that are in short supply.

[1651] "Means for analyzing a user's emotional state" is a general term for systems or algorithms that analyze a user's facial expressions, voice, or text input to identify the user's current emotional state.

[1652] "Means for suggesting appropriate recipes to users based on their emotional state" is a general term for algorithms and systems that take into account the user's emotional state and suggest to the user the recipe that is best suited to that situation.

[1653] "Means for managing expiration dates based on food ingredient information" is a general term for systems and algorithms that automatically manage the expiration dates of each ingredient based on scanned food ingredient information and notify users.

[1654] "Means for notifying users of shortages of ingredients" is a general term for systems and algorithms that notify users when they are running low on ingredients that they frequently use or need, based on the inventory status of ingredients.

[1655] This invention relates to a system that efficiently manages food ingredients in a refrigerator and supports a dietary lifestyle that matches the user's emotions. Specifically, it is composed of a terminal installed in the refrigerator, a server connected via the Internet, and an application or web interface used by the user. It also incorporates an emotion engine that recognizes the user's emotions.

[1656] Program processing

[1657] Food scanning

[1658] The device periodically scans each shelf and drawer in the refrigerator. Specifically, the device's built-in camera (e.g., a high-resolution camera) automatically starts up at 10:00 every morning and captures images of the inside of the refrigerator. This image data is temporarily stored in the device's internal memory. The encrypted image data is then sent to a server via the Internet via a Wi-Fi module (e.g., ESP8266).

[1659] Data analysis

[1660] The server analyzes the received image data. Specifically, it uses an image recognition algorithm (e.g., TensorFlow) on the server to identify the type, quantity, and location of ingredients. The results of this analysis are registered in a database, and the list of ingredients in the refrigerator is automatically updated. At the same time, it also determines expiration dates and identifies any missing ingredients.

[1661] emotion recognition

[1662] The emotion engine analyzes the user's facial expressions, voice, and text input. For example, when a user launches an app, the device's camera and microphone capture the user's facial expressions and voice. For analysis, OpenFace and Google Cloud Speech-to-Text API are used, for example. Based on this, the emotion engine identifies the user's emotional state, and this information is sent to the server.

[1663] Recipe suggestions and missing ingredient notifications

[1664] The server suggests appropriate recipes to the user based on the data obtained from the emotion engine and the scanned ingredient information. For example, if the user is feeling stressed, it will suggest a relaxing recipe such as a "spinach smoothie." If it is determined that an ingredient is in short supply, it will generate a notification such as "We are running low on cabbage, so we recommend you buy some," and send it to the user's application.

[1665] Specific examples

[1666] For example, if a user puts eggs, milk, and tomatoes in the refrigerator, the system will do the following:

[1667] 1. Every morning at 10:00, the device scans the refrigerator and captures images of eggs, milk, and tomatoes.

[1668] 2. The server analyzes the received image in detail and identifies it as 6 eggs, 500ml of milk, and 3 tomatoes.

[1669] 3. The server registers this information in a database and updates the list of ingredients in the refrigerator.

[1670] 4. The emotion engine analyzes the user's facial expressions and voice and identifies their current emotion as "stress."

[1671] 5. Based on the ingredient list and emotional data, the server suggests a relaxing recipe: "Spinach Smoothie."

[1672] 6. The server analyzes the user's past data and notifies them, "Since you tend to make salads frequently, we recommend that you buy more lettuce."

[1673] 7. The user reviews these suggestions through the application and plans to make a "spinach smoothie" and buy some lettuce.

[1674] In this way, the system efficiently supports the user's diet and makes suggestions tailored to their emotions, helping them achieve a more satisfying life.

[1675] Prompt Sentence Examples

[1676] "I have eggs, milk, and tomatoes in my fridge, but I've been feeling stressed lately. Can you suggest a relaxing recipe using these ingredients?"

[1677] "Also, check out the other ingredients you need for salads you make frequently."

[1678] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1679] Step 1:

[1680] The device scans each shelf and drawer in the refrigerator.

[1681] Input: The device's camera captures images of the inside of the refrigerator according to the user's schedule (e.g., every morning at 10:00).

[1682] Data processing: The captured images are stored in the device's internal memory.

[1683] Specific operation: The camera operates to take pictures of the shelves inside the refrigerator in sequence and generate image data.

[1684] Output: Image data of scanned ingredients.

[1685] Step 2:

[1686] The device encrypts the scanned image data and sends it to a server via the Internet.

[1687] Input: The image data generated in step 1.

[1688] Data processing: Image data is encrypted and transmitted through the Wi-Fi module.

[1689] Specific operation: The device uses Wi-Fi to send encrypted image data to the server.

[1690] Output: Encrypted image data sent to the server.

[1691] Step 3:

[1692] The server analyzes the received image data.

[1693] Input: Encrypted image data.

[1694] Data processing: Using an image recognition algorithm (e.g., TensorFlow) on the server, the type, quantity, and location of ingredients are identified, and this information is saved and updated in the database.

[1695] Specific operation: The server runs an image recognition algorithm and stores the identified ingredient data in a database.

[1696] Output: Parsed ingredient data, updated database.

[1697] Step 4:

[1698] An emotion engine analyzes the user's emotional state.

[1699] Input: User facial expressions, voice, and text input.

[1700] Data Processing: The emotion engine analyzes this data and identifies the user's emotional state (e.g., stress, joy, sadness) using tools (e.g., OpenFace, Google Cloud Speech-to-Text API).

[1701] Specific operation: The camera captures the user's facial expressions, the microphone records audio, and the emotion engine analyzes it.

[1702] Output: Parsed emotional state data.

[1703] Step 5:

[1704] The server suggests appropriate recipes based on emotional state data and scanned ingredient data.

[1705] Input: Emotional state data, food ingredient data.

[1706] Data processing: The server searches the database for recipes that match the emotion and suggests them to the user.

[1707] Specific operation: The server performs a database search and selects recipes that match the emotion.

[1708] Output: The proposed recipe.

[1709] Step 6:

[1710] The server identifies the missing ingredients based on the scanned ingredient data and notifies the user.

[1711] Input: Ingredient data, shortage conditions.

[1712] Data processing: The server checks the ingredient list and identifies if any frequently used or necessary ingredients are missing.

[1713] Specific operation: The server analyzes the ingredient list and extracts information about missing ingredients.

[1714] Output: Identification of missing ingredients and recommended purchase notifications.

[1715] Step 7:

[1716] The server sends the suggested recipe and missing ingredient notification to the user's application.

[1717] Input: Suggested recipes, notifications of missing ingredients.

[1718] Data processing: Generation and transmission of notification data.

[1719] Specific operation: The server generates a notification and sends it to the user's smartphone.

[1720] Output: Notification to the user's smartphone.

[1721] Step 8:

[1722] The user checks the notification through the application and makes a plan to create the suggested recipe or purchase the missing ingredients.

[1723] Enter: notifications on your smartphone.

[1724] Data processing: The user decides what to do based on the notification.

[1725] Specific action: The user opens the application on their smartphone and checks the notification.

[1726] Output: Recipe creation or ingredient shopping planning.

[1727] (Application example 2)

[1728] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1729] Currently, refrigerator food management systems are limited to managing ingredients and suggesting recipes. However, they are unable to respond to the user's emotional state and are unable to provide more personalized services based on the user's emotions and eating habits. Therefore, a new system is needed that can provide comprehensive support for the user's eating habits by suggesting cooking methods based on emotions and suggesting the purchase of missing ingredients.

[1730] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for scanning ingredients in the refrigerator, means for comparing the scanned ingredient information with a database, means for suggesting to the user recipes that can be made with ingredients on hand based on the compared information, means for suggesting to the user ingredients that are missing, emotion analysis means for recognizing the user's emotional state, means for suggesting recipes appropriate for the user's emotional state based on the emotion data recognized by the emotion analysis means, and means for suggesting the purchase of ingredients that are missing. This makes it possible to suggest dietary habits that take the user's emotional state into consideration, thereby improving user satisfaction and achieving efficient ingredient management.

[1731] "Means for scanning food items in a refrigerator" refers to technology that uses image capture devices such as cameras and sensors and image recognition algorithms to detect the type, quantity, and location of food items in a refrigerator.

[1732] The "means for comparing the scanned ingredient information with a database" is a technology that compares the acquired ingredient information in the refrigerator with a pre-registered ingredient database to identify matching ingredients and new ingredients that have been added.

[1733] "Means for suggesting recipes to users that can be made using ingredients on hand" refers to technology that provides users with recipes that can be made using ingredients in the refrigerator, based on recipe information stored in a database.

[1734] The "means of suggesting missing ingredients to the user" is a technology that compares the list of ingredients in the refrigerator with recipe information and notifies the user if any ingredients needed to make a specific dish are missing.

[1735] "Emotion analysis means for recognizing the user's emotional state" is a technology that analyzes data such as the user's facial expressions, voice, and text input to identify the user's emotional state.

[1736] "Means for suggesting recipes appropriate to the emotional state based on the emotional data recognized by the emotion analysis means" refers to a technology that suggests the best dishes and drinks for a user based on the user's emotional state (e.g., stress, joy).

[1737] The "means for suggesting the purchase of ingredients that are in short supply" is a technology that notifies the user to purchase ingredients that are in short supply and supports the purchase based on the ingredient data in the refrigerator and the user's emotional state.

[1738] The following describes in detail an embodiment of the present invention. The system comprises a terminal installed in a refrigerator, a server connected via the Internet, and an application or web interface used by a user.

[1739] Food scanning method

[1740] The device is equipped with an image capture device such as a camera or sensor to scan the food items in the refrigerator. This periodically captures images of the food items in the refrigerator. The image data is analyzed using an image recognition algorithm to identify the type and quantity of food items and their expiration dates. This process uses the open source libraries OpenCV and Keras / TensorFlow.

[1741] Database matching method

[1742] The scanned food information is sent to a server via the Internet. The server compares the received data with a database to manage which food ingredients are stored in the refrigerator and in what quantities. This comparison means updates and registers new ingredients to the database.

[1743] Recipe suggestion method

[1744] The server is equipped with a means for suggesting possible recipes based on the user's available ingredients. The server analyzes the recipe information in the database and notifies the user of recipes that can be made using the ingredients in the refrigerator. The server also takes into account the user's past preference data.

[1745] Means for suggesting insufficient ingredients

[1746] The server analyzes the information to identify ingredients that are in short supply and notifies the user, allowing them to know what ingredients they need to buy. Furthermore, the server also provides a function to link with delivery services and purchase the suggested ingredients online.

[1747] Emotion analysis means

[1748] The system also includes an emotion analysis mechanism to recognize the user's emotional state by analyzing the user's facial expressions, voice, and text input data to identify emotions, using a pre-trained emotion recognition model (Keras / TensorFlow).

[1749] Emotion-based recipe suggestion method

[1750] Based on the emotional data recognized by the emotion analysis means, the server suggests cooking recipes suited to the user's emotional state, for example, if the user is under stress, it suggests dishes and drinks that have a relaxing effect.

[1751] Proposal method for purchasing ingredients that are in short supply

[1752] The server notifies the user to purchase ingredients that are in short supply based on the food data in the refrigerator and the user's emotional state, and also provides an interface for purchasing ingredients online.

[1753] Specific examples

[1754] For example, if a user comes home tired from work, the system scans the refrigerator to obtain food ingredient data, and if it detects a "stressed state" from the user's facial expression, it will suggest recipes such as relaxing herbal tea or spinach smoothie. If any ingredients are missing, the system will notify the user so that they can purchase them directly online.

[1755] Prompt Sentence Examples

[1756] Scan images of your refrigerator to suggest recipes based on current ingredients and your emotions, and let you know if you're missing any ingredients.

[1757] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1758] Step 1:

[1759] The device scans the ingredients in the refrigerator.

[1760] Input: Image data of the inside of the refrigerator.

[1761] Output: Image data of the acquired ingredients.

[1762] Specific operation: The camera installed in the refrigerator will automatically start up at a set time (for example, 10:00 every morning) and take pictures of each shelf inside the refrigerator.

[1763] Step 2:

[1764] The terminal transmits the acquired image data to the server.

[1765] Input: Image data of the acquired ingredients.

[1766] Output: Image data sent to the server.

[1767] How it works: After capturing an image, it automatically sends the image data to a server over the Internet. Encryption is used to ensure security.

[1768] Step 3:

[1769] The server analyzes the image data and extracts ingredient information.

[1770] Input: The submitted image data.

[1771] Output: Parsed ingredient information (e.g. type, quantity, expiration date).

[1772] Specific operation: The server runs an image recognition algorithm using Keras / TensorFlow to identify the type and quantity of ingredients from the image, and also determine the expiration date of each ingredient.

[1773] Step 4:

[1774] The server compares the extracted ingredient information with the database.

[1775] Input: Parsed ingredient information.

[1776] Output: Updated ingredient information in the database.

[1777] Specific operation: The server updates the ingredient list by updating the database with new ingredients and information on ingredients that have been consumed.

[1778] Step 5:

[1779] A user uses the sentiment analysis means through an application.

[1780] Input: User's facial expression, voice, and text data.

[1781] Output: The perceived emotional state of the user.

[1782] How it works: The user launches the app and inputs facial and voice data via the camera and microphone into the emotion recognition model, which analyzes this data to identify the user's current emotional state (e.g., stress, joy).

[1783] Step 6:

[1784] The server suggests cooking methods that suit the emotional state based on emotion analysis data.

[1785] Input: User's emotional state, food ingredient information from the database.

[1786] Output: A list of suggested dishes appropriate for the emotional state.

[1787] Specific operation: Based on the emotion recognition results, the server selects the recipe that best suits the user's current emotion (e.g., "spinach smoothie" if stressed) and generates a list of suggestions.

[1788] Step 7:

[1789] The server identifies the ingredients that are in short supply and notifies the user to purchase them.

[1790] Input: Updated database of ingredients, suggested recipes.

[1791] Output: A list of missing ingredients with suggested purchases.

[1792] What happens: The server checks if the ingredients needed for the proposed recipe are in its database, and if any ingredients are missing (e.g., lettuce), it lists them and generates a notification.

[1793] Step 8:

[1794] The user checks the suggested recipe and missing ingredients and uses the delivery service.

[1795] Input: Suggested recipe, list of missing ingredients suggested for purchase.

[1796] Output: Decision to purchase or use delivery service.

[1797] What it does: The user uses the app to review suggested recipes and a list of missing ingredients, and then purchases ingredients online or orders food delivery as needed.

[1798] Through these steps, the system can provide personalized dietary support tailored to the user's emotional state.

[1799] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1800] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1801] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1802] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1803] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1804] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1805] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1806] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1807] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1808] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1809] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1810] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1811] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1812] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1813] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1814] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1815] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1816] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1817] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1818] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1819] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1820] The following is further disclosed regarding the above embodiment.

[1821] (Claim 1)

[1822] means for scanning food items in a refrigerator;

[1823] means for comparing the scanned ingredient information with a database;

[1824] A means for suggesting recipes to the user that can be made with ingredients on hand based on the collated information;

[1825] A means for suggesting missing ingredients to the user based on the collated information;

[1826] A system including:

[1827] (Claim 2)

[1828] 10. The system of claim 1, wherein the means for scanning ingredients in the refrigerator includes an image capture device and an image recognition algorithm.

[1829] (Claim 3)

[1830] The system according to claim 1, further comprising a means for managing expiration dates based on the food ingredient information.

[1831] "Example 1"

[1832] (Claim 1)

[1833] a means for periodically scanning food items in the refrigerator;

[1834] means for transmitting the scanned ingredient information to a server;

[1835] A means for performing detailed analysis based on the scan data, recognizing the type, amount, and expiration date of ingredients, and updating the database;

[1836] A means for suggesting recipes that can be made with current ingredients to the user based on the updated database information;

[1837] a means for notifying the user of the ingredients that are in short supply based on the updated database information;

[1838] A system including:

[1839] (Claim 2)

[1840] 10. The system of claim 1, wherein the means for scanning ingredients in the refrigerator includes a camera and an initial image recognition algorithm.

[1841] (Claim 3)

[1842] The system according to claim 1, further comprising means for referring to past preference data based on the ingredient information and suggesting more suitable recipes and ingredients that are lacking.

[1843] "Application Example 1"

[1844] (Claim 1)

[1845] means for scanning food items in a refrigerator;

[1846] means for comparing the scanned ingredient information with a database;

[1847] A means for suggesting recipes to the user that can be made with ingredients on hand based on the collated information;

[1848] A means for suggesting missing ingredients to the user based on the collated information;

[1849] means for automatically ordering the missing ingredients;

[1850] A system including:

[1851] (Claim 2)

[1852] 10. The system of claim 1, wherein the means for scanning ingredients in the refrigerator includes an image capture device and an image recognition algorithm.

[1853] (Claim 3)

[1854] The system according to claim 1, further comprising a means for managing expiration dates based on the food ingredient information.

[1855] "Example 2: Combining Emotion Engines"

[1856] (Claim 1)

[1857] means for scanning food items in a refrigerator;

[1858] means for comparing the scanned ingredient information with a database;

[1859] A means for suggesting recipes to the user that can be made with ingredients on hand based on the collated information;

[1860] A means for suggesting missing ingredients to the user based on the collated information;

[1861] means for analyzing the emotional state of a user;

[1862] means for suggesting an appropriate recipe to the user based on the emotional state;

[1863] a means for notifying a user of ingredients that are in short supply based on the emotional state and ingredient information;

[1864] A system including:

[1865] (Claim 2)

[1866] 10. The system of claim 1, wherein the means for scanning ingredients in the refrigerator includes an image capture device and an image recognition algorithm.

[1867] (Claim 3)

[1868] The system according to claim 1, further comprising a means for managing expiration dates based on the food ingredient information.

[1869] "Application example 2 when combining emotion engines"

[1870] (Claim 1)

[1871] means for scanning food items in a refrigerator;

[1872] means for comparing the scanned ingredient information with a database;

[1873] A means for suggesting recipes to the user that can be made using ingredients on hand based on the collated information;

[1874] A means for suggesting missing ingredients to the user based on the collated information;

[1875] emotion analysis means for recognizing the emotional state of a user;

[1876] means for suggesting a recipe suited to an emotional state based on the emotional data recognized by the emotion analysis means;

[1877] A means of suggesting the purchase of ingredients that are in short supply,

[1878] A system including:

[1879] (Claim 2)

[1880] 10. The system of claim 1, wherein the means for scanning ingredients in the refrigerator includes an image capture device and an image recognition algorithm.

[1881] (Claim 3)

[1882] The system according to claim 1, further comprising a means for managing expiration dates based on the food ingredient information. [Explanation of symbols]

[1883] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for scanning food items in a refrigerator; means for comparing the scanned ingredient information with a database; A means for suggesting recipes to the user that can be made with ingredients on hand based on the collated information; A means for suggesting missing ingredients to the user based on the collated information; A system including:

2. The system of claim 1 , wherein the means for scanning ingredients in the refrigerator includes an image capture device and an image recognition algorithm.

3. The system according to claim 1 , further comprising a means for managing expiration dates based on the food ingredient information.

Images