System

Abstract

A system is provided.SOLUTION: The system includes an input means for inputting food consumption habits and cooking preferences of a user, a means for receiving information from the input means and storing the information in a database, a means for automatically photographing an image in a refrigerator periodically, a means for receiving and analyzing image data obtained by the photographing means, a means for generating a shopping list based on the analyzed inventory information and the stored consumption habits, a means for notifying a user terminal of the generated shopping list, and a means for proposing an optimal recipe based on the shopping list.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] The present invention relates to a system for efficiently managing ingredients and preparing meals at home. Conventionally, users have often found it complicated to avoid forgetting to buy ingredients, discarding surplus ingredients, and selecting recipes. It has also been difficult to accurately grasp the inventory status in the refrigerator and create a list of necessary ingredients. This can lead to unnecessary shopping and time wasted, and can also result in a decline in the quality of meals at home. Therefore, there has been a demand for a system that can learn a user's ingredient consumption habits, automatically grasp the inventory status in the refrigerator, and suggest optimal shopping lists and recipes. [Means for solving the problem]

[0005] The present invention solves the above-mentioned problems by providing a system that includes an input means for a user to input their ingredient consumption habits and cooking preferences, a means for receiving information from the input means and saving the information in a database, a photographing means for automatically taking images of the inside of the refrigerator periodically, a means for receiving and analyzing the image data obtained by the photographing means, a means for generating a shopping list based on the analyzed inventory information and the saved consumption habits, a means for notifying a user terminal of the generated shopping list, and a means for suggesting optimal recipes based on the shopping list. This allows the user to always know the inventory status in the refrigerator, easily obtain an optimal shopping list, and efficiently cook by receiving recipes based on the ingredients they have on hand.

[0006] The "input means" is an interface that allows the user to input information such as food consumption habits, cooking preferences, and allergy information.

[0007] The "means for saving in a database" is a system for accumulating information obtained from the input means and saving it in a database.

[0008] The "photography means" is a device such as a camera that automatically takes photos of the inventory status inside the refrigerator on a regular basis.

[0009] The "means for receiving and analyzing image data" is a system for receiving image data obtained by the photographing means and analyzing it using an image recognition algorithm.

[0010] The "means for generating a shopping list" is a system that lists the ingredients needed based on the analyzed inventory information and stored consumption habits.

[0011] The "means for notifying the user terminal" is a system for sending the generated shopping list to the user's terminal and notifying the user.

[0012] The "means for suggesting optimal recipes" is a system that selects and suggests appropriate recipes to users based on the user's food consumption habits and the current inventory status in the refrigerator.

[0013] The "learning means" is a system that uses input data to learn the user's food consumption patterns using a machine learning algorithm. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0015] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0018] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0019] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0020] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0022] [First embodiment]

[0023] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0024] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0027] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0030] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0034] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0035] The present invention relates to a system that learns a user's food consumption habits and refrigerator inventory status and proposes optimal shopping lists and recipes. Specific embodiments for implementing the system of the present invention will be described below.

[0036] System configuration

[0037] The system mainly includes the following components:

[0038] 1. Input Method

[0039] Using a dedicated application, users input their food consumption habits, cooking preferences, allergy information, etc. This information is used as the initial data for the system.

[0040] 2. How to store data in a database

[0041] The terminal transmits the data entered by the user to the server.

[0042] The server stores the received data in a database, which records detailed food consumption habits for each user.

[0043] 3. Filming Method

[0044] The device (refrigerator camera) periodically takes photos of the inventory status inside the refrigerator. For example, photos are taken automatically at set times, such as every morning or after meals.

[0045] 4. Means of receiving and analyzing image data

[0046] The terminal transmits the captured image data to the server.

[0047] The server analyzes the received image data using an image recognition algorithm to identify the inventory status within the refrigerator, and the analysis results are stored in a database.

[0048] 5. How to generate a shopping list

[0049] The server predicts the ingredients needed based on the user's food consumption patterns and the latest refrigerator inventory data, and generates a shopping list.

[0050] 6. Means of notifying the user terminal

[0051] The server sends the generated shopping list and suggested recipes to the user's device, where a dedicated application notifies and displays them.

[0052] 7. A way to suggest the best recipes

[0053] The server selects and suggests appropriate recipes to the user based on the user's food consumption habits and current refrigerator inventory.

[0054] Program processing

[0055] Next, the processing of the system program will be explained in natural language.

[0056] 1. User Initial Registration

[0057] Users enter their food consumption habits, cooking preferences, and allergy information through a dedicated application.

[0058] The terminal transmits these input data to the server.

[0059] The server receives the data, stores it in a database, and initializes a machine learning model to learn the user's food consumption patterns.

[0060] 2. Check the refrigerator inventory

[0061] The device (refrigerator camera) automatically takes pictures of the inside of the refrigerator at 6:00 every morning.

[0062] The terminal transmits the captured image data to the server.

[0063] The server uses image recognition algorithms to analyze the image data and determine the inventory status of the refrigerator, and the analysis results are updated in a database.

[0064] 3. Shopping list generation and notifications

[0065] The server predicts the ingredients needed at 6pm based on the user's food consumption patterns and the latest inventory status, and generates a shopping list.

[0066] The server sends the generated shopping list and recommended recipes to the user's device.

[0067] The device notifies the user through a dedicated app and displays shopping lists and recipes.

[0068] 4. User shopping and cooking

[0069] Users open the app, check their shopping list, and purchase the ingredients they need.

[0070] Users cook using recipes suggested by the app based on the ingredients they have purchased and the inventory they have in stock.

[0071] Specific examples

[0072] For example, if a user specifies that they consume milk and eggs every week, the system will learn this. While checking the refrigerator inventory each day, it will determine that there is one liter of milk and two eggs left. When the user goes shopping, the system will automatically generate a shopping list that includes "one liter of milk and six eggs" and notify the app. The system will also suggest recipes such as omelets and French toast based on the ingredients in the refrigerator.

[0073] In this way, the system of the present invention allows users to avoid forgetting to buy ingredients or wasting them, and allows them to shop and cook efficiently.

[0074] The processing flow will be explained below.

[0075] Step 1:

[0076] Users install a dedicated application and enter their food consumption habits, cooking preferences, and allergy information, specifically the ingredients they use each week and the dishes they make on specific days.

[0077] Step 2:

[0078] The device sends the user's input data to the server, including the frequency of food consumption, type of cuisine, and a list of ingredients that may cause allergies.

[0079] Step 3:

[0080] The server stores the received data in a database, where the data is managed separately for each user.

[0081] Step 4:

[0082] The server uses the stored data to initialize a machine learning model and learns the user's food consumption patterns. For example, the model can learn that the user uses eggs every Monday.

[0083] Step 5:

[0084] The device (refrigerator camera) automatically takes a photo of the inside of the refrigerator every morning at 6:00 or at a time specified by the user.

[0085] Step 6:

[0086] The device sends the captured image data to the server. This image must show the entire interior of the refrigerator.

[0087] Step 7:

[0088] The server then uses an image recognition algorithm to analyze the received image data and identify the types and quantities of food items in the refrigerator, such as whether there is one liter of milk left or two eggs.

[0089] Step 8:

[0090] The server updates the analysis results and stores them in a database, which is updated in real time.

[0091] Step 9:

[0092] The server generates a shopping list every day at 6pm based on the user's food consumption patterns and the latest inventory information. For example, one liter of milk and six eggs are added to the list.

[0093] Step 10:

[0094] The server generates a shopping list and selects the best recipes based on the user's preferences, such as omelet and French toast recipes.

[0095] Step 11:

[0096] The server sends the shopping list and recipes to the user's device.

[0097] Step 12:

[0098] The device will display a notification to the user through a dedicated application, and the user can open the app to view the displayed shopping list and recipes.

[0099] Step 13:

[0100] The user goes to the supermarket and buys the ingredients they need based on the shopping list, for example, adding 1 liter of milk and 6 eggs to their cart.

[0101] Step 14:

[0102] The app suggests recipes based on the ingredients purchased and the inventory in the refrigerator, allowing users to cook dishes such as omelets and French toast.

[0103] Example 1

[0104] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0105] Conventional refrigerator management systems require users to manually manage food inventory, which is time-consuming and can lead to forgotten purchases and wasteful consumption. It is also difficult to provide effective shopping lists and recipes that take into account the user's food consumption habits. This leads to inefficiencies in the user's lifestyle and wastes time and energy.

[0106] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0107] In this invention, the server includes input means for inputting a user's ingredient consumption habits and cooking preferences, means for receiving information from the input means and storing the information in a database, photographing means for automatically photographing images of the inside of the refrigerator periodically, means for receiving and analyzing image data obtained by the photographing means, means for generating a shopping list based on the analyzed inventory information and the stored consumption habits, means for notifying a user terminal of the generated shopping list, means for using a machine learning algorithm to predict optimal ingredients based on the suggested shopping list, and means for suggesting recipes based on the optimal ingredients and the user's consumption habits, thereby enabling users to efficiently manage ingredient consumption and inventory without hassle.

[0108] "User" refers to a person who uses the system to input their food consumption habits and cooking preferences and receive suggested shopping lists and recipes.

[0109] "Food consumption habits" refers to data that indicates the tendency and frequency of food ingredients consumed by a user during a specific period.

[0110] "Cooking preferences" refers to information about the user's favorite dishes and cooking methods.

[0111] "Input means" refers to an interface that allows a user to input their food consumption habits and cooking preferences into the system.

[0112] "Database" refers to an information system for storing information received from users, analysis results, inventory information, etc.

[0113] "Photographing means" refers to a device or equipment that automatically takes pictures of the inside of the refrigerator on a regular basis.

[0114] "Image data" refers to photographs and video data of the inside of the refrigerator captured by a photographing means.

[0115] "Analysis means" refers to a system or method that uses an image recognition algorithm to analyze captured image data and identify inventory information within the refrigerator.

[0116] "Inventory information" refers to data about the ingredients currently in the refrigerator and their quantities.

[0117] A "shopping list" is a list created by predicting the ingredients a user will need in the future.

[0118] "Notification means" refers to a system or method for sending the generated shopping list and suggested recipes to a user terminal.

[0119] "Machine learning algorithm" refers to a calculation method or program that learns from a user's food consumption habits and refrigerator inventory information to predict the best ingredients.

[0120] "Recipe suggestion means" refers to a system or method for providing a user with a recipe suitable for cooking based on optimal ingredients and the user's consumption habits.

[0121] The present invention relates to a system that learns a user's food consumption habits and refrigerator inventory status, and proposes optimal shopping lists and recipes. This system is implemented using the following hardware and software.

[0122] 1. Hardware Configuration

[0123] The system includes the following hardware:

[0124] User device: Mobile device such as a smartphone or tablet

[0125] Refrigerator camera: A camera that periodically takes pictures of the inside of a refrigerator

[0126] Server: A computer system that runs a database and image recognition and machine learning algorithms.

[0127] 2. Software Configuration

[0128] The system includes the following software:

[0129] Dedicated application: An app for users to input information about food consumption habits, cooking preferences, and allergies

[0130] Database management system: Databases such as MySQL (registered trademark) and PostgreSQL

[0131] Image recognition algorithm: Object detection algorithm using OpenCV and TENSORFLOW (registered trademark)

[0132] Machine learning algorithms: predictive models using scikit-learn and TensorFlow

[0133] 3. System Operation

[0134] Onboarding a User

[0135] Users enter their food consumption habits, cooking preferences, allergy information, etc. through a dedicated application. This information is used as the initial data for the system.

[0136] The terminal transmits the data entered by the user to the server.

[0137] The server stores the received data in a database and initializes and starts training a machine learning model to learn the user's food consumption patterns.

[0138] Checking the inventory in the refrigerator

[0139] The device (refrigerator camera) automatically takes pictures of the inside of the refrigerator at a fixed time every day. This time can be set by the user.

[0140] The terminal transmits the captured image data to the server.

[0141] The server analyzes the image data using an image recognition algorithm to identify the inventory status of the refrigerator, and the analysis results are reflected in a database.

[0142] Shopping list generation and notifications

[0143] The server predicts the ingredients needed based on the user's ingredient consumption patterns and the latest inventory status, and generates a shopping list.

[0144] The server sends the generated shopping list to the user's device, and a dedicated app notifies the user and displays the list to them.

[0145] Recipe Suggestions

[0146] The server selects and suggests appropriate recipes based on the user's consumption habits and refrigerator inventory. Recipe suggestions are made via a dedicated app.

[0147] 4. Specific Examples

[0148] For example, if a user specifies that they "eat eggs and bread for breakfast every day," the system learns this and predicts the ingredients they will need based on their consumption patterns over the course of a week. A camera inside the refrigerator checks the inventory every day and generates a shopping list based on the analyzed results. When the user opens the app on their smartphone, a shopping list of "10 eggs and 1 bag of bread" is displayed. The system also suggests recipes such as "French toast" and "omelette" based on the inventory status in the refrigerator.

[0149] Prompt Sentence Examples

[0150] "Can you recommend a simple and delicious recipe for dinner? I have eggs and milk in the fridge."

[0151] In this way, the system of the present invention allows users to receive efficient support regarding the management and consumption of ingredients, preventing them from forgetting to buy food or wasting it.

[0152] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0153] Step 1: Onboarding users

[0154] Users open a dedicated app and enter data such as food consumption habits, cooking preferences, and allergy information.

[0155] Input: Food consumption habits, cooking preferences, allergy information

[0156] Specific action: The user enters the required information into a form within the app and presses the submit button.

[0157] The terminal receives input data and transmits it to the server.

[0158] Output: Sending user-entered data

[0159] The server stores the received data in a database.

[0160] Input: User's consumption habits, food preferences, and allergy information

[0161] Data processing: Store information in the database in JSON format and organize it by user ID.

[0162] Output: Database update

[0163] The server initializes the machine learning model with the user's consumption patterns and performs initial training.

[0164] Input: Initial database information

[0165] Data Computing: Model training with machine learning algorithms

[0166] Output: The model after initial training

[0167] Step 2: Take a photo of the refrigerator

[0168] The device (refrigerator camera) automatically takes pictures of the inside of the refrigerator at 6:00 every morning.

[0169] What it does: The camera's timer will activate and take images of each shelf inside the refrigerator.

[0170] Input: Real-time video of the inside of the refrigerator

[0171] Output: Captured image data

[0172] The terminal transmits the captured image data to the server.

[0173] Specific operation: Compresses image data and sends it to the server via an HTTP POST request.

[0174] Input: Captured image

[0175] Output: Image data sent to the server

[0176] Step 3: Sending and analyzing image data

[0177] The server analyzes the received image data.

[0178] Input: Image data sent from the device

[0179] Specific operation: The server uses Python's OpenCV library to apply image recognition algorithms.

[0180] Data processing: Perform object detection from image data to identify inventory.

[0181] Output: Inventory information as analysis result

[0182] The server stores the analysis results in a database and updates inventory data.

[0183] Input: Parsed inventory information

[0184] Data operation: Update the inventory table in the database.

[0185] Output: Updated database

[0186] Step 4: Create a shopping list

[0187] The server generates a shopping list based on the user's consumption patterns and the latest inventory information.

[0188] Input: Latest database information

[0189] Data Computing: Using machine learning models to predict food shortages.

[0190] Specific operation: Formalize the model's prediction results as a shopping list.

[0191] Output: Generated shopping list

[0192] Step 5: Shopping List and Recipe Notifications

[0193] The server notifies the user's terminal of the generated shopping list and recommended recipes.

[0194] Input: Shopping list, recommended recipes

[0195] Specific operation: Sends a push notification from the server and sends data to the API endpoint.

[0196] Output: Notification data sent to the user's device

[0197] The device notifies the user through a dedicated app and displays shopping lists and recipes.

[0198] Input: Notification data from the server

[0199] What it does: Your device will display a push notification and make your lists and recipes available in the app.

[0200] Output: User confirmation

[0201] Step 6: User Shopping and Cooking

[0202] Users open the app, check their shopping list and purchase the ingredients they need.

[0203] Action: Open the app, browse the list and make a purchase.

[0204] Input: Shopping List

[0205] Output: Purchased ingredients

[0206] Users cook using recipes suggested by the app based on the ingredients they have purchased and the inventory they have in stock.

[0207] Specific actions: Cook while looking at the recipe displayed on the app.

[0208] Input: Recommended Recipe

[0209] Output: Cooked food

[0210] (Application example 1)

[0211] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0212] Conventional refrigerator inventory management and food consumption management systems required users to check inventory and manually create shopping lists, which was time-consuming and often led to unnecessary food purchases or shortages. Furthermore, because there was no integration with delivery services, users had to go through the trouble of ordering separately. Furthermore, there was a lack of a mechanism for efficiently learning users' food consumption patterns, making it difficult to suggest appropriate recipes.

[0213] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0214] In this invention, the server includes an input means for inputting the user's ingredient consumption habits and cooking preferences, a photographing means for automatically taking periodic images of the inside of the refrigerator, and a means for automatically placing orders with an affiliated food delivery service. This automates refrigerator inventory management, reducing the user's workload and preventing shortages and unnecessary purchases of ingredients. Furthermore, the server automatically connects with the delivery service, allowing the user to quickly order the necessary ingredients. Furthermore, the server facilitates appropriate recipe suggestions based on the user's ingredient consumption patterns, enhancing user convenience.

[0215] A "user" is an individual or entity that uses the system to input their food consumption habits and cooking preferences.

[0216] "Input means" refers to a device or software that allows a user to input their food consumption habits and cooking preferences.

[0217] A "means for storing in a database" is a process and system that receives information from an input means and stores the information in a database.

[0218] The "photography means" refers to a camera and its control device that automatically captures images of the inside of the refrigerator on a regular basis.

[0219] The "analysis means" refers to hardware and software for receiving and analyzing image data obtained by the imaging means.

[0220] The "means for generating a shopping list" is a mechanism for creating a shopping list based on the analyzed inventory information and the stored consumption habits.

[0221] The "notification means" is a mechanism for transmitting the generated shopping list to the user terminal and informing the user.

[0222] The "means for suggesting recipes" is a system for suggesting optimal cooking recipes to a user based on a shopping list.

[0223] "Means for automatically placing orders with affiliated food delivery services" refers to a system that automatically sends an order to an affiliated food delivery service when necessary ingredients are in short supply.

[0224] This invention relates to a system that learns a user's food consumption habits and refrigerator inventory status, proposes optimal shopping lists and recipes, and even automatically orders the necessary ingredients. Specific embodiments for carrying out this invention are described below.

[0225] System configuration

[0226] The system includes the following components:

[0227] 1. Input Method

[0228] Users use a dedicated application to input their food consumption habits, cooking preferences, and allergy information.

[0229] 2. How to store data in a database

[0230] The terminal sends the input data from the user to the server.

[0231] The server stores the received data in a database, recording each user's detailed food consumption habits.

[0232] 3. Filming Method

[0233] A camera installed inside the refrigerator periodically takes pictures of the inventory inside the refrigerator.

[0234] 4. Analysis method

[0235] The terminal transmits the captured image data to the server.

[0236] The server uses image recognition algorithms to analyze the image data and identify the inventory status of the refrigerator, and the analysis results are stored in a database.

[0237] 5. How to generate a shopping list

[0238] The server predicts the ingredients needed based on the user's food consumption patterns and the latest refrigerator inventory data, and generates a shopping list.

[0239] 6. Means of Notification

[0240] The server sends the generated shopping list and suggested recipes to the user's device, where a dedicated application notifies and displays them.

[0241] 7. Proposed measures

[0242] The server selects and suggests appropriate recipes to users based on their consumption habits and inventory status.

[0243] 8. How to Order

[0244] The server automatically places orders with affiliated food delivery services.

[0245] Program processing

[0246] Next, we explain in natural language the program processing required to implement the system. The main hardware used is a camera installed inside the refrigerator, the user's smartphone, and a server. The software used is Python, OpenCV (image recognition library), Django (web framework), and TensorFlow (machine learning model).

[0247] Onboarding a User

[0248] Users input their food consumption habits, cooking preferences, and allergy information through a dedicated application. The device sends this data to the server, which receives it and stores it in a database. The server then initializes a machine learning model to learn the user's food consumption patterns.

[0249] Checking the inventory in the refrigerator

[0250] A camera installed inside the refrigerator automatically takes pictures of the inside of the refrigerator at 6:00 every morning. The device sends the captured image data to a server, which then analyzes the image data using an image recognition algorithm (OpenCV). The inventory status inside the refrigerator is identified and the results are updated in the database.

[0251] Shopping list generation and notifications

[0252] The server predicts what ingredients a user will need before dinner based on their food consumption patterns and the latest inventory status, and generates a shopping list. The generated shopping list and recommended recipes are sent to the user's device, and a dedicated app notifies the user.

[0253] Automated ordering and delivery

[0254] If a specific ingredient is in short supply, the server automatically places an order with a partner food delivery service, calls an API, and notifies the user of the order confirmation and delivery status.

[0255] Examples of concrete examples and prompts

[0256] For example, suppose a user initially registers their weekly consumption of milk and eggs. If the inventory in the refrigerator drops to 1 liter of milk and 2 eggs, the system will generate a shopping list for "1 liter of milk and 6 eggs," place an automatic order, and deliver the ingredients by 10:00 a.m. the next day. It will also suggest "omelette and French toast" as recipe suggestions.

[0257] Example prompt for a generative AI model:

[0258] You will help design an application that automatically orders ingredients and suggests the best grocery shopping list and recipes based on the user's food consumption habits and refrigerator inventory. Specifically, your functional requirements will be as follows:

[0259] 1. Initial User Registration

[0260] 2. Refrigerator inventory management and regular checks

[0261] 3. Learning food consumption patterns

[0262] 4. Optimal recipe suggestions

[0263] 5. Automated ordering and food delivery integration

[0264] Thank you very much for your cooperation.

[0265] In this way, the system of the present invention eliminates the need for users to perform cumbersome inventory checks or manually order ingredients, realizing convenient and efficient ingredient management.

[0266] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0267] Step 1:

[0268] The user inputs their food consumption habits and cooking preferences.

[0269] Specific operation: The user uses a dedicated application to input information such as food consumption habits, cooking preferences, and allergy information.

[0270] Input: User's food consumption habits, cooking preferences, allergy information, etc.

[0271] Output: User data stored in a dedicated application.

[0272] Step 2:

[0273] The terminal transmits the input data to the server.

[0274] Specific operation: The application sends the user's input data to the cloud server.

[0275] Input: User's food consumption habits, cooking preferences, allergy information, etc.

[0276] Output: User data stored on the server.

[0277] Step 3:

[0278] The server stores the received data in a database.

[0279] Specific operation: The server stores the received data in a database management system (e.g., MySQL).

[0280] Input: User's food consumption habits, cooking preferences, allergy information, etc.

[0281] Output: Detailed user data stored in a database.

[0282] Step 4:

[0283] A camera installed inside the refrigerator periodically takes pictures of the inventory inside the refrigerator.

[0284] Specific operation: The camera inside the refrigerator automatically takes a photo of the inside of the refrigerator at 6:00 every morning.

[0285] Input: Current state inside the refrigerator.

[0286] Output: Image data of the inside of the refrigerator.

[0287] Step 5:

[0288] The terminal transmits the captured image data to the server.

[0289] Specific operation: The camera inside the refrigerator or its control device sends the captured image data to a cloud server.

[0290] Input: Image data of the inside of the refrigerator.

[0291] Output: Image data stored on the server.

[0292] Step 6:

[0293] The server analyzes the image data using an image recognition algorithm.

[0294] Specific operation: The server uses an image recognition library such as OpenCV to analyze image data from inside the refrigerator and determine the inventory status.

[0295] Input: Image data of the inside of the refrigerator.

[0296] Output: Parsed inventory information.

[0297] Step 7:

[0298] The server stores and updates the analysis results in a database.

[0299] Specific operation: The server saves the parsed inventory information in the database and updates the existing inventory information.

[0300] Input: Parsed inventory information.

[0301] Output: The updated database.

[0302] Step 8:

[0303] The server predicts the ingredients needed and generates a shopping list based on the user's food consumption patterns and the latest refrigerator inventory data.

[0304] Specific operation: The server learns the user's past consumption patterns and compares them with current inventory data to generate a shopping list.

[0305] Input: User's food consumption patterns, latest refrigerator inventory data.

[0306] Output: Shopping list.

[0307] Step 9:

[0308] The server sends the generated shopping list and recommended recipes to the user's terminal.

[0309] Specific operation: The server sends the generated shopping list and recipes to a dedicated app and notifies it.

[0310] Input: shopping list, recommended recipes.

[0311] Output: The shopping list and recipes displayed on the user's device.

[0312] Step 10:

[0313] If the server runs out of a particular ingredient, it will automatically place an order with an affiliated food delivery service.

[0314] Specific operation: If the server is short on necessary ingredients, it calls the API of the partner delivery service and automatically places the order.

[0315] Input: Missing ingredient information.

[0316] Output: The order sent to the food delivery service.

[0317] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0318] The present invention relates to a system that learns a user's food consumption habits and refrigerator inventory status, and proposes optimal shopping lists and recipes, and further includes a function to recognize the user's emotions and adjust the system's operation based on the recognition results. Specific embodiments for implementing the system of the present invention are described below.

[0319] System configuration

[0320] The system mainly includes the following components:

[0321] 1. Input Method

[0322] Using a dedicated application, users input their food consumption habits, cooking preferences, allergy information, emotional state, etc. This information is used as the initial data for the system.

[0323] 2. How to store data in a database

[0324] The terminal transmits the data entered by the user to the server.

[0325] The server stores the received data in a database, which records detailed food consumption habits and emotional information for each user.

[0326] 3. Filming Method

[0327] The device (refrigerator camera) periodically takes photos of the inventory status inside the refrigerator. For example, photos are taken automatically at set times, such as every morning or after meals.

[0328] 4. Means of receiving and analyzing image data

[0329] The terminal transmits the captured image data to the server.

[0330] The server analyzes the received image data using an image recognition algorithm to identify the inventory status within the refrigerator, and the analysis results are stored in a database.

[0331] 5. Emotion Engine

[0332] The server includes an emotion engine that recognizes the user's emotional state through user input and interaction. The emotion engine analyzes emotions from text analysis, speech recognition, and user facial expressions.

[0333] 6. How to generate a shopping list

[0334] The server predicts the ingredients needed and generates a shopping list based on the user's food consumption patterns, the latest refrigerator inventory data, and their emotional state.

[0335] 7. Means of notifying the user terminal

[0336] The server sends the generated shopping list and suggested recipes to the user's device, where a dedicated application notifies and displays them.

[0337] 8. A way to suggest the best recipes

[0338] The server selects and suggests appropriate recipes to the user based on the user's food consumption habits, current refrigerator inventory, and emotional information.

[0339] Program processing

[0340] Next, the processing of the system program will be explained in natural language.

[0341] 1. User Initial Registration

[0342] Users enter their food consumption habits, cooking preferences, allergy information, and emotional state through a dedicated application.

[0343] The terminal transmits these input data to the server.

[0344] The server receives the data and stores it in a database, which is managed for each user and used as the initial data for the machine learning model and emotion engine.

[0345] 2. Check the refrigerator inventory

[0346] The device (refrigerator camera) automatically takes pictures of the inside of the refrigerator at a set time every morning.

[0347] The terminal transmits the captured image data to the server.

[0348] The server uses image recognition algorithms to analyze the image data and identify the inventory status of the refrigerator, and the analysis results are stored in a database.

[0349] 3. Emotion Recognition and Data Update

[0350] When a user uses the dedicated app, the emotion engine recognizes emotions from the user's input, voice, and facial expressions.

[0351] The server stores the emotional information analyzed by the emotion engine in a database and reflects it in the user's consumption patterns and recipe selection.

[0352] 4. Shopping list generation and notifications

[0353] The server predicts the ingredients needed based on the user's food consumption patterns, the latest inventory status, and emotional information, and generates a shopping list at a set time in the evening.

[0354] The server selects optimal recipes based on the user's preferences and emotional state along with the generated shopping list and sends them to the user's device.

[0355] The device displays notifications to the user through a dedicated app, allowing the user to check their shopping list and recipes.

[0356] 5. User shopping and cooking

[0357] Users can purchase the ingredients they need at the supermarket based on their shopping list. When making a purchase, the system also suggests ingredients and recipes that will boost their mood based on the "recommended mood" provided by the emotion engine.

[0358] The app suggests recipes based on the ingredients purchased and the inventory in the refrigerator, allowing users to cook. For example, they can make herbal tea to relax or a nutritious meal to energize.

[0359] The system of the present invention allows users to avoid forgetting to buy ingredients and avoiding waste, allowing them to shop and cook efficiently. It also suggests optimal recipes and ingredients based on the user's emotional state, improving the quality and satisfaction of meals.

[0360] The processing flow will be explained below.

[0361] Step 1:

[0362] Users install a dedicated application and input their food consumption habits, cooking preferences, allergy information, and emotional state, such as the ingredients they use each week, the dishes they make on specific days, and their current mood (e.g., energetic, tired, or wanting to relax).

[0363] Step 2:

[0364] The device sends the user's input data to the server, including the frequency of food consumption, type of cuisine, a list of ingredients that may cause allergies, and the user's emotional information.

[0365] Step 3:

[0366] The server stores the received data in a database, where it manages detailed information about each user's food consumption habits and emotions.

[0367] Step 4:

[0368] The server uses the saved data to initialize the machine learning model and emotion engine, learning the user's food consumption patterns and emotional patterns. For example, the model can register the user's habit of eating eggs every Monday and their emotional reactions to certain ingredients.

[0369] Step 5:

[0370] The device (refrigerator camera) automatically takes a photo of the inside of the refrigerator every morning at a set time or at a time specified by the user.

[0371] Step 6:

[0372] The device sends the captured image data to the server. This image must show the entire interior of the refrigerator.

[0373] Step 7:

[0374] The server then uses an image recognition algorithm to analyze the received image data and identify the types and quantities of food items in the refrigerator, such as whether there is one liter of milk left or two eggs.

[0375] Step 8:

[0376] The server updates the analysis results and stores them in a database, which is updated in real time.

[0377] Step 9:

[0378] The server generates a shopping list for a specific time in the evening based on the user's food consumption patterns, latest inventory information, and emotional information. For example, one liter of milk and six eggs are added to the list.

[0379] Step 10:

[0380] The server then generates a shopping list and selects the optimal recipe based on the user's preferences and emotional state. For example, if the user is looking to relax, it will suggest recipes using herbal teas and ingredients with relaxing effects.

[0381] Step 11:

[0382] The server sends the shopping list and recipes to the user's device.

[0383] Step 12:

[0384] The device will display a notification to the user through a dedicated application, and the user can open the app to view the displayed shopping list and recipes.

[0385] Step 13:

[0386] Users can purchase the ingredients they need at the supermarket based on their shopping list. At the time of purchase, the emotion engine also suggests ingredients and recipes that will boost their mood based on the "recommended mood" provided by the emotion engine.

[0387] Step 14:

[0388] The app suggests recipes based on the ingredients purchased and the stock in the refrigerator, allowing users to cook. For example, if a user is tired, they can choose a dish that will restore stamina or a dish that is easy to make.

[0389] In this way, this system incorporating an emotion engine helps users avoid forgetting to buy ingredients and avoiding waste, allowing them to shop and cook efficiently. Furthermore, since suggestions are made that take into account the user's emotional state, it can improve the quality and satisfaction of meals.

[0390] Example 2

[0391] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0392] In today's world, many people lead busy lives, and food waste due to forgetting to buy ingredients or expiry dates is a major problem. It is also not easy to find optimal recipes based on one's own food consumption habits and preferences. Furthermore, there are few systems that provide meal suggestions that take into account daily emotional states, and there is a need for systems that can help improve the quality and satisfaction of meals.

[0393] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0394] In this invention, the server includes: means for a user to input information about food ingredient consumption habits, cooking preferences, allergy information, and emotional state; means for receiving information from the input means using a data transmission means and storing it in a database; means for automatically taking periodic images of the inside of the refrigerator; means for receiving image data obtained by the image taking means and analyzing it using an image recognition algorithm; means for generating a shopping list based on the analyzed inventory information and the consumption habits and emotional information stored in the database; means for notifying a user terminal of the generated shopping list and optimal recipes; and means for analyzing the user's emotional state using an emotion engine and reflecting the analysis results in system operation. This allows the user to use ingredients efficiently and improve the quality and satisfaction of their meals by receiving optimal recipes based on their emotional state.

[0395] "Food consumption habits" is information about how often and what types of food ingredients a user consumes within a certain period of time.

[0396] "Cooking preferences" is information relating to the types of dishes, characteristics of ingredients, and cooking methods that the user particularly likes.

[0397] "Allergy information" is information about ingredients or substances to which the user is allergic.

[0398] An "emotional state" is information about the sensations or moods a user is experiencing at a particular point in time.

[0399] "Data transmission means" is a communication means for transmitting input information to a server.

[0400] "Database" refers to a data storage system for storing a user's food consumption habits, cooking preferences, allergy information, and emotional state.

[0401] An "image recognition algorithm" is a computer program that analyzes captured image data and identifies objects and characters contained in the image.

[0402] An "emotion engine" is a system for analyzing a user's emotional state from input text, voice, facial expressions, etc.

[0403] A "shopping list" is a list of ingredients to be purchased that is generated based on the current inventory in the refrigerator, the user's ingredient consumption habits, and their emotional state.

[0404] A "recipe" is a description of the steps and ingredients needed to make a dish.

[0405] A "user terminal" is an electronic device that allows a user to access the system and input or receive information.

[0406] The "means for automatically taking images of the inside of the refrigerator" is a system that periodically takes photos of the inventory status inside the refrigerator using a camera.

[0407] A "machine learning algorithm" is a technology that learns a user's food consumption habits and emotional information and predicts future consumption patterns.

[0408] "Analysis results" are analytical data obtained by the server using image recognition algorithms and emotion engines.

[0409] The present invention relates to a system that learns a user's food consumption habits and refrigerator inventory status, and proposes optimal shopping lists and recipes, and further includes a function that recognizes the user's emotions and adjusts the system's operation based on the recognition results. Specific embodiments for carrying out the present invention will be described below.

[0410] The system is composed of hardware and software such as a dedicated application, an in-fridge camera, an image recognition algorithm, a machine learning algorithm, an emotion engine, a database, a server, and a user terminal.

[0411] First, the user installs a dedicated application and inputs initial information such as their food consumption habits, cooking preferences, allergy information, and emotional state. This information is sent from the user's terminal to the server using a data transmission means, and the server stores the received information in a database. For example, users might enter information such as "I eat chicken once a week," "I like Italian food," "I have a nut allergy," and "I want to relax."

[0412] To understand the inventory status inside the refrigerator, the device (in-refrigerator camera) automatically takes an image of the inside of the refrigerator at a set time every morning. The captured image data is sent from the device to a server. The server analyzes the image data using an image recognition algorithm such as Google (registered trademark) Cloud Vision API to determine the inventory status inside the refrigerator. For example, it may determine that there are three tomatoes, one bag of pasta, and two pieces of cheese in the refrigerator. The analysis results are stored in a database.

[0413] When a user uses the dedicated app, the emotion engine analyzes the user's input text, voice, and facial expressions in real time. For example, if a user inputs "I'm not feeling well," the text analysis identifies that emotion. The server stores the emotion engine's analysis results in a database and reflects them in the user's consumption patterns and recipe selection.

[0414] The server predicts the ingredients needed based on the user's food consumption patterns, the inventory status in the refrigerator, and emotional information. For example, if the user inputs "I use chicken twice a week" but there is no chicken in the refrigerator, the server will add more chicken. The server generates a shopping list at a set time in the evening. Along with the generated shopping list, it selects the optimal recipe based on the user's preferences and emotional state. For example, if the user inputs "I want to relax," a recipe for herbal tea with a relaxing effect will be selected. The server sends the shopping list and recipe to the user's device and displays a notification to the user through a dedicated app.

[0415] Based on the shopping list, the user purchases the ingredients they need at the supermarket. When making a purchase, the app also suggests ingredients and recipes that will boost their mood based on the "recommended mood" provided by the emotion engine. For example, it might suggest, "I feel like relaxing today, so why not try some cardamom tea?" The user then cooks a meal using recipes suggested by the app based on the ingredients they purchased and the inventory in their refrigerator. For example, they could make chicken and basil pasta using the tomatoes, pasta, and cheese in their refrigerator.

[0416] This system allows users to shop and cook efficiently, preventing them from forgetting to buy ingredients or wasting them. It also suggests optimal recipes and ingredients based on the user's emotional state, improving the quality and satisfaction of meals.

[0417] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0418] Step 1: Onboarding users

[0419] The user launches a dedicated application and inputs initial information such as food consumption habits, cooking preferences, allergy information, and emotional state.

[0420] For example, you can enter information such as "I eat chicken once a week," "I like Italian food," "I have a nut allergy," or "I want to relax."

[0421] Input data: food consumption habits, cooking preferences, allergy information, emotional state

[0422] The terminal transmits the input data to the server.

[0423] Data transmission: From user device to server

[0424] The server stores the received data in a database and manages it for each user.

[0425] Data storage: The server stores data in a database

[0426] Step 2: Check the refrigerator inventory

[0427] The device (refrigerator camera) automatically takes pictures of the inside of the refrigerator at a set time every morning.

[0428] For example, capture an image of the inside of your refrigerator at 8am.

[0429] Input data: Image data of the inside of the refrigerator

[0430] The terminal transmits the captured image data to the server.

[0431] Data transmission: From the camera inside the refrigerator to the server

[0432] The server uses image recognition algorithms to analyze the image data and determine the inventory status within the refrigerator.

[0433] Data analysis: Using image recognition algorithms such as Google Cloud Vision API

[0434] For example: "Identify that there are 3 tomatoes, 1 bag of pasta, and 2 cheeses in the refrigerator."

[0435] Analysis result: Inventory information (3 tomatoes, 1 bag of pasta, 2 cheeses)

[0436] The server stores the analysis results in a database.

[0437] Data storage: The server stores data in a database

[0438] Step 3: Emotion recognition and data update

[0439] When a user uses the dedicated app, the emotion engine analyzes the user's input text, voice, and facial expressions in real time.

[0440] For example, if a user types "I'm feeling down," text analysis will identify that emotion.

[0441] Input data: text, voice, facial expression data

[0442] The server obtains the analysis results of the emotion engine.

[0443] Data analysis: Emotion engine uses text analysis, speech recognition, and facial expression recognition

[0444] Example: "The user wants to relax."

[0445] Analysis results: Emotional state information

[0446] The server stores the analyzed emotional information in a database and reflects it in the user's consumption patterns and recipe selection.

[0447] Data storage: The server stores data in a database

[0448] Step 4: Create a shopping list

[0449] The server predicts the ingredients needed based on the user's food consumption patterns, the inventory status in the refrigerator, and emotional information.

[0450] For example, if a user inputs "I use chicken twice a week" but there is no chicken in the refrigerator, new chicken will be added.

[0451] Input data: food consumption patterns, refrigerator inventory, emotional information

[0452] The server generates a shopping list at a fixed time in the evening.

[0453] Data generation: Shopping list (e.g. chicken, basil)

[0454] Generated data: Shopping list (chicken, basil)

[0455] Step 5: Recipe suggestions and notifications

[0456] The server generates a shopping list and selects the best recipe based on the user's preferences and emotional state.

[0457] For example, if the user types "I want to relax," a recipe for a relaxing herbal tea will be selected.

[0458] Input data: shopping list, food consumption habits, emotional information

[0459] The server sends the shopping list and recipes to the user's device.

[0460] Data transmission: From the server to the user's device

[0461] The device will display a notification to the user through a dedicated app.

[0462] Notifications: Shopping Lists and Recipes

[0463] For example: "Your shopping list has been updated. Please purchase chicken and basil. My recommended recipe is chicken and basil pasta."

[0464] Step 6: User Shopping and Cooking

[0465] The user purchases the necessary ingredients at the supermarket based on the shopping list.

[0466] For example, buy chicken and basil according to the list.

[0467] The emotional engine suggests ingredients and recipes that will boost your mood based on your "recommended mood."

[0468] For example, suggest, "I'm feeling relaxed today, how about trying some cardamom tea?"

[0469] Users cook using recipes suggested by the app based on the ingredients they have purchased and what is in their refrigerator.

[0470] For example, make chicken and basil pasta using the tomatoes, pasta, and cheese in your fridge.

[0471] Through these steps, the system prevents users from forgetting to buy ingredients or wasting them, enabling them to shop and cook efficiently. It also improves the quality and satisfaction of meals by suggesting optimal recipes and ingredients based on the user's emotional state.

[0472] (Application example 2)

[0473] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0474] In today's busy daily lives, it is extremely difficult for users to effectively manage their food consumption habits and refrigerator inventory. In addition, there is a need for systems that provide optimal shopping lists and recipes while taking into account the user's emotional state while preparing meals. However, conventional systems are unable to incorporate information, including the user's emotional state, making it difficult to prevent unnecessary food purchases and food waste. Furthermore, the virtual store's purchasing experience is often insufficient, causing user stress. A system that solves these problems is needed.

[0475] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0476] In this invention, the server includes a means for allowing a user to input information about their food ingredient consumption habits, cooking preferences, allergy information, and emotional state; a means for automatically taking periodic images of the inside of the refrigerator; a means for receiving and analyzing the image data obtained by the image taking means; a means for analyzing emotions from the user's input, voice, and facial expressions using an emotion recognition engine; and a means for supporting the shopping experience in the virtual store. This allows the user to significantly reduce food waste and shop and cook efficiently. Furthermore, the server can suggest ingredients and provide recipes based on the user's emotional state, further improving the shopping experience in the virtual store.

[0477] Definitions of important words

[0478] "User" refers to an individual who uses the system to manage ingredients and receive recipe suggestions.

[0479] "Food consumption habits" refers to the types, amounts, and frequency of food ingredients that a user consumes on a daily basis.

[0480] "Cooking preferences" refers to the types and seasonings of food that a user likes.

[0481] "Allergy information" refers to foods or ingredients to which the user is allergic.

[0482] "Emotional state" refers to the user's emotional condition, such as stress or relaxation.

[0483] "Input means" refers to an interface through which a user inputs information into a system.

[0484] "Photographing means" refers to a device or function that automatically photographs images inside the refrigerator.

[0485] "Image data" refers to a photograph of the inside of the refrigerator taken by a photographing means.

[0486] "Analysis means" refers to a device or function that includes an algorithm for analyzing image data and identifying the inventory status and ingredients in the refrigerator.

[0487] An "emotion recognition engine" refers to technology or software for analyzing emotions from user input, voice, and facial expressions.

[0488] A "shopping list" is a list of all the ingredients you need.

[0489] "User terminal" refers to a device through which a user interfaces with the system.

[0490] "Notification means" refers to a technique or device for notifying the user of the generated shopping list or recipe.

[0491] "Recipe suggestion means" refers to a device or function that suggests optimal recipes based on a shopping list.

[0492] A "virtual store" refers to a virtual shopping environment for purchasing food ingredients over the Internet.

[0493] "Supporting Means" refers to technology or devices that support the virtual store shopping experience.

[0494] The present invention relates to a system that learns a user's food consumption habits and refrigerator inventory status, recognizes the user's emotions, and proposes optimal shopping lists and recipes. Specific embodiments for implementing the system of the present invention are described below.

[0495] System configuration

[0496] The system mainly uses the following hardware and software:

[0497] Hardware

[0498] 1. Smartphone: A device where users input information.

[0499] 2. Refrigerator camera: A device that periodically takes pictures of the inside of the refrigerator.

[0500] 3. Server: A computer that stores, analyzes, and notifies data.

[0501] software

[0502] 1. TensorFlow: Machine learning framework.

[0503] 2. OpenCV: Image recognition library.

[0504] 3. Google Cloud Vision API: Image analysis solution.

[0505] 4. Azure® Cognitive Services: Sentiment Analysis Service.

[0506] Specific processing of the program

[0507] The program of this system operates in the following manner.

[0508] 1. User Initial Registration

[0509] Users input their food consumption habits, cooking preferences, allergy information, and emotional state through a dedicated smartphone app.

[0510] The terminal sends this information to the server, which stores it in a database.

[0511] 2. Check the refrigerator inventory

[0512] The camera inside the refrigerator periodically takes pictures of the inside of the refrigerator.

[0513] The device sends the captured image data to a server, which then analyzes the image data using the Google Cloud Vision API and OpenCV to determine inventory status.

[0514] The analysis results are stored in a database.

[0515] 3. Emotion Recognition and Data Update

[0516] When a user uses the app, the emotion recognition engine analyzes emotions from the user's input, voice, and facial expressions. Emotion analysis is performed using Azure Cognitive Services.

[0517] The server stores the results of the sentiment analysis in a database and reflects them in food consumption patterns and recipe selection.

[0518] 4. Shopping list generation and notifications

[0519] The server generates a shopping list based on the user's consumption patterns, latest inventory status, and emotional information.

[0520] After generating the list, the most suitable recipe is selected and notified to the user's smartphone.

[0521] Users can access shopping lists and recipes through the app.

[0522] 5. Virtual store shopping support

[0523] Users can purchase ingredients suggested within the app at a virtual store.

[0524] The server provides a recommendation system to support the shopping experience in the virtual store.

[0525] Specific examples

[0526] For example, if a user types "I'm stressed today," the emotion analysis engine will determine the emotion "stress" and suggest herbal teas or relaxing dishes that will help reduce stress.

[0527] Example prompt sentence:

[0528] Write a Python program that recognizes emotions from text input and suggests ingredients and recipes based on them. Use the following libraries: TensorFlow, OpenCV, Google Cloud Vision API, and Azure Cognitive Services. If the user types "I'm tired today," suggest a corresponding recipe for a relaxing dish, such as chamomile tea or soup.

[0529] This system allows users to efficiently manage ingredients and purchase the ingredients they need without waste. It also suggests optimal recipes based on the user's emotional state, improving the quality and satisfaction of meals.

[0530] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0531] Program processing steps

[0532] Step 1:

[0533] Onboarding a User

[0534] Users launch the smartphone app and enter their food consumption habits, cooking preferences, allergy information, and emotional state.

[0535] Input: User's food consumption habits, cooking preferences, allergy information, and emotional state.

[0536] Data processing: Format user input information in JSON format.

[0537] The terminal transmits the input information to the server.

[0538] The server stores this data in a database.

[0539] Output: User information stored in the database.

[0540] Step 2:

[0541] Checking the inventory in the refrigerator

[0542] The camera inside the refrigerator automatically takes pictures of the inside of the refrigerator at a set time (e.g., every morning).

[0543] Input: Image data of the inside of the refrigerator.

[0544] Data processing: Image data is sent to the server in JPEG format.

[0545] The terminal transmits the captured image data to the server.

[0546] The server uses Google Cloud Vision API and OpenCV to analyze the image data and determine the inventory status in the refrigerator.

[0547] Data Computing: Image recognition algorithms are used to identify ingredients and generate inventory lists.

[0548] The analysis results are stored in a database.

[0549] Output: Refrigerator inventory information stored in a database.

[0550] Step 3:

[0551] Emotion Recognition and Data Update

[0552] While using a smartphone app, a user inputs their emotional state by typing or speaking.

[0553] Input: Text and / or audio data of the user's emotional state.

[0554] Data processing: Converting voice data into text and extracting sentiment keywords.

[0555] The terminal transmits the emotion data to the server.

[0556] The server uses Azure Cognitive Services to perform sentiment analysis and identify the user's emotional state.

[0557] Data Computing: Emotion recognition algorithms determine and store emotional states.

[0558] Output: User's emotional state information stored in a database.

[0559] Step 4:

[0560] Shopping list generation and notifications

[0561] The server generates a shopping list based on stored consumption patterns, up-to-date inventory status, and emotional information.

[0562] Input: User consumption habits, refrigerator inventory, and emotional information.

[0563] Data Computing: Predicting ingredient needs using machine learning algorithms.

[0564] The generated shopping list and the optimal recipe are selected and sent to the user's device.

[0565] Output: Shopping list and suggested recipes sent to the user's device.

[0566] Step 5:

[0567] Virtual store shopping support

[0568] Users purchase ingredients at a virtual store based on a shopping list presented within the app.

[0569] Input: Shopping list and virtual store catalog information.

[0570] Data processing: Add selected ingredients to cart.

[0571] The server uses a recommendation system to support the virtual store shopping experience.

[0572] Output: Order completed in the virtual store and purchase notification sent to the user.

[0573] This system allows users to efficiently manage ingredients and purchase the ingredients they need without waste. It also suggests optimal recipes based on the user's emotional state, improving the quality and satisfaction of meals.

[0574] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0575] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0576] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0577] [Second embodiment]

[0578] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0579] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0580] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0581] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0582] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0583] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0584] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0585] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0586] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0587] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0588] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0589] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0590] The present invention relates to a system that learns a user's food consumption habits and refrigerator inventory status and proposes optimal shopping lists and recipes. Specific embodiments for implementing the system of the present invention will be described below.

[0591] System configuration

[0592] The system mainly includes the following components:

[0593] 1. Input Method

[0594] Using a dedicated application, users input their food consumption habits, cooking preferences, allergy information, etc. This information is used as the initial data for the system.

[0595] 2. How to store data in a database

[0596] The terminal transmits the data entered by the user to the server.

[0597] The server stores the received data in a database, which records detailed food consumption habits for each user.

[0598] 3. Filming Method

[0599] The device (refrigerator camera) periodically takes photos of the inventory status inside the refrigerator. For example, photos are taken automatically at set times, such as every morning or after meals.

[0600] 4. Means of receiving and analyzing image data

[0601] The terminal transmits the captured image data to the server.

[0602] The server analyzes the received image data using an image recognition algorithm to identify the inventory status within the refrigerator, and the analysis results are stored in a database.

[0603] 5. How to generate a shopping list

[0604] The server predicts the ingredients needed based on the user's food consumption patterns and the latest refrigerator inventory data, and generates a shopping list.

[0605] 6. Means of notifying the user terminal

[0606] The server sends the generated shopping list and suggested recipes to the user's device, where a dedicated application notifies and displays them.

[0607] 7. A way to suggest the best recipes

[0608] The server selects and suggests appropriate recipes to the user based on the user's food consumption habits and current refrigerator inventory.

[0609] Program processing

[0610] Next, the processing of the system program will be explained in natural language.

[0611] 1. User Initial Registration

[0612] Users enter their food consumption habits, cooking preferences, and allergy information through a dedicated application.

[0613] The terminal transmits these input data to the server.

[0614] The server receives the data, stores it in a database, and initializes a machine learning model to learn the user's food consumption patterns.

[0615] 2. Check the refrigerator inventory

[0616] The device (refrigerator camera) automatically takes pictures of the inside of the refrigerator at 6:00 every morning.

[0617] The terminal transmits the captured image data to the server.

[0618] The server uses image recognition algorithms to analyze the image data and determine the inventory status of the refrigerator, and the analysis results are updated in a database.

[0619] 3. Shopping list generation and notifications

[0620] The server predicts the ingredients needed at 6pm based on the user's food consumption patterns and the latest inventory status, and generates a shopping list.

[0621] The server sends the generated shopping list and recommended recipes to the user's device.

[0622] The device notifies the user through a dedicated app and displays shopping lists and recipes.

[0623] 4. User shopping and cooking

[0624] Users open the app, check their shopping list, and purchase the ingredients they need.

[0625] Users cook using recipes suggested by the app based on the ingredients they have purchased and the inventory they have in stock.

[0626] Specific examples

[0627] For example, if a user specifies that they consume milk and eggs every week, the system will learn this. While checking the refrigerator inventory each day, it will determine that there is one liter of milk and two eggs left. When the user goes shopping, the system will automatically generate a shopping list that includes "one liter of milk and six eggs" and notify the app. The system will also suggest recipes such as omelets and French toast based on the ingredients in the refrigerator.

[0628] In this way, the system of the present invention allows users to avoid forgetting to buy ingredients or wasting them, and allows them to shop and cook efficiently.

[0629] The processing flow will be explained below.

[0630] Step 1:

[0631] Users install a dedicated application and enter their food consumption habits, cooking preferences, and allergy information, specifically the ingredients they use each week and the dishes they make on specific days.

[0632] Step 2:

[0633] The device sends the user's input data to the server, including the frequency of food consumption, type of cuisine, and a list of ingredients that may cause allergies.

[0634] Step 3:

[0635] The server stores the received data in a database, where the data is managed separately for each user.

[0636] Step 4:

[0637] The server uses the stored data to initialize a machine learning model and learns the user's food consumption patterns. For example, the model can learn that the user uses eggs every Monday.

[0638] Step 5:

[0639] The device (refrigerator camera) automatically takes a photo of the inside of the refrigerator every morning at 6:00 or at a time specified by the user.

[0640] Step 6:

[0641] The device sends the captured image data to the server. This image must show the entire interior of the refrigerator.

[0642] Step 7:

[0643] The server then uses an image recognition algorithm to analyze the received image data and identify the types and quantities of food items in the refrigerator, such as whether there is one liter of milk left or two eggs.

[0644] Step 8:

[0645] The server updates the analysis results and stores them in a database, which is updated in real time.

[0646] Step 9:

[0647] The server generates a shopping list every day at 6pm based on the user's food consumption patterns and the latest inventory information. For example, one liter of milk and six eggs are added to the list.

[0648] Step 10:

[0649] The server generates a shopping list and selects the best recipes based on the user's preferences, such as omelet and French toast recipes.

[0650] Step 11:

[0651] The server sends the shopping list and recipes to the user's device.

[0652] Step 12:

[0653] The device will display a notification to the user through a dedicated application, and the user can open the app to view the displayed shopping list and recipes.

[0654] Step 13:

[0655] The user goes to the supermarket and buys the ingredients they need based on the shopping list, for example, adding 1 liter of milk and 6 eggs to their cart.

[0656] Step 14:

[0657] The app suggests recipes based on the ingredients purchased and the inventory in the refrigerator, allowing users to cook dishes such as omelets and French toast.

[0658] Example 1

[0659] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0660] Conventional refrigerator management systems require users to manually manage food inventory, which is time-consuming and can lead to forgotten purchases and wasteful consumption. It is also difficult to provide effective shopping lists and recipes that take into account the user's food consumption habits. This leads to inefficiencies in the user's lifestyle and wastes time and energy.

[0661] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0662] In this invention, the server includes input means for inputting a user's ingredient consumption habits and cooking preferences, means for receiving information from the input means and storing the information in a database, photographing means for automatically photographing images of the inside of the refrigerator periodically, means for receiving and analyzing image data obtained by the photographing means, means for generating a shopping list based on the analyzed inventory information and the stored consumption habits, means for notifying a user terminal of the generated shopping list, means for using a machine learning algorithm to predict optimal ingredients based on the suggested shopping list, and means for suggesting recipes based on the optimal ingredients and the user's consumption habits, thereby enabling users to efficiently manage ingredient consumption and inventory without hassle.

[0663] "User" refers to a person who uses the system to input their food consumption habits and cooking preferences and receive suggested shopping lists and recipes.

[0664] "Food consumption habits" refers to data that indicates the tendency and frequency of food ingredients consumed by a user during a specific period.

[0665] "Cooking preferences" refers to information about the user's favorite dishes and cooking methods.

[0666] "Input means" refers to an interface that allows a user to input their food consumption habits and cooking preferences into the system.

[0667] "Database" refers to an information system for storing information received from users, analysis results, inventory information, etc.

[0668] "Photographing means" refers to a device or equipment that automatically takes pictures of the inside of the refrigerator on a regular basis.

[0669] "Image data" refers to photographs and video data of the inside of the refrigerator captured by a photographing means.

[0670] "Analysis means" refers to a system or method that uses an image recognition algorithm to analyze captured image data and identify inventory information within the refrigerator.

[0671] "Inventory information" refers to data about the ingredients currently in the refrigerator and their quantities.

[0672] A "shopping list" is a list created by predicting the ingredients a user will need in the future.

[0673] "Notification means" refers to a system or method for sending the generated shopping list and suggested recipes to a user terminal.

[0674] "Machine learning algorithm" refers to a calculation method or program that learns from a user's food consumption habits and refrigerator inventory information to predict the best ingredients.

[0675] "Recipe suggestion means" refers to a system or method for providing a user with a recipe suitable for cooking based on optimal ingredients and the user's consumption habits.

[0676] The present invention relates to a system that learns a user's food consumption habits and refrigerator inventory status, and proposes optimal shopping lists and recipes. This system is implemented using the following hardware and software.

[0677] 1. Hardware Configuration

[0678] The system includes the following hardware:

[0679] User device: Mobile device such as a smartphone or tablet

[0680] Refrigerator camera: A camera that periodically takes pictures of the inside of a refrigerator

[0681] Server: A computer system that runs a database and image recognition and machine learning algorithms.

[0682] 2. Software Configuration

[0683] The system includes the following software:

[0684] Dedicated application: An app for users to input information about food consumption habits, cooking preferences, and allergies

[0685] Database Management Systems: Databases such as MySQL or PostgreSQL

[0686] Image recognition algorithm: Object detection algorithm using OpenCV and TensorFlow

[0687] Machine learning algorithms: predictive models using scikit-learn and TensorFlow

[0688] 3. System Operation

[0689] Onboarding a User

[0690] Users enter their food consumption habits, cooking preferences, allergy information, etc. through a dedicated application. This information is used as the initial data for the system.

[0691] The terminal transmits the data entered by the user to the server.

[0692] The server stores the received data in a database and initializes and starts training a machine learning model to learn the user's food consumption patterns.

[0693] Checking the inventory in the refrigerator

[0694] The device (refrigerator camera) automatically takes pictures of the inside of the refrigerator at a fixed time every day. This time can be set by the user.

[0695] The terminal transmits the captured image data to the server.

[0696] The server analyzes the image data using an image recognition algorithm to identify the inventory status of the refrigerator, and the analysis results are reflected in a database.

[0697] Shopping list generation and notifications

[0698] The server predicts the ingredients needed based on the user's ingredient consumption patterns and the latest inventory status, and generates a shopping list.

[0699] The server sends the generated shopping list to the user's device, and a dedicated app notifies the user and displays the list to them.

[0700] Recipe Suggestions

[0701] The server selects and suggests appropriate recipes based on the user's consumption habits and refrigerator inventory. Recipe suggestions are made via a dedicated app.

[0702] 4. Specific Examples

[0703] For example, if a user specifies that they "eat eggs and bread for breakfast every day," the system learns this and predicts the ingredients they will need based on their consumption patterns over the course of a week. A camera inside the refrigerator checks the inventory every day and generates a shopping list based on the analyzed results. When the user opens the app on their smartphone, a shopping list of "10 eggs and 1 bag of bread" is displayed. The system also suggests recipes such as "French toast" and "omelette" based on the inventory status in the refrigerator.

[0704] Prompt Sentence Examples

[0705] "Can you recommend a simple and delicious recipe for dinner? I have eggs and milk in the fridge."

[0706] In this way, the system of the present invention allows users to receive efficient support regarding the management and consumption of ingredients, preventing them from forgetting to buy food or wasting it.

[0707] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0708] Step 1: Onboarding users

[0709] Users open a dedicated app and enter data such as food consumption habits, cooking preferences, and allergy information.

[0710] Input: Food consumption habits, cooking preferences, allergy information

[0711] Specific action: The user enters the required information into a form within the app and presses the submit button.

[0712] The terminal receives input data and transmits it to the server.

[0713] Output: Sending user-entered data

[0714] The server stores the received data in a database.

[0715] Input: User's consumption habits, food preferences, and allergy information

[0716] Data processing: Store information in the database in JSON format and organize it by user ID.

[0717] Output: Database update

[0718] The server initializes the machine learning model with the user's consumption patterns and performs initial training.

[0719] Input: Initial database information

[0720] Data Computing: Model training with machine learning algorithms

[0721] Output: The model after initial training

[0722] Step 2: Take a photo of the refrigerator

[0723] The device (refrigerator camera) automatically takes pictures of the inside of the refrigerator at 6:00 every morning.

[0724] What it does: The camera's timer will activate and take images of each shelf inside the refrigerator.

[0725] Input: Real-time video of the inside of the refrigerator

[0726] Output: Captured image data

[0727] The terminal transmits the captured image data to the server.

[0728] Specific operation: Compresses image data and sends it to the server via an HTTP POST request.

[0729] Input: Captured image

[0730] Output: Image data sent to the server

[0731] Step 3: Sending and analyzing image data

[0732] The server analyzes the received image data.

[0733] Input: Image data sent from the device

[0734] Specific operation: The server uses Python's OpenCV library to apply image recognition algorithms.

[0735] Data processing: Perform object detection from image data to identify inventory.

[0736] Output: Inventory information as analysis result

[0737] The server stores the analysis results in a database and updates inventory data.

[0738] Input: Parsed inventory information

[0739] Data operation: Update the inventory table in the database.

[0740] Output: Updated database

[0741] Step 4: Create a shopping list

[0742] The server generates a shopping list based on the user's consumption patterns and the latest inventory information.

[0743] Input: Latest database information

[0744] Data Computing: Using machine learning models to predict food shortages.

[0745] Specific operation: Formalize the model's prediction results as a shopping list.

[0746] Output: Generated shopping list

[0747] Step 5: Shopping List and Recipe Notifications

[0748] The server notifies the user's terminal of the generated shopping list and recommended recipes.

[0749] Input: Shopping list, recommended recipes

[0750] Specific operation: Sends a push notification from the server and sends data to the API endpoint.

[0751] Output: Notification data sent to the user's device

[0752] The device notifies the user through a dedicated app and displays shopping lists and recipes.

[0753] Input: Notification data from the server

[0754] What it does: Your device will display a push notification and make your lists and recipes available in the app.

[0755] Output: User confirmation

[0756] Step 6: User Shopping and Cooking

[0757] Users open the app, check their shopping list and purchase the ingredients they need.

[0758] Action: Open the app, browse the list and make a purchase.

[0759] Input: Shopping List

[0760] Output: Purchased ingredients

[0761] Users cook using recipes suggested by the app based on the ingredients they have purchased and the inventory they have in stock.

[0762] Specific actions: Cook while looking at the recipe displayed on the app.

[0763] Input: Recommended Recipe

[0764] Output: Cooked food

[0765] (Application example 1)

[0766] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0767] Conventional refrigerator inventory management and food consumption management systems required users to check inventory and manually create shopping lists, which was time-consuming and often led to unnecessary food purchases or shortages. Furthermore, because there was no integration with delivery services, users had to go through the trouble of ordering separately. Furthermore, there was a lack of a mechanism for efficiently learning users' food consumption patterns, making it difficult to suggest appropriate recipes.

[0768] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0769] In this invention, the server includes an input means for inputting the user's ingredient consumption habits and cooking preferences, a photographing means for automatically taking periodic images of the inside of the refrigerator, and a means for automatically placing orders with an affiliated food delivery service. This automates refrigerator inventory management, reducing the user's workload and preventing shortages and unnecessary purchases of ingredients. Furthermore, the server automatically connects with the delivery service, allowing the user to quickly order the necessary ingredients. Furthermore, the server facilitates appropriate recipe suggestions based on the user's ingredient consumption patterns, enhancing user convenience.

[0770] A "user" is an individual or entity that uses the system to input their food consumption habits and cooking preferences.

[0771] "Input means" refers to a device or software that allows a user to input their food consumption habits and cooking preferences.

[0772] A "means for storing in a database" is a process and system that receives information from an input means and stores the information in a database.

[0773] The "photography means" refers to a camera and its control device that automatically captures images of the inside of the refrigerator on a regular basis.

[0774] The "analysis means" refers to hardware and software for receiving and analyzing image data obtained by the imaging means.

[0775] The "means for generating a shopping list" is a mechanism for creating a shopping list based on the analyzed inventory information and the stored consumption habits.

[0776] The "notification means" is a mechanism for transmitting the generated shopping list to the user terminal and informing the user.

[0777] The "means for suggesting recipes" is a system for suggesting optimal cooking recipes to a user based on a shopping list.

[0778] "Means for automatically placing orders with affiliated food delivery services" refers to a system that automatically sends an order to an affiliated food delivery service when necessary ingredients are in short supply.

[0779] This invention relates to a system that learns a user's food consumption habits and refrigerator inventory status, proposes optimal shopping lists and recipes, and even automatically orders the necessary ingredients. Specific embodiments for carrying out this invention are described below.

[0780] System configuration

[0781] The system includes the following components:

[0782] 1. Input Method

[0783] Users use a dedicated application to input their food consumption habits, cooking preferences, and allergy information.

[0784] 2. How to store data in a database

[0785] The terminal sends the input data from the user to the server.

[0786] The server stores the received data in a database, recording each user's detailed food consumption habits.

[0787] 3. Filming Method

[0788] A camera installed inside the refrigerator periodically takes pictures of the inventory inside the refrigerator.

[0789] 4. Analysis method

[0790] The terminal transmits the captured image data to the server.

[0791] The server uses image recognition algorithms to analyze the image data and identify the inventory status of the refrigerator, and the analysis results are stored in a database.

[0792] 5. How to generate a shopping list

[0793] The server predicts the ingredients needed based on the user's food consumption patterns and the latest refrigerator inventory data, and generates a shopping list.

[0794] 6. Means of Notification

[0795] The server sends the generated shopping list and suggested recipes to the user's device, where a dedicated application notifies and displays them.

[0796] 7. Proposed measures

[0797] The server selects and suggests appropriate recipes to users based on their consumption habits and inventory status.

[0798] 8. How to Order

[0799] The server automatically places orders with affiliated food delivery services.

[0800] Program processing

[0801] Next, we explain in natural language the program processing required to implement the system. The main hardware used is a camera installed inside the refrigerator, the user's smartphone, and a server. The software used is Python, OpenCV (image recognition library), Django (web framework), and TensorFlow (machine learning model).

[0802] Onboarding a User

[0803] Users input their food consumption habits, cooking preferences, and allergy information through a dedicated application. The device sends this data to the server, which receives it and stores it in a database. The server then initializes a machine learning model to learn the user's food consumption patterns.

[0804] Checking the inventory in the refrigerator

[0805] A camera installed inside the refrigerator automatically takes pictures of the inside of the refrigerator at 6:00 every morning. The device sends the captured image data to a server, which then analyzes the image data using an image recognition algorithm (OpenCV). The inventory status inside the refrigerator is identified and the results are updated in the database.

[0806] Shopping list generation and notifications

[0807] The server predicts what ingredients a user will need before dinner based on their food consumption patterns and the latest inventory status, and generates a shopping list. The generated shopping list and recommended recipes are sent to the user's device, and a dedicated app notifies the user.

[0808] Automated ordering and delivery

[0809] If a specific ingredient is in short supply, the server automatically places an order with a partner food delivery service, calls an API, and notifies the user of the order confirmation and delivery status.

[0810] Examples of concrete examples and prompts

[0811] For example, suppose a user initially registers their weekly consumption of milk and eggs. If the inventory in the refrigerator drops to 1 liter of milk and 2 eggs, the system will generate a shopping list for "1 liter of milk and 6 eggs," place an automatic order, and deliver the ingredients by 10:00 a.m. the next day. It will also suggest "omelette and French toast" as recipe suggestions.

[0812] Example prompt for a generative AI model:

[0813] You will help design an application that automatically orders ingredients and suggests the best grocery shopping list and recipes based on the user's food consumption habits and refrigerator inventory. Specifically, your functional requirements will be as follows:

[0814] 1. Initial User Registration

[0815] 2. Refrigerator inventory management and regular checks

[0816] 3. Learning food consumption patterns

[0817] 4. Optimal recipe suggestions

[0818] 5. Automated ordering and food delivery integration

[0819] Thank you very much for your cooperation.

[0820] In this way, the system of the present invention eliminates the need for users to perform cumbersome inventory checks or manually order ingredients, realizing convenient and efficient ingredient management.

[0821] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0822] Step 1:

[0823] The user inputs their food consumption habits and cooking preferences.

[0824] Specific operation: The user uses a dedicated application to input information such as food consumption habits, cooking preferences, and allergy information.

[0825] Input: User's food consumption habits, cooking preferences, allergy information, etc.

[0826] Output: User data stored in a dedicated application.

[0827] Step 2:

[0828] The terminal transmits the input data to the server.

[0829] Specific operation: The application sends the user's input data to the cloud server.

[0830] Input: User's food consumption habits, cooking preferences, allergy information, etc.

[0831] Output: User data stored on the server.

[0832] Step 3:

[0833] The server stores the received data in a database.

[0834] Specific operation: The server stores the received data in a database management system (e.g., MySQL).

[0835] Input: User's food consumption habits, cooking preferences, allergy information, etc.

[0836] Output: Detailed user data stored in a database.

[0837] Step 4:

[0838] A camera installed inside the refrigerator periodically takes pictures of the inventory inside the refrigerator.

[0839] Specific operation: The camera inside the refrigerator automatically takes a photo of the inside of the refrigerator at 6:00 every morning.

[0840] Input: Current state inside the refrigerator.

[0841] Output: Image data of the inside of the refrigerator.

[0842] Step 5:

[0843] The terminal transmits the captured image data to the server.

[0844] Specific operation: The camera inside the refrigerator or its control device sends the captured image data to a cloud server.

[0845] Input: Image data of the inside of the refrigerator.

[0846] Output: Image data stored on the server.

[0847] Step 6:

[0848] The server analyzes the image data using an image recognition algorithm.

[0849] Specific operation: The server uses an image recognition library such as OpenCV to analyze image data from inside the refrigerator and determine the inventory status.

[0850] Input: Image data of the inside of the refrigerator.

[0851] Output: Parsed inventory information.

[0852] Step 7:

[0853] The server stores and updates the analysis results in a database.

[0854] Specific operation: The server saves the parsed inventory information in the database and updates the existing inventory information.

[0855] Input: Parsed inventory information.

[0856] Output: The updated database.

[0857] Step 8:

[0858] The server predicts the ingredients needed and generates a shopping list based on the user's food consumption patterns and the latest refrigerator inventory data.

[0859] Specific operation: The server learns the user's past consumption patterns and compares them with current inventory data to generate a shopping list.

[0860] Input: User's food consumption patterns, latest refrigerator inventory data.

[0861] Output: Shopping list.

[0862] Step 9:

[0863] The server sends the generated shopping list and recommended recipes to the user's terminal.

[0864] Specific operation: The server sends the generated shopping list and recipes to a dedicated app and notifies it.

[0865] Input: shopping list, recommended recipes.

[0866] Output: The shopping list and recipes displayed on the user's device.

[0867] Step 10:

[0868] If the server runs out of a particular ingredient, it will automatically place an order with an affiliated food delivery service.

[0869] Specific operation: If the server is short on necessary ingredients, it calls the API of the partner delivery service and automatically places the order.

[0870] Input: Missing ingredient information.

[0871] Output: The order sent to the food delivery service.

[0872] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0873] The present invention relates to a system that learns a user's food consumption habits and refrigerator inventory status, and proposes optimal shopping lists and recipes, and further includes a function to recognize the user's emotions and adjust the system's operation based on the recognition results. Specific embodiments for implementing the system of the present invention are described below.

[0874] System configuration

[0875] The system mainly includes the following components:

[0876] 1. Input Method

[0877] Using a dedicated application, users input their food consumption habits, cooking preferences, allergy information, emotional state, etc. This information is used as the initial data for the system.

[0878] 2. How to store data in a database

[0879] The terminal transmits the data entered by the user to the server.

[0880] The server stores the received data in a database, which records detailed food consumption habits and emotional information for each user.

[0881] 3. Filming Method

[0882] The device (refrigerator camera) periodically takes photos of the inventory status inside the refrigerator. For example, photos are taken automatically at set times, such as every morning or after meals.

[0883] 4. Means of receiving and analyzing image data

[0884] The terminal transmits the captured image data to the server.

[0885] The server analyzes the received image data using an image recognition algorithm to identify the inventory status within the refrigerator, and the analysis results are stored in a database.

[0886] 5. Emotion Engine

[0887] The server includes an emotion engine that recognizes the user's emotional state through user input and interaction. The emotion engine analyzes emotions from text analysis, speech recognition, and user facial expressions.

[0888] 6. How to generate a shopping list

[0889] The server predicts the ingredients needed and generates a shopping list based on the user's food consumption patterns, the latest refrigerator inventory data, and their emotional state.

[0890] 7. Means of notifying the user terminal

[0891] The server sends the generated shopping list and suggested recipes to the user's device, where a dedicated application notifies and displays them.

[0892] 8. A way to suggest the best recipes

[0893] The server selects and suggests appropriate recipes to the user based on the user's food consumption habits, current refrigerator inventory, and emotional information.

[0894] Program processing

[0895] Next, the processing of the system program will be explained in natural language.

[0896] 1. User Initial Registration

[0897] Users enter their food consumption habits, cooking preferences, allergy information, and emotional state through a dedicated application.

[0898] The terminal transmits these input data to the server.

[0899] The server receives the data and stores it in a database, which is managed for each user and used as the initial data for the machine learning model and emotion engine.

[0900] 2. Check the refrigerator inventory

[0901] The device (refrigerator camera) automatically takes pictures of the inside of the refrigerator at a set time every morning.

[0902] The terminal transmits the captured image data to the server.

[0903] The server uses image recognition algorithms to analyze the image data and identify the inventory status of the refrigerator, and the analysis results are stored in a database.

[0904] 3. Emotion Recognition and Data Update

[0905] When a user uses the dedicated app, the emotion engine recognizes emotions from the user's input, voice, and facial expressions.

[0906] The server stores the emotional information analyzed by the emotion engine in a database and reflects it in the user's consumption patterns and recipe selection.

[0907] 4. Shopping list generation and notifications

[0908] The server predicts the ingredients needed based on the user's food consumption patterns, the latest inventory status, and emotional information, and generates a shopping list at a set time in the evening.

[0909] The server selects optimal recipes based on the user's preferences and emotional state along with the generated shopping list and sends them to the user's device.

[0910] The device displays notifications to the user through a dedicated app, allowing the user to check their shopping list and recipes.

[0911] 5. User shopping and cooking

[0912] Users can purchase the ingredients they need at the supermarket based on their shopping list. When making a purchase, the system also suggests ingredients and recipes that will boost their mood based on the "recommended mood" provided by the emotion engine.

[0913] The app suggests recipes based on the ingredients purchased and the inventory in the refrigerator, allowing users to cook. For example, they can make herbal tea to relax or a nutritious meal to energize.

[0914] The system of the present invention allows users to avoid forgetting to buy ingredients and avoiding waste, allowing them to shop and cook efficiently. It also suggests optimal recipes and ingredients based on the user's emotional state, improving the quality and satisfaction of meals.

[0915] The processing flow will be explained below.

[0916] Step 1:

[0917] Users install a dedicated application and input their food consumption habits, cooking preferences, allergy information, and emotional state, such as the ingredients they use each week, the dishes they make on specific days, and their current mood (e.g., energetic, tired, or wanting to relax).

[0918] Step 2:

[0919] The device sends the user's input data to the server, including the frequency of food consumption, type of cuisine, a list of ingredients that may cause allergies, and the user's emotional information.

[0920] Step 3:

[0921] The server stores the received data in a database, where it manages detailed information about each user's food consumption habits and emotions.

[0922] Step 4:

[0923] The server uses the saved data to initialize the machine learning model and emotion engine, learning the user's food consumption patterns and emotional patterns. For example, the model can register the user's habit of eating eggs every Monday and their emotional reactions to certain ingredients.

[0924] Step 5:

[0925] The device (refrigerator camera) automatically takes a photo of the inside of the refrigerator every morning at a set time or at a time specified by the user.

[0926] Step 6:

[0927] The device sends the captured image data to the server. This image must show the entire interior of the refrigerator.

[0928] Step 7:

[0929] The server then uses an image recognition algorithm to analyze the received image data and identify the types and quantities of food items in the refrigerator, such as whether there is one liter of milk left or two eggs.

[0930] Step 8:

[0931] The server updates the analysis results and stores them in a database, which is updated in real time.

[0932] Step 9:

[0933] The server generates a shopping list for a specific time in the evening based on the user's food consumption patterns, latest inventory information, and emotional information. For example, one liter of milk and six eggs are added to the list.

[0934] Step 10:

[0935] The server then generates a shopping list and selects the optimal recipe based on the user's preferences and emotional state. For example, if the user is looking to relax, it will suggest recipes using herbal teas and ingredients with relaxing effects.

[0936] Step 11:

[0937] The server sends the shopping list and recipes to the user's device.

[0938] Step 12:

[0939] The device will display a notification to the user through a dedicated application, and the user can open the app to view the displayed shopping list and recipes.

[0940] Step 13:

[0941] Users can purchase the ingredients they need at the supermarket based on their shopping list. At the time of purchase, the emotion engine also suggests ingredients and recipes that will boost their mood based on the "recommended mood" provided by the emotion engine.

[0942] Step 14:

[0943] The app suggests recipes based on the ingredients purchased and the stock in the refrigerator, allowing users to cook. For example, if a user is tired, they can choose a dish that will restore stamina or a dish that is easy to make.

[0944] In this way, this system incorporating an emotion engine helps users avoid forgetting to buy ingredients and avoiding waste, allowing them to shop and cook efficiently. Furthermore, since suggestions are made that take into account the user's emotional state, it can improve the quality and satisfaction of meals.

[0945] Example 2

[0946] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0947] In today's world, many people lead busy lives, and food waste due to forgetting to buy ingredients or expiry dates is a major problem. It is also not easy to find optimal recipes based on one's own food consumption habits and preferences. Furthermore, there are few systems that provide meal suggestions that take into account daily emotional states, and there is a need for systems that can help improve the quality and satisfaction of meals.

[0948] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0949] In this invention, the server includes: means for a user to input information about food ingredient consumption habits, cooking preferences, allergy information, and emotional state; means for receiving information from the input means using a data transmission means and storing it in a database; means for automatically taking periodic images of the inside of the refrigerator; means for receiving image data obtained by the image taking means and analyzing it using an image recognition algorithm; means for generating a shopping list based on the analyzed inventory information and the consumption habits and emotional information stored in the database; means for notifying a user terminal of the generated shopping list and optimal recipes; and means for analyzing the user's emotional state using an emotion engine and reflecting the analysis results in system operation. This allows the user to use ingredients efficiently and improve the quality and satisfaction of their meals by receiving optimal recipes based on their emotional state.

[0950] "Food consumption habits" is information about how often and what types of food ingredients a user consumes within a certain period of time.

[0951] "Cooking preferences" is information relating to the types of dishes, characteristics of ingredients, and cooking methods that the user particularly likes.

[0952] "Allergy information" is information about ingredients or substances to which the user is allergic.

[0953] An "emotional state" is information about the sensations or moods a user is experiencing at a particular point in time.

[0954] "Data transmission means" is a communication means for transmitting input information to a server.

[0955] "Database" refers to a data storage system for storing a user's food consumption habits, cooking preferences, allergy information, and emotional state.

[0956] An "image recognition algorithm" is a computer program that analyzes captured image data and identifies objects and characters contained in the image.

[0957] An "emotion engine" is a system for analyzing a user's emotional state from input text, voice, facial expressions, etc.

[0958] A "shopping list" is a list of ingredients to be purchased that is generated based on the current inventory in the refrigerator, the user's ingredient consumption habits, and their emotional state.

[0959] A "recipe" is a description of the steps and ingredients needed to make a dish.

[0960] A "user terminal" is an electronic device that allows a user to access the system and input or receive information.

[0961] The "means for automatically taking images of the inside of the refrigerator" is a system that periodically takes photos of the inventory status inside the refrigerator using a camera.

[0962] A "machine learning algorithm" is a technology that learns a user's food consumption habits and emotional information and predicts future consumption patterns.

[0963] "Analysis results" are analytical data obtained by the server using image recognition algorithms and emotion engines.

[0964] The present invention relates to a system that learns a user's food consumption habits and refrigerator inventory status, and proposes optimal shopping lists and recipes, and further includes a function that recognizes the user's emotions and adjusts the system's operation based on the recognition results. Specific embodiments for carrying out the present invention will be described below.

[0965] The system is composed of hardware and software such as a dedicated application, an in-fridge camera, an image recognition algorithm, a machine learning algorithm, an emotion engine, a database, a server, and a user terminal.

[0966] First, the user installs a dedicated application and inputs initial information such as their food consumption habits, cooking preferences, allergy information, and emotional state. This information is sent from the user's terminal to the server using a data transmission means, and the server stores the received information in a database. For example, users might enter information such as "I eat chicken once a week," "I like Italian food," "I have a nut allergy," and "I want to relax."

[0967] To understand the inventory status in the refrigerator, the device (in-fridge camera) automatically takes pictures of the inside of the refrigerator at a set time every morning. The captured image data is sent from the device to a server. The server analyzes the image data using an image recognition algorithm such as Google Cloud Vision API to determine the inventory status in the refrigerator. For example, it may determine that there are three tomatoes, one bag of pasta, and two pieces of cheese in the refrigerator. The analysis results are stored in a database.

[0968] When a user uses the dedicated app, the emotion engine analyzes the user's input text, voice, and facial expressions in real time. For example, if a user inputs "I'm not feeling well," the text analysis identifies that emotion. The server stores the emotion engine's analysis results in a database and reflects them in the user's consumption patterns and recipe selection.

[0969] The server predicts the ingredients needed based on the user's food consumption patterns, the inventory status in the refrigerator, and emotional information. For example, if the user inputs "I use chicken twice a week" but there is no chicken in the refrigerator, the server will add more chicken. The server generates a shopping list at a set time in the evening. Along with the generated shopping list, it selects the optimal recipe based on the user's preferences and emotional state. For example, if the user inputs "I want to relax," a recipe for herbal tea with a relaxing effect will be selected. The server sends the shopping list and recipe to the user's device and displays a notification to the user through a dedicated app.

[0970] Based on the shopping list, the user purchases the ingredients they need at the supermarket. When making a purchase, the app also suggests ingredients and recipes that will boost their mood based on the "recommended mood" provided by the emotion engine. For example, it might suggest, "I feel like relaxing today, so why not try some cardamom tea?" The user then cooks a meal using recipes suggested by the app based on the ingredients they purchased and the inventory in their refrigerator. For example, they could make chicken and basil pasta using the tomatoes, pasta, and cheese in their refrigerator.

[0971] This system allows users to shop and cook efficiently, preventing them from forgetting to buy ingredients or wasting them. It also suggests optimal recipes and ingredients based on the user's emotional state, improving the quality and satisfaction of meals.

[0972] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0973] Step 1: Onboarding users

[0974] The user launches a dedicated application and inputs initial information such as food consumption habits, cooking preferences, allergy information, and emotional state.

[0975] For example, you can enter information such as "I eat chicken once a week," "I like Italian food," "I have a nut allergy," or "I want to relax."

[0976] Input data: food consumption habits, cooking preferences, allergy information, emotional state

[0977] The terminal transmits the input data to the server.

[0978] Data transmission: From user device to server

[0979] The server stores the received data in a database and manages it for each user.

[0980] Data storage: The server stores data in a database

[0981] Step 2: Check the refrigerator inventory

[0982] The device (refrigerator camera) automatically takes pictures of the inside of the refrigerator at a set time every morning.

[0983] For example, capture an image of the inside of your refrigerator at 8am.

[0984] Input data: Image data of the inside of the refrigerator

[0985] The terminal transmits the captured image data to the server.

[0986] Data transmission: From the camera inside the refrigerator to the server

[0987] The server uses image recognition algorithms to analyze the image data and determine the inventory status within the refrigerator.

[0988] Data analysis: Using image recognition algorithms such as Google Cloud Vision API

[0989] For example: "Identify that there are 3 tomatoes, 1 bag of pasta, and 2 cheeses in the refrigerator."

[0990] Analysis result: Inventory information (3 tomatoes, 1 bag of pasta, 2 cheeses)

[0991] The server stores the analysis results in a database.

[0992] Data storage: The server stores data in a database

[0993] Step 3: Emotion recognition and data update

[0994] When a user uses the dedicated app, the emotion engine analyzes the user's input text, voice, and facial expressions in real time.

[0995] For example, if a user types "I'm feeling down," text analysis will identify that emotion.

[0996] Input data: text, voice, facial expression data

[0997] The server obtains the analysis results of the emotion engine.

[0998] Data analysis: Emotion engine uses text analysis, speech recognition, and facial expression recognition

[0999] Example: "The user wants to relax."

[1000] Analysis results: Emotional state information

[1001] The server stores the analyzed emotional information in a database and reflects it in the user's consumption patterns and recipe selection.

[1002] Data storage: The server stores data in a database

[1003] Step 4: Create a shopping list

[1004] The server predicts the ingredients needed based on the user's food consumption patterns, the inventory status in the refrigerator, and emotional information.

[1005] For example, if a user inputs "I use chicken twice a week" but there is no chicken in the refrigerator, new chicken will be added.

[1006] Input data: food consumption patterns, refrigerator inventory, emotional information

[1007] The server generates a shopping list at a fixed time in the evening.

[1008] Data generation: Shopping list (e.g. chicken, basil)

[1009] Generated data: Shopping list (chicken, basil)

[1010] Step 5: Recipe suggestions and notifications

[1011] The server generates a shopping list and selects the best recipe based on the user's preferences and emotional state.

[1012] For example, if the user types "I want to relax," a recipe for a relaxing herbal tea will be selected.

[1013] Input data: shopping list, food consumption habits, emotional information

[1014] The server sends the shopping list and recipes to the user's device.

[1015] Data transmission: From the server to the user's device

[1016] The device will display a notification to the user through a dedicated app.

[1017] Notifications: Shopping Lists and Recipes

[1018] For example: "Your shopping list has been updated. Please purchase chicken and basil. My recommended recipe is chicken and basil pasta."

[1019] Step 6: User Shopping and Cooking

[1020] The user purchases the necessary ingredients at the supermarket based on the shopping list.

[1021] For example, buy chicken and basil according to the list.

[1022] The emotional engine suggests ingredients and recipes that will boost your mood based on your "recommended mood."

[1023] For example, suggest, "I'm feeling relaxed today, how about trying some cardamom tea?"

[1024] Users cook using recipes suggested by the app based on the ingredients they have purchased and what is in their refrigerator.

[1025] For example, make chicken and basil pasta using the tomatoes, pasta, and cheese in your fridge.

[1026] Through these steps, the system prevents users from forgetting to buy ingredients or wasting them, enabling them to shop and cook efficiently. It also improves the quality and satisfaction of meals by suggesting optimal recipes and ingredients based on the user's emotional state.

[1027] (Application example 2)

[1028] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1029] In today's busy daily lives, it is extremely difficult for users to effectively manage their food consumption habits and refrigerator inventory. In addition, there is a need for systems that provide optimal shopping lists and recipes while taking into account the user's emotional state while preparing meals. However, conventional systems are unable to incorporate information, including the user's emotional state, making it difficult to prevent unnecessary food purchases and food waste. Furthermore, the virtual store's purchasing experience is often insufficient, causing user stress. A system that solves these problems is needed.

[1030] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1031] In this invention, the server includes a means for allowing a user to input information about their food ingredient consumption habits, cooking preferences, allergy information, and emotional state; a means for automatically taking periodic images of the inside of the refrigerator; a means for receiving and analyzing the image data obtained by the image taking means; a means for analyzing emotions from the user's input, voice, and facial expressions using an emotion recognition engine; and a means for supporting the shopping experience in the virtual store. This allows the user to significantly reduce food waste and shop and cook efficiently. Furthermore, the server can suggest ingredients and provide recipes based on the user's emotional state, further improving the shopping experience in the virtual store.

[1032] Definitions of important words

[1033] "User" refers to an individual who uses the system to manage ingredients and receive recipe suggestions.

[1034] "Food consumption habits" refers to the types, amounts, and frequency of food ingredients that a user consumes on a daily basis.

[1035] "Cooking preferences" refers to the types and seasonings of food that a user likes.

[1036] "Allergy information" refers to foods or ingredients to which the user is allergic.

[1037] "Emotional state" refers to the user's emotional condition, such as stress or relaxation.

[1038] "Input means" refers to an interface through which a user inputs information into a system.

[1039] "Photographing means" refers to a device or function that automatically photographs images inside the refrigerator.

[1040] "Image data" refers to a photograph of the inside of the refrigerator taken by a photographing means.

[1041] "Analysis means" refers to a device or function that includes an algorithm for analyzing image data and identifying the inventory status and ingredients in the refrigerator.

[1042] An "emotion recognition engine" refers to technology or software for analyzing emotions from user input, voice, and facial expressions.

[1043] A "shopping list" is a list of all the ingredients you need.

[1044] "User terminal" refers to a device through which a user interfaces with the system.

[1045] "Notification means" refers to a technique or device for notifying the user of the generated shopping list or recipe.

[1046] "Recipe suggestion means" refers to a device or function that suggests optimal recipes based on a shopping list.

[1047] A "virtual store" refers to a virtual shopping environment for purchasing food ingredients over the Internet.

[1048] "Supporting Means" refers to technology or devices that support the virtual store shopping experience.

[1049] The present invention relates to a system that learns a user's food consumption habits and refrigerator inventory status, recognizes the user's emotions, and proposes optimal shopping lists and recipes. Specific embodiments for implementing the system of the present invention are described below.

[1050] System configuration

[1051] The system mainly uses the following hardware and software:

[1052] Hardware

[1053] 1. Smartphone: A device where users input information.

[1054] 2. Refrigerator camera: A device that periodically takes pictures of the inside of the refrigerator.

[1055] 3. Server: A computer that stores, analyzes, and notifies data.

[1056] software

[1057] 1. TensorFlow: Machine learning framework.

[1058] 2. OpenCV: Image recognition library.

[1059] 3. Google Cloud Vision API: Image analysis solution.

[1060] 4. Azure Cognitive Services: Sentiment analysis service.

[1061] Specific processing of the program

[1062] The program of this system operates in the following manner.

[1063] 1. User Initial Registration

[1064] Users input their food consumption habits, cooking preferences, allergy information, and emotional state through a dedicated smartphone app.

[1065] The terminal sends this information to the server, which stores it in a database.

[1066] 2. Check the refrigerator inventory

[1067] The camera inside the refrigerator periodically takes pictures of the inside of the refrigerator.

[1068] The device sends the captured image data to a server, which then analyzes the image data using the Google Cloud Vision API and OpenCV to determine inventory status.

[1069] The analysis results are stored in a database.

[1070] 3. Emotion Recognition and Data Update

[1071] When a user uses the app, the emotion recognition engine analyzes emotions from the user's input, voice, and facial expressions. Emotion analysis is performed using Azure Cognitive Services.

[1072] The server stores the results of the sentiment analysis in a database and reflects them in food consumption patterns and recipe selection.

[1073] 4. Shopping list generation and notifications

[1074] The server generates a shopping list based on the user's consumption patterns, latest inventory status, and emotional information.

[1075] After generating the list, the most suitable recipe is selected and notified to the user's smartphone.

[1076] Users can access shopping lists and recipes through the app.

[1077] 5. Virtual store shopping support

[1078] Users can purchase ingredients suggested within the app at a virtual store.

[1079] The server provides a recommendation system to support the shopping experience in the virtual store.

[1080] Specific examples

[1081] For example, if a user types "I'm stressed today," the emotion analysis engine will determine the emotion "stress" and suggest herbal teas or relaxing dishes that will help reduce stress.

[1082] Example prompt sentence:

[1083] Write a Python program that recognizes emotions from text input and suggests ingredients and recipes based on them. Use the following libraries: TensorFlow, OpenCV, Google Cloud Vision API, and Azure Cognitive Services. If the user types "I'm tired today," suggest a corresponding recipe for a relaxing dish, such as chamomile tea or soup.

[1084] This system allows users to efficiently manage ingredients and purchase the ingredients they need without waste. It also suggests optimal recipes based on the user's emotional state, improving the quality and satisfaction of meals.

[1085] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1086] Program processing steps

[1087] Step 1:

[1088] Onboarding a User

[1089] Users launch the smartphone app and enter their food consumption habits, cooking preferences, allergy information, and emotional state.

[1090] Input: User's food consumption habits, cooking preferences, allergy information, and emotional state.

[1091] Data processing: Format user input information in JSON format.

[1092] The terminal transmits the input information to the server.

[1093] The server stores this data in a database.

[1094] Output: User information stored in the database.

[1095] Step 2:

[1096] Checking the inventory in the refrigerator

[1097] The camera inside the refrigerator automatically takes pictures of the inside of the refrigerator at a set time (e.g., every morning).

[1098] Input: Image data of the inside of the refrigerator.

[1099] Data processing: Image data is sent to the server in JPEG format.

[1100] The terminal transmits the captured image data to the server.

[1101] The server uses Google Cloud Vision API and OpenCV to analyze the image data and determine the inventory status in the refrigerator.

[1102] Data Computing: Image recognition algorithms are used to identify ingredients and generate inventory lists.

[1103] The analysis results are stored in a database.

[1104] Output: Refrigerator inventory information stored in a database.

[1105] Step 3:

[1106] Emotion Recognition and Data Update

[1107] While using a smartphone app, a user inputs their emotional state by typing or speaking.

[1108] Input: Text and / or audio data of the user's emotional state.

[1109] Data processing: Converting voice data into text and extracting sentiment keywords.

[1110] The terminal transmits the emotion data to the server.

[1111] The server uses Azure Cognitive Services to perform sentiment analysis and identify the user's emotional state.

[1112] Data Computing: Emotion recognition algorithms determine and store emotional states.

[1113] Output: User's emotional state information stored in a database.

[1114] Step 4:

[1115] Shopping list generation and notifications

[1116] The server generates a shopping list based on stored consumption patterns, up-to-date inventory status, and emotional information.

[1117] Input: User consumption habits, refrigerator inventory, and emotional information.

[1118] Data Computing: Predicting ingredient needs using machine learning algorithms.

[1119] The generated shopping list and the optimal recipe are selected and sent to the user's device.

[1120] Output: Shopping list and suggested recipes sent to the user's device.

[1121] Step 5:

[1122] Virtual store shopping support

[1123] Users purchase ingredients at a virtual store based on a shopping list presented within the app.

[1124] Input: Shopping list and virtual store catalog information.

[1125] Data processing: Add selected ingredients to cart.

[1126] The server uses a recommendation system to support the virtual store shopping experience.

[1127] Output: Order completed in the virtual store and purchase notification sent to the user.

[1128] This system allows users to efficiently manage ingredients and purchase the ingredients they need without waste. It also suggests optimal recipes based on the user's emotional state, improving the quality and satisfaction of meals.

[1129] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1130] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1131] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1132] [Third embodiment]

[1133] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1134] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1135] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1136] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1137] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1138] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1139] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1140] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1141] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1142] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1143] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1144] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1145] The present invention relates to a system that learns a user's food consumption habits and refrigerator inventory status and proposes optimal shopping lists and recipes. Specific embodiments for implementing the system of the present invention will be described below.

[1146] System configuration

[1147] The system mainly includes the following components:

[1148] 1. Input Method

[1149] Using a dedicated application, users input their food consumption habits, cooking preferences, allergy information, etc. This information is used as the initial data for the system.

[1150] 2. How to store data in a database

[1151] The terminal transmits the data entered by the user to the server.

[1152] The server stores the received data in a database, which records detailed food consumption habits for each user.

[1153] 3. Filming Method

[1154] The device (refrigerator camera) periodically takes photos of the inventory status inside the refrigerator. For example, photos are taken automatically at set times, such as every morning or after meals.

[1155] 4. Means of receiving and analyzing image data

[1156] The terminal transmits the captured image data to the server.

[1157] The server analyzes the received image data using an image recognition algorithm to identify the inventory status within the refrigerator, and the analysis results are stored in a database.

[1158] 5. How to generate a shopping list

[1159] The server predicts the ingredients needed based on the user's food consumption patterns and the latest refrigerator inventory data, and generates a shopping list.

[1160] 6. Means of notifying the user terminal

[1161] The server sends the generated shopping list and suggested recipes to the user's device, where a dedicated application notifies and displays them.

[1162] 7. A way to suggest the best recipes

[1163] The server selects and suggests appropriate recipes to the user based on the user's food consumption habits and current refrigerator inventory.

[1164] Program processing

[1165] Next, the processing of the system program will be explained in natural language.

[1166] 1. User Initial Registration

[1167] Users enter their food consumption habits, cooking preferences, and allergy information through a dedicated application.

[1168] The terminal transmits these input data to the server.

[1169] The server receives the data, stores it in a database, and initializes a machine learning model to learn the user's food consumption patterns.

[1170] 2. Check the refrigerator inventory

[1171] The device (refrigerator camera) automatically takes pictures of the inside of the refrigerator at 6:00 every morning.

[1172] The terminal transmits the captured image data to the server.

[1173] The server uses image recognition algorithms to analyze the image data and determine the inventory status of the refrigerator, and the analysis results are updated in a database.

[1174] 3. Shopping list generation and notifications

[1175] The server predicts the ingredients needed at 6pm based on the user's food consumption patterns and the latest inventory status, and generates a shopping list.

[1176] The server sends the generated shopping list and recommended recipes to the user's device.

[1177] The device notifies the user through a dedicated app and displays shopping lists and recipes.

[1178] 4. User shopping and cooking

[1179] Users open the app, check their shopping list, and purchase the ingredients they need.

[1180] Users cook using recipes suggested by the app based on the ingredients they have purchased and the inventory they have in stock.

[1181] Specific examples

[1182] For example, if a user specifies that they consume milk and eggs every week, the system will learn this. While checking the refrigerator inventory each day, it will determine that there is one liter of milk and two eggs left. When the user goes shopping, the system will automatically generate a shopping list that includes "one liter of milk and six eggs" and notify the app. The system will also suggest recipes such as omelets and French toast based on the ingredients in the refrigerator.

[1183] In this way, the system of the present invention allows users to avoid forgetting to buy ingredients or wasting them, and allows them to shop and cook efficiently.

[1184] The processing flow will be explained below.

[1185] Step 1:

[1186] Users install a dedicated application and enter their food consumption habits, cooking preferences, and allergy information, specifically the ingredients they use each week and the dishes they make on specific days.

[1187] Step 2:

[1188] The device sends the user's input data to the server, including the frequency of food consumption, type of cuisine, and a list of ingredients that may cause allergies.

[1189] Step 3:

[1190] The server stores the received data in a database, where the data is managed separately for each user.

[1191] Step 4:

[1192] The server uses the stored data to initialize a machine learning model and learns the user's food consumption patterns. For example, the model can learn that the user uses eggs every Monday.

[1193] Step 5:

[1194] The device (refrigerator camera) automatically takes a photo of the inside of the refrigerator every morning at 6:00 or at a time specified by the user.

[1195] Step 6:

[1196] The device sends the captured image data to the server. This image must show the entire interior of the refrigerator.

[1197] Step 7:

[1198] The server then uses an image recognition algorithm to analyze the received image data and identify the types and quantities of food items in the refrigerator, such as whether there is one liter of milk left or two eggs.

[1199] Step 8:

[1200] The server updates the analysis results and stores them in a database, which is updated in real time.

[1201] Step 9:

[1202] The server generates a shopping list every day at 6pm based on the user's food consumption patterns and the latest inventory information. For example, one liter of milk and six eggs are added to the list.

[1203] Step 10:

[1204] The server generates a shopping list and selects the best recipes based on the user's preferences, such as omelet and French toast recipes.

[1205] Step 11:

[1206] The server sends the shopping list and recipes to the user's device.

[1207] Step 12:

[1208] The device will display a notification to the user through a dedicated application, and the user can open the app to view the displayed shopping list and recipes.

[1209] Step 13:

[1210] The user goes to the supermarket and buys the ingredients they need based on the shopping list, for example, adding 1 liter of milk and 6 eggs to their cart.

[1211] Step 14:

[1212] The app suggests recipes based on the ingredients purchased and the inventory in the refrigerator, allowing users to cook dishes such as omelets and French toast.

[1213] Example 1

[1214] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1215] Conventional refrigerator management systems require users to manually manage food inventory, which is time-consuming and can lead to forgotten purchases and wasteful consumption. It is also difficult to provide effective shopping lists and recipes that take into account the user's food consumption habits. This leads to inefficiencies in the user's lifestyle and wastes time and energy.

[1216] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1217] In this invention, the server includes input means for inputting a user's ingredient consumption habits and cooking preferences, means for receiving information from the input means and storing the information in a database, photographing means for automatically photographing images of the inside of the refrigerator periodically, means for receiving and analyzing image data obtained by the photographing means, means for generating a shopping list based on the analyzed inventory information and the stored consumption habits, means for notifying a user terminal of the generated shopping list, means for using a machine learning algorithm to predict optimal ingredients based on the suggested shopping list, and means for suggesting recipes based on the optimal ingredients and the user's consumption habits, thereby enabling users to efficiently manage ingredient consumption and inventory without hassle.

[1218] "User" refers to a person who uses the system to input their food consumption habits and cooking preferences and receive suggested shopping lists and recipes.

[1219] "Food consumption habits" refers to data that indicates the tendency and frequency of food ingredients consumed by a user during a specific period.

[1220] "Cooking preferences" refers to information about the user's favorite dishes and cooking methods.

[1221] "Input means" refers to an interface that allows a user to input their food consumption habits and cooking preferences into the system.

[1222] "Database" refers to an information system for storing information received from users, analysis results, inventory information, etc.

[1223] "Photographing means" refers to a device or equipment that automatically takes pictures of the inside of the refrigerator on a regular basis.

[1224] "Image data" refers to photographs and video data of the inside of the refrigerator captured by a photographing means.

[1225] "Analysis means" refers to a system or method that uses an image recognition algorithm to analyze captured image data and identify inventory information within the refrigerator.

[1226] "Inventory information" refers to data about the ingredients currently in the refrigerator and their quantities.

[1227] A "shopping list" is a list created by predicting the ingredients a user will need in the future.

[1228] "Notification means" refers to a system or method for sending the generated shopping list and suggested recipes to a user terminal.

[1229] "Machine learning algorithm" refers to a calculation method or program that learns from a user's food consumption habits and refrigerator inventory information to predict the best ingredients.

[1230] "Recipe suggestion means" refers to a system or method for providing a user with a recipe suitable for cooking based on optimal ingredients and the user's consumption habits.

[1231] The present invention relates to a system that learns a user's food consumption habits and refrigerator inventory status, and proposes optimal shopping lists and recipes. This system is implemented using the following hardware and software.

[1232] 1. Hardware Configuration

[1233] The system includes the following hardware:

[1234] User device: Mobile device such as a smartphone or tablet

[1235] Refrigerator camera: A camera that periodically takes pictures of the inside of a refrigerator

[1236] Server: A computer system that runs a database and image recognition and machine learning algorithms.

[1237] 2. Software Configuration

[1238] The system includes the following software:

[1239] Dedicated application: An app for users to input information about food consumption habits, cooking preferences, and allergies

[1240] Database Management Systems: Databases such as MySQL or PostgreSQL

[1241] Image recognition algorithm: Object detection algorithm using OpenCV and TensorFlow

[1242] Machine learning algorithms: predictive models using scikit-learn and TensorFlow

[1243] 3. System Operation

[1244] Onboarding a User

[1245] Users enter their food consumption habits, cooking preferences, allergy information, etc. through a dedicated application. This information is used as the initial data for the system.

[1246] The terminal transmits the data entered by the user to the server.

[1247] The server stores the received data in a database and initializes and starts training a machine learning model to learn the user's food consumption patterns.

[1248] Checking the inventory in the refrigerator

[1249] The device (refrigerator camera) automatically takes pictures of the inside of the refrigerator at a fixed time every day. This time can be set by the user.

[1250] The terminal transmits the captured image data to the server.

[1251] The server analyzes the image data using an image recognition algorithm to identify the inventory status of the refrigerator, and the analysis results are reflected in a database.

[1252] Shopping list generation and notifications

[1253] The server predicts the ingredients needed based on the user's ingredient consumption patterns and the latest inventory status, and generates a shopping list.

[1254] The server sends the generated shopping list to the user's device, and a dedicated app notifies the user and displays the list to them.

[1255] Recipe Suggestions

[1256] The server selects and suggests appropriate recipes based on the user's consumption habits and refrigerator inventory. Recipe suggestions are made via a dedicated app.

[1257] 4. Specific Examples

[1258] For example, if a user specifies that they "eat eggs and bread for breakfast every day," the system learns this and predicts the ingredients they will need based on their consumption patterns over the course of a week. A camera inside the refrigerator checks the inventory every day and generates a shopping list based on the analyzed results. When the user opens the app on their smartphone, a shopping list of "10 eggs and 1 bag of bread" is displayed. The system also suggests recipes such as "French toast" and "omelette" based on the inventory status in the refrigerator.

[1259] Prompt Sentence Examples

[1260] "Can you recommend a simple and delicious recipe for dinner? I have eggs and milk in the fridge."

[1261] In this way, the system of the present invention allows users to receive efficient support regarding the management and consumption of ingredients, preventing them from forgetting to buy food or wasting it.

[1262] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1263] Step 1: Onboarding users

[1264] Users open a dedicated app and enter data such as food consumption habits, cooking preferences, and allergy information.

[1265] Input: Food consumption habits, cooking preferences, allergy information

[1266] Specific action: The user enters the required information into a form within the app and presses the submit button.

[1267] The terminal receives input data and transmits it to the server.

[1268] Output: Sending user-entered data

[1269] The server stores the received data in a database.

[1270] Input: User's consumption habits, food preferences, and allergy information

[1271] Data processing: Store information in the database in JSON format and organize it by user ID.

[1272] Output: Database update

[1273] The server initializes the machine learning model with the user's consumption patterns and performs initial training.

[1274] Input: Initial database information

[1275] Data Computing: Model training with machine learning algorithms

[1276] Output: The model after initial training

[1277] Step 2: Take a photo of the refrigerator

[1278] The device (refrigerator camera) automatically takes pictures of the inside of the refrigerator at 6:00 every morning.

[1279] What it does: The camera's timer will activate and take images of each shelf inside the refrigerator.

[1280] Input: Real-time video of the inside of the refrigerator

[1281] Output: Captured image data

[1282] The terminal transmits the captured image data to the server.

[1283] Specific operation: Compresses image data and sends it to the server via an HTTP POST request.

[1284] Input: Captured image

[1285] Output: Image data sent to the server

[1286] Step 3: Sending and analyzing image data

[1287] The server analyzes the received image data.

[1288] Input: Image data sent from the device

[1289] Specific operation: The server uses Python's OpenCV library to apply image recognition algorithms.

[1290] Data processing: Perform object detection from image data to identify inventory.

[1291] Output: Inventory information as analysis result

[1292] The server stores the analysis results in a database and updates inventory data.

[1293] Input: Parsed inventory information

[1294] Data operation: Update the inventory table in the database.

[1295] Output: Updated database

[1296] Step 4: Create a shopping list

[1297] The server generates a shopping list based on the user's consumption patterns and the latest inventory information.

[1298] Input: Latest database information

[1299] Data Computing: Using machine learning models to predict food shortages.

[1300] Specific operation: Formalize the model's prediction results as a shopping list.

[1301] Output: Generated shopping list

[1302] Step 5: Shopping List and Recipe Notifications

[1303] The server notifies the user's terminal of the generated shopping list and recommended recipes.

[1304] Input: Shopping list, recommended recipes

[1305] Specific operation: Sends a push notification from the server and sends data to the API endpoint.

[1306] Output: Notification data sent to the user's device

[1307] The device notifies the user through a dedicated app and displays shopping lists and recipes.

[1308] Input: Notification data from the server

[1309] What it does: Your device will display a push notification and make your lists and recipes available in the app.

[1310] Output: User confirmation

[1311] Step 6: User Shopping and Cooking

[1312] Users open the app, check their shopping list and purchase the ingredients they need.

[1313] Action: Open the app, browse the list and make a purchase.

[1314] Input: Shopping List

[1315] Output: Purchased ingredients

[1316] Users cook using recipes suggested by the app based on the ingredients they have purchased and the inventory they have in stock.

[1317] Specific actions: Cook while looking at the recipe displayed on the app.

[1318] Input: Recommended Recipe

[1319] Output: Cooked food

[1320] (Application example 1)

[1321] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1322] Conventional refrigerator inventory management and food consumption management systems required users to check inventory and manually create shopping lists, which was time-consuming and often led to unnecessary food purchases or shortages. Furthermore, because there was no integration with delivery services, users had to go through the trouble of ordering separately. Furthermore, there was a lack of a mechanism for efficiently learning users' food consumption patterns, making it difficult to suggest appropriate recipes.

[1323] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1324] In this invention, the server includes an input means for inputting the user's ingredient consumption habits and cooking preferences, a photographing means for automatically taking periodic images of the inside of the refrigerator, and a means for automatically placing orders with an affiliated food delivery service. This automates refrigerator inventory management, reducing the user's workload and preventing shortages and unnecessary purchases of ingredients. Furthermore, the server automatically connects with the delivery service, allowing the user to quickly order the necessary ingredients. Furthermore, the server facilitates appropriate recipe suggestions based on the user's ingredient consumption patterns, enhancing user convenience.

[1325] A "user" is an individual or entity that uses the system to input their food consumption habits and cooking preferences.

[1326] "Input means" refers to a device or software that allows a user to input their food consumption habits and cooking preferences.

[1327] A "means for storing in a database" is a process and system that receives information from an input means and stores the information in a database.

[1328] The "photography means" refers to a camera and its control device that automatically captures images of the inside of the refrigerator on a regular basis.

[1329] The "analysis means" refers to hardware and software for receiving and analyzing image data obtained by the imaging means.

[1330] The "means for generating a shopping list" is a mechanism for creating a shopping list based on the analyzed inventory information and the stored consumption habits.

[1331] The "notification means" is a mechanism for transmitting the generated shopping list to the user terminal and informing the user.

[1332] The "means for suggesting recipes" is a system for suggesting optimal cooking recipes to a user based on a shopping list.

[1333] "Means for automatically placing orders with affiliated food delivery services" refers to a system that automatically sends an order to an affiliated food delivery service when necessary ingredients are in short supply.

[1334] This invention relates to a system that learns a user's food consumption habits and refrigerator inventory status, proposes optimal shopping lists and recipes, and even automatically orders the necessary ingredients. Specific embodiments for carrying out this invention are described below.

[1335] System configuration

[1336] The system includes the following components:

[1337] 1. Input Method

[1338] Users use a dedicated application to input their food consumption habits, cooking preferences, and allergy information.

[1339] 2. How to store data in a database

[1340] The terminal sends the input data from the user to the server.

[1341] The server stores the received data in a database, recording each user's detailed food consumption habits.

[1342] 3. Filming Method

[1343] A camera installed inside the refrigerator periodically takes pictures of the inventory inside the refrigerator.

[1344] 4. Analysis method

[1345] The terminal transmits the captured image data to the server.

[1346] The server uses image recognition algorithms to analyze the image data and identify the inventory status of the refrigerator, and the analysis results are stored in a database.

[1347] 5. How to generate a shopping list

[1348] The server predicts the ingredients needed based on the user's food consumption patterns and the latest refrigerator inventory data, and generates a shopping list.

[1349] 6. Means of Notification

[1350] The server sends the generated shopping list and suggested recipes to the user's device, where a dedicated application notifies and displays them.

[1351] 7. Proposed measures

[1352] The server selects and suggests appropriate recipes to users based on their consumption habits and inventory status.

[1353] 8. How to Order

[1354] The server automatically places orders with affiliated food delivery services.

[1355] Program processing

[1356] Next, we explain in natural language the program processing required to implement the system. The main hardware used is a camera installed inside the refrigerator, the user's smartphone, and a server. The software used is Python, OpenCV (image recognition library), Django (web framework), and TensorFlow (machine learning model).

[1357] Onboarding a User

[1358] Users input their food consumption habits, cooking preferences, and allergy information through a dedicated application. The device sends this data to the server, which receives it and stores it in a database. The server then initializes a machine learning model to learn the user's food consumption patterns.

[1359] Checking the inventory in the refrigerator

[1360] A camera installed inside the refrigerator automatically takes pictures of the inside of the refrigerator at 6:00 every morning. The device sends the captured image data to a server, which then analyzes the image data using an image recognition algorithm (OpenCV). The inventory status inside the refrigerator is identified and the results are updated in the database.

[1361] Shopping list generation and notifications

[1362] The server predicts what ingredients a user will need before dinner based on their food consumption patterns and the latest inventory status, and generates a shopping list. The generated shopping list and recommended recipes are sent to the user's device, and a dedicated app notifies the user.

[1363] Automated ordering and delivery

[1364] If a specific ingredient is in short supply, the server automatically places an order with a partner food delivery service, calls an API, and notifies the user of the order confirmation and delivery status.

[1365] Examples of concrete examples and prompts

[1366] For example, suppose a user initially registers their weekly consumption of milk and eggs. If the inventory in the refrigerator drops to 1 liter of milk and 2 eggs, the system will generate a shopping list for "1 liter of milk and 6 eggs," place an automatic order, and deliver the ingredients by 10:00 a.m. the next day. It will also suggest "omelette and French toast" as recipe suggestions.

[1367] Example prompt for a generative AI model:

[1368] You will help design an application that automatically orders ingredients and suggests the best grocery shopping list and recipes based on the user's food consumption habits and refrigerator inventory. Specifically, your functional requirements will be as follows:

[1369] 1. Initial User Registration

[1370] 2. Refrigerator inventory management and regular checks

[1371] 3. Learning food consumption patterns

[1372] 4. Optimal recipe suggestions

[1373] 5. Automated ordering and food delivery integration

[1374] Thank you very much for your cooperation.

[1375] In this way, the system of the present invention eliminates the need for users to perform cumbersome inventory checks or manually order ingredients, realizing convenient and efficient ingredient management.

[1376] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1377] Step 1:

[1378] The user inputs their food consumption habits and cooking preferences.

[1379] Specific operation: The user uses a dedicated application to input information such as food consumption habits, cooking preferences, and allergy information.

[1380] Input: User's food consumption habits, cooking preferences, allergy information, etc.

[1381] Output: User data stored in a dedicated application.

[1382] Step 2:

[1383] The terminal transmits the input data to the server.

[1384] Specific operation: The application sends the user's input data to the cloud server.

[1385] Input: User's food consumption habits, cooking preferences, allergy information, etc.

[1386] Output: User data stored on the server.

[1387] Step 3:

[1388] The server stores the received data in a database.

[1389] Specific operation: The server stores the received data in a database management system (e.g., MySQL).

[1390] Input: User's food consumption habits, cooking preferences, allergy information, etc.

[1391] Output: Detailed user data stored in a database.

[1392] Step 4:

[1393] A camera installed inside the refrigerator periodically takes pictures of the inventory inside the refrigerator.

[1394] Specific operation: The camera inside the refrigerator automatically takes a photo of the inside of the refrigerator at 6:00 every morning.

[1395] Input: Current state inside the refrigerator.

[1396] Output: Image data of the inside of the refrigerator.

[1397] Step 5:

[1398] The terminal transmits the captured image data to the server.

[1399] Specific operation: The camera inside the refrigerator or its control device sends the captured image data to a cloud server.

[1400] Input: Image data of the inside of the refrigerator.

[1401] Output: Image data stored on the server.

[1402] Step 6:

[1403] The server analyzes the image data using an image recognition algorithm.

[1404] Specific operation: The server uses an image recognition library such as OpenCV to analyze image data from inside the refrigerator and determine the inventory status.

[1405] Input: Image data of the inside of the refrigerator.

[1406] Output: Parsed inventory information.

[1407] Step 7:

[1408] The server stores and updates the analysis results in a database.

[1409] Specific operation: The server saves the parsed inventory information in the database and updates the existing inventory information.

[1410] Input: Parsed inventory information.

[1411] Output: The updated database.

[1412] Step 8:

[1413] The server predicts the ingredients needed and generates a shopping list based on the user's food consumption patterns and the latest refrigerator inventory data.

[1414] Specific operation: The server learns the user's past consumption patterns and compares them with current inventory data to generate a shopping list.

[1415] Input: User's food consumption patterns, latest refrigerator inventory data.

[1416] Output: Shopping list.

[1417] Step 9:

[1418] The server sends the generated shopping list and recommended recipes to the user's terminal.

[1419] Specific operation: The server sends the generated shopping list and recipes to a dedicated app and notifies it.

[1420] Input: shopping list, recommended recipes.

[1421] Output: The shopping list and recipes displayed on the user's device.

[1422] Step 10:

[1423] If the server runs out of a particular ingredient, it will automatically place an order with an affiliated food delivery service.

[1424] Specific operation: If the server is short on necessary ingredients, it calls the API of the partner delivery service and automatically places the order.

[1425] Input: Missing ingredient information.

[1426] Output: The order sent to the food delivery service.

[1427] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1428] The present invention relates to a system that learns a user's food consumption habits and refrigerator inventory status, and proposes optimal shopping lists and recipes, and further includes a function to recognize the user's emotions and adjust the system's operation based on the recognition results. Specific embodiments for implementing the system of the present invention are described below.

[1429] System configuration

[1430] The system mainly includes the following components:

[1431] 1. Input Method

[1432] Using a dedicated application, users input their food consumption habits, cooking preferences, allergy information, emotional state, etc. This information is used as the initial data for the system.

[1433] 2. How to store data in a database

[1434] The terminal transmits the data entered by the user to the server.

[1435] The server stores the received data in a database, which records detailed food consumption habits and emotional information for each user.

[1436] 3. Filming Method

[1437] The device (refrigerator camera) periodically takes photos of the inventory status inside the refrigerator. For example, photos are taken automatically at set times, such as every morning or after meals.

[1438] 4. Means of receiving and analyzing image data

[1439] The terminal transmits the captured image data to the server.

[1440] The server analyzes the received image data using an image recognition algorithm to identify the inventory status within the refrigerator, and the analysis results are stored in a database.

[1441] 5. Emotion Engine

[1442] The server includes an emotion engine that recognizes the user's emotional state through user input and interaction. The emotion engine analyzes emotions from text analysis, speech recognition, and user facial expressions.

[1443] 6. How to generate a shopping list

[1444] The server predicts the ingredients needed and generates a shopping list based on the user's food consumption patterns, the latest refrigerator inventory data, and their emotional state.

[1445] 7. Means of notifying the user terminal

[1446] The server sends the generated shopping list and suggested recipes to the user's device, where a dedicated application notifies and displays them.

[1447] 8. A way to suggest the best recipes

[1448] The server selects and suggests appropriate recipes to the user based on the user's food consumption habits, current refrigerator inventory, and emotional information.

[1449] Program processing

[1450] Next, the processing of the system program will be explained in natural language.

[1451] 1. User Initial Registration

[1452] Users enter their food consumption habits, cooking preferences, allergy information, and emotional state through a dedicated application.

[1453] The terminal transmits these input data to the server.

[1454] The server receives the data and stores it in a database, which is managed for each user and used as the initial data for the machine learning model and emotion engine.

[1455] 2. Check the refrigerator inventory

[1456] The device (refrigerator camera) automatically takes pictures of the inside of the refrigerator at a set time every morning.

[1457] The terminal transmits the captured image data to the server.

[1458] The server uses image recognition algorithms to analyze the image data and identify the inventory status of the refrigerator, and the analysis results are stored in a database.

[1459] 3. Emotion Recognition and Data Update

[1460] When a user uses the dedicated app, the emotion engine recognizes emotions from the user's input, voice, and facial expressions.

[1461] The server stores the emotional information analyzed by the emotion engine in a database and reflects it in the user's consumption patterns and recipe selection.

[1462] 4. Shopping list generation and notifications

[1463] The server predicts the ingredients needed based on the user's food consumption patterns, the latest inventory status, and emotional information, and generates a shopping list at a set time in the evening.

[1464] The server selects optimal recipes based on the user's preferences and emotional state along with the generated shopping list and sends them to the user's device.

[1465] The device displays notifications to the user through a dedicated app, allowing the user to check their shopping list and recipes.

[1466] 5. User shopping and cooking

[1467] Users can purchase the ingredients they need at the supermarket based on their shopping list. When making a purchase, the system also suggests ingredients and recipes that will boost their mood based on the "recommended mood" provided by the emotion engine.

[1468] The app suggests recipes based on the ingredients purchased and the inventory in the refrigerator, allowing users to cook. For example, they can make herbal tea to relax or a nutritious meal to energize.

[1469] The system of the present invention allows users to avoid forgetting to buy ingredients and avoiding waste, allowing them to shop and cook efficiently. It also suggests optimal recipes and ingredients based on the user's emotional state, improving the quality and satisfaction of meals.

[1470] The processing flow will be explained below.

[1471] Step 1:

[1472] Users install a dedicated application and input their food consumption habits, cooking preferences, allergy information, and emotional state, such as the ingredients they use each week, the dishes they make on specific days, and their current mood (e.g., energetic, tired, or wanting to relax).

[1473] Step 2:

[1474] The device sends the user's input data to the server, including the frequency of food consumption, type of cuisine, a list of ingredients that may cause allergies, and the user's emotional information.

[1475] Step 3:

[1476] The server stores the received data in a database, where it manages detailed information about each user's food consumption habits and emotions.

[1477] Step 4:

[1478] The server uses the saved data to initialize the machine learning model and emotion engine, learning the user's food consumption patterns and emotional patterns. For example, the model can register the user's habit of eating eggs every Monday and their emotional reactions to certain ingredients.

[1479] Step 5:

[1480] The device (refrigerator camera) automatically takes a photo of the inside of the refrigerator every morning at a set time or at a time specified by the user.

[1481] Step 6:

[1482] The device sends the captured image data to the server. This image must show the entire interior of the refrigerator.

[1483] Step 7:

[1484] The server then uses an image recognition algorithm to analyze the received image data and identify the types and quantities of food items in the refrigerator, such as whether there is one liter of milk left or two eggs.

[1485] Step 8:

[1486] The server updates the analysis results and stores them in a database, which is updated in real time.

[1487] Step 9:

[1488] The server generates a shopping list for a specific time in the evening based on the user's food consumption patterns, latest inventory information, and emotional information. For example, one liter of milk and six eggs are added to the list.

[1489] Step 10:

[1490] The server then generates a shopping list and selects the optimal recipe based on the user's preferences and emotional state. For example, if the user is looking to relax, it will suggest recipes using herbal teas and ingredients with relaxing effects.

[1491] Step 11:

[1492] The server sends the shopping list and recipes to the user's device.

[1493] Step 12:

[1494] The device will display a notification to the user through a dedicated application, and the user can open the app to view the displayed shopping list and recipes.

[1495] Step 13:

[1496] Users can purchase the ingredients they need at the supermarket based on their shopping list. At the time of purchase, the emotion engine also suggests ingredients and recipes that will boost their mood based on the "recommended mood" provided by the emotion engine.

[1497] Step 14:

[1498] The app suggests recipes based on the ingredients purchased and the stock in the refrigerator, allowing users to cook. For example, if a user is tired, they can choose a dish that will restore stamina or a dish that is easy to make.

[1499] In this way, this system incorporating an emotion engine helps users avoid forgetting to buy ingredients and avoiding waste, allowing them to shop and cook efficiently. Furthermore, since suggestions are made that take into account the user's emotional state, it can improve the quality and satisfaction of meals.

[1500] Example 2

[1501] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1502] In today's world, many people lead busy lives, and food waste due to forgetting to buy ingredients or expiry dates is a major problem. It is also not easy to find optimal recipes based on one's own food consumption habits and preferences. Furthermore, there are few systems that provide meal suggestions that take into account daily emotional states, and there is a need for systems that can help improve the quality and satisfaction of meals.

[1503] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1504] In this invention, the server includes: means for a user to input information about food ingredient consumption habits, cooking preferences, allergy information, and emotional state; means for receiving information from the input means using a data transmission means and storing it in a database; means for automatically taking periodic images of the inside of the refrigerator; means for receiving image data obtained by the image taking means and analyzing it using an image recognition algorithm; means for generating a shopping list based on the analyzed inventory information and the consumption habits and emotional information stored in the database; means for notifying a user terminal of the generated shopping list and optimal recipes; and means for analyzing the user's emotional state using an emotion engine and reflecting the analysis results in system operation. This allows the user to use ingredients efficiently and improve the quality and satisfaction of their meals by receiving optimal recipes based on their emotional state.

[1505] "Food consumption habits" is information about how often and what types of food ingredients a user consumes within a certain period of time.

[1506] "Cooking preferences" is information relating to the types of dishes, characteristics of ingredients, and cooking methods that the user particularly likes.

[1507] "Allergy information" is information about ingredients or substances to which the user is allergic.

[1508] An "emotional state" is information about the sensations or moods a user is experiencing at a particular point in time.

[1509] "Data transmission means" is a communication means for transmitting input information to a server.

[1510] "Database" refers to a data storage system for storing a user's food consumption habits, cooking preferences, allergy information, and emotional state.

[1511] An "image recognition algorithm" is a computer program that analyzes captured image data and identifies objects and characters contained in the image.

[1512] An "emotion engine" is a system for analyzing a user's emotional state from input text, voice, facial expressions, etc.

[1513] A "shopping list" is a list of ingredients to be purchased that is generated based on the current inventory in the refrigerator, the user's ingredient consumption habits, and their emotional state.

[1514] A "recipe" is a description of the steps and ingredients needed to make a dish.

[1515] A "user terminal" is an electronic device that allows a user to access the system and input or receive information.

[1516] The "means for automatically taking images of the inside of the refrigerator" is a system that periodically takes photos of the inventory status inside the refrigerator using a camera.

[1517] A "machine learning algorithm" is a technology that learns a user's food consumption habits and emotional information and predicts future consumption patterns.

[1518] "Analysis results" are analytical data obtained by the server using image recognition algorithms and emotion engines.

[1519] The present invention relates to a system that learns a user's food consumption habits and refrigerator inventory status, and proposes optimal shopping lists and recipes, and further includes a function that recognizes the user's emotions and adjusts the system's operation based on the recognition results. Specific embodiments for carrying out the present invention will be described below.

[1520] The system is composed of hardware and software such as a dedicated application, an in-fridge camera, an image recognition algorithm, a machine learning algorithm, an emotion engine, a database, a server, and a user terminal.

[1521] First, the user installs a dedicated application and inputs initial information such as their food consumption habits, cooking preferences, allergy information, and emotional state. This information is sent from the user's terminal to the server using a data transmission means, and the server stores the received information in a database. For example, users might enter information such as "I eat chicken once a week," "I like Italian food," "I have a nut allergy," and "I want to relax."

[1522] To understand the inventory status in the refrigerator, the device (in-fridge camera) automatically takes pictures of the inside of the refrigerator at a set time every morning. The captured image data is sent from the device to a server. The server analyzes the image data using an image recognition algorithm such as Google Cloud Vision API to determine the inventory status in the refrigerator. For example, it may determine that there are three tomatoes, one bag of pasta, and two pieces of cheese in the refrigerator. The analysis results are stored in a database.

[1523] When a user uses the dedicated app, the emotion engine analyzes the user's input text, voice, and facial expressions in real time. For example, if a user inputs "I'm not feeling well," the text analysis identifies that emotion. The server stores the emotion engine's analysis results in a database and reflects them in the user's consumption patterns and recipe selection.

[1524] The server predicts the ingredients needed based on the user's food consumption patterns, the inventory status in the refrigerator, and emotional information. For example, if the user inputs "I use chicken twice a week" but there is no chicken in the refrigerator, the server will add more chicken. The server generates a shopping list at a set time in the evening. Along with the generated shopping list, it selects the optimal recipe based on the user's preferences and emotional state. For example, if the user inputs "I want to relax," a recipe for herbal tea with a relaxing effect will be selected. The server sends the shopping list and recipe to the user's device and displays a notification to the user through a dedicated app.

[1525] Based on the shopping list, the user purchases the ingredients they need at the supermarket. When making a purchase, the app also suggests ingredients and recipes that will boost their mood based on the "recommended mood" provided by the emotion engine. For example, it might suggest, "I feel like relaxing today, so why not try some cardamom tea?" The user then cooks a meal using recipes suggested by the app based on the ingredients they purchased and the inventory in their refrigerator. For example, they could make chicken and basil pasta using the tomatoes, pasta, and cheese in their refrigerator.

[1526] This system allows users to shop and cook efficiently, preventing them from forgetting to buy ingredients or wasting them. It also suggests optimal recipes and ingredients based on the user's emotional state, improving the quality and satisfaction of meals.

[1527] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1528] Step 1: Onboarding users

[1529] The user launches a dedicated application and inputs initial information such as food consumption habits, cooking preferences, allergy information, and emotional state.

[1530] For example, you can enter information such as "I eat chicken once a week," "I like Italian food," "I have a nut allergy," or "I want to relax."

[1531] Input data: food consumption habits, cooking preferences, allergy information, emotional state

[1532] The terminal transmits the input data to the server.

[1533] Data transmission: From user device to server

[1534] The server stores the received data in a database and manages it for each user.

[1535] Data storage: The server stores data in a database

[1536] Step 2: Check the refrigerator inventory

[1537] The device (refrigerator camera) automatically takes pictures of the inside of the refrigerator at a set time every morning.

[1538] For example, capture an image of the inside of your refrigerator at 8am.

[1539] Input data: Image data of the inside of the refrigerator

[1540] The terminal transmits the captured image data to the server.

[1541] Data transmission: From the camera inside the refrigerator to the server

[1542] The server uses image recognition algorithms to analyze the image data and determine the inventory status within the refrigerator.

[1543] Data analysis: Using image recognition algorithms such as Google Cloud Vision API

[1544] For example: "Identify that there are 3 tomatoes, 1 bag of pasta, and 2 cheeses in the refrigerator."

[1545] Analysis result: Inventory information (3 tomatoes, 1 bag of pasta, 2 cheeses)

[1546] The server stores the analysis results in a database.

[1547] Data storage: The server stores data in a database

[1548] Step 3: Emotion recognition and data update

[1549] When a user uses the dedicated app, the emotion engine analyzes the user's input text, voice, and facial expressions in real time.

[1550] For example, if a user types "I'm feeling down," text analysis will identify that emotion.

[1551] Input data: text, voice, facial expression data

[1552] The server obtains the analysis results of the emotion engine.

[1553] Data analysis: Emotion engine uses text analysis, speech recognition, and facial expression recognition

[1554] Example: "The user wants to relax."

[1555] Analysis results: Emotional state information

[1556] The server stores the analyzed emotional information in a database and reflects it in the user's consumption patterns and recipe selection.

[1557] Data storage: The server stores data in a database

[1558] Step 4: Create a shopping list

[1559] The server predicts the ingredients needed based on the user's food consumption patterns, the inventory status in the refrigerator, and emotional information.

[1560] For example, if a user inputs "I use chicken twice a week" but there is no chicken in the refrigerator, new chicken will be added.

[1561] Input data: food consumption patterns, refrigerator inventory, emotional information

[1562] The server generates a shopping list at a fixed time in the evening.

[1563] Data generation: Shopping list (e.g. chicken, basil)

[1564] Generated data: Shopping list (chicken, basil)

[1565] Step 5: Recipe suggestions and notifications

[1566] The server generates a shopping list and selects the best recipe based on the user's preferences and emotional state.

[1567] For example, if the user types "I want to relax," a recipe for a relaxing herbal tea will be selected.

[1568] Input data: shopping list, food consumption habits, emotional information

[1569] The server sends the shopping list and recipes to the user's device.

[1570] Data transmission: From the server to the user's device

[1571] The device will display a notification to the user through a dedicated app.

[1572] Notifications: Shopping Lists and Recipes

[1573] For example: "Your shopping list has been updated. Please purchase chicken and basil. My recommended recipe is chicken and basil pasta."

[1574] Step 6: User Shopping and Cooking

[1575] The user purchases the necessary ingredients at the supermarket based on the shopping list.

[1576] For example, buy chicken and basil according to the list.

[1577] The emotional engine suggests ingredients and recipes that will boost your mood based on your "recommended mood."

[1578] For example, suggest, "I'm feeling relaxed today, how about trying some cardamom tea?"

[1579] Users cook using recipes suggested by the app based on the ingredients they have purchased and what is in their refrigerator.

[1580] For example, make chicken and basil pasta using the tomatoes, pasta, and cheese in your fridge.

[1581] Through these steps, the system prevents users from forgetting to buy ingredients or wasting them, enabling them to shop and cook efficiently. It also improves the quality and satisfaction of meals by suggesting optimal recipes and ingredients based on the user's emotional state.

[1582] (Application example 2)

[1583] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1584] In today's busy daily lives, it is extremely difficult for users to effectively manage their food consumption habits and refrigerator inventory. In addition, there is a need for systems that provide optimal shopping lists and recipes while taking into account the user's emotional state while preparing meals. However, conventional systems are unable to incorporate information, including the user's emotional state, making it difficult to prevent unnecessary food purchases and food waste. Furthermore, the virtual store's purchasing experience is often insufficient, causing user stress. A system that solves these problems is needed.

[1585] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1586] In this invention, the server includes a means for allowing a user to input information about their food ingredient consumption habits, cooking preferences, allergy information, and emotional state; a means for automatically taking periodic images of the inside of the refrigerator; a means for receiving and analyzing the image data obtained by the image taking means; a means for analyzing emotions from the user's input, voice, and facial expressions using an emotion recognition engine; and a means for supporting the shopping experience in the virtual store. This allows the user to significantly reduce food waste and shop and cook efficiently. Furthermore, the server can suggest ingredients and provide recipes based on the user's emotional state, further improving the shopping experience in the virtual store.

[1587] Definitions of important words

[1588] "User" refers to an individual who uses the system to manage ingredients and receive recipe suggestions.

[1589] "Food consumption habits" refers to the types, amounts, and frequency of food ingredients that a user consumes on a daily basis.

[1590] "Cooking preferences" refers to the types and seasonings of food that a user likes.

[1591] "Allergy information" refers to foods or ingredients to which the user is allergic.

[1592] "Emotional state" refers to the user's emotional condition, such as stress or relaxation.

[1593] "Input means" refers to an interface through which a user inputs information into a system.

[1594] "Photographing means" refers to a device or function that automatically photographs images inside the refrigerator.

[1595] "Image data" refers to a photograph of the inside of the refrigerator taken by a photographing means.

[1596] "Analysis means" refers to a device or function that includes an algorithm for analyzing image data and identifying the inventory status and ingredients in the refrigerator.

[1597] An "emotion recognition engine" refers to technology or software for analyzing emotions from user input, voice, and facial expressions.

[1598] A "shopping list" is a list of all the ingredients you need.

[1599] "User terminal" refers to a device through which a user interfaces with the system.

[1600] "Notification means" refers to a technique or device for notifying the user of the generated shopping list or recipe.

[1601] "Recipe suggestion means" refers to a device or function that suggests optimal recipes based on a shopping list.

[1602] A "virtual store" refers to a virtual shopping environment for purchasing food ingredients over the Internet.

[1603] "Supporting Means" refers to technology or devices that support the virtual store shopping experience.

[1604] The present invention relates to a system that learns a user's food consumption habits and refrigerator inventory status, recognizes the user's emotions, and proposes optimal shopping lists and recipes. Specific embodiments for implementing the system of the present invention are described below.

[1605] System configuration

[1606] The system mainly uses the following hardware and software:

[1607] Hardware

[1608] 1. Smartphone: A device where users input information.

[1609] 2. Refrigerator camera: A device that periodically takes pictures of the inside of the refrigerator.

[1610] 3. Server: A computer that stores, analyzes, and notifies data.

[1611] software

[1612] 1. TensorFlow: Machine learning framework.

[1613] 2. OpenCV: Image recognition library.

[1614] 3. Google Cloud Vision API: Image analysis solution.

[1615] 4. Azure Cognitive Services: Sentiment analysis service.

[1616] Specific processing of the program

[1617] The program of this system operates in the following manner.

[1618] 1. User Initial Registration

[1619] Users input their food consumption habits, cooking preferences, allergy information, and emotional state through a dedicated smartphone app.

[1620] The terminal sends this information to the server, which stores it in a database.

[1621] 2. Check the refrigerator inventory

[1622] The camera inside the refrigerator periodically takes pictures of the inside of the refrigerator.

[1623] The device sends the captured image data to a server, which then analyzes the image data using the Google Cloud Vision API and OpenCV to determine inventory status.

[1624] The analysis results are stored in a database.

[1625] 3. Emotion Recognition and Data Update

[1626] When a user uses the app, the emotion recognition engine analyzes emotions from the user's input, voice, and facial expressions. Emotion analysis is performed using Azure Cognitive Services.

[1627] The server stores the results of the sentiment analysis in a database and reflects them in food consumption patterns and recipe selection.

[1628] 4. Shopping list generation and notifications

[1629] The server generates a shopping list based on the user's consumption patterns, latest inventory status, and emotional information.

[1630] After generating the list, the most suitable recipe is selected and notified to the user's smartphone.

[1631] Users can access shopping lists and recipes through the app.

[1632] 5. Virtual store shopping support

[1633] Users can purchase ingredients suggested within the app at a virtual store.

[1634] The server provides a recommendation system to support the shopping experience in the virtual store.

[1635] Specific examples

[1636] For example, if a user types "I'm stressed today," the emotion analysis engine will determine the emotion "stress" and suggest herbal teas or relaxing dishes that will help reduce stress.

[1637] Example prompt sentence:

[1638] Write a Python program that recognizes emotions from text input and suggests ingredients and recipes based on them. Use the following libraries: TensorFlow, OpenCV, Google Cloud Vision API, and Azure Cognitive Services. If the user types "I'm tired today," suggest a corresponding recipe for a relaxing dish, such as chamomile tea or soup.

[1639] This system allows users to efficiently manage ingredients and purchase the ingredients they need without waste. It also suggests optimal recipes based on the user's emotional state, improving the quality and satisfaction of meals.

[1640] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1641] Program processing steps

[1642] Step 1:

[1643] Onboarding a User

[1644] Users launch the smartphone app and enter their food consumption habits, cooking preferences, allergy information, and emotional state.

[1645] Input: User's food consumption habits, cooking preferences, allergy information, and emotional state.

[1646] Data processing: Format user input information in JSON format.

[1647] The terminal transmits the input information to the server.

[1648] The server stores this data in a database.

[1649] Output: User information stored in the database.

[1650] Step 2:

[1651] Checking the inventory in the refrigerator

[1652] The camera inside the refrigerator automatically takes pictures of the inside of the refrigerator at a set time (e.g., every morning).

[1653] Input: Image data of the inside of the refrigerator.

[1654] Data processing: Image data is sent to the server in JPEG format.

[1655] The terminal transmits the captured image data to the server.

[1656] The server uses Google Cloud Vision API and OpenCV to analyze the image data and determine the inventory status in the refrigerator.

[1657] Data Computing: Image recognition algorithms are used to identify ingredients and generate inventory lists.

[1658] The analysis results are stored in a database.

[1659] Output: Refrigerator inventory information stored in a database.

[1660] Step 3:

[1661] Emotion Recognition and Data Update

[1662] While using a smartphone app, a user inputs their emotional state by typing or speaking.

[1663] Input: Text and / or audio data of the user's emotional state.

[1664] Data processing: Converting voice data into text and extracting sentiment keywords.

[1665] The terminal transmits the emotion data to the server.

[1666] The server uses Azure Cognitive Services to perform sentiment analysis and identify the user's emotional state.

[1667] Data Computing: Emotion recognition algorithms determine and store emotional states.

[1668] Output: User's emotional state information stored in a database.

[1669] Step 4:

[1670] Shopping list generation and notifications

[1671] The server generates a shopping list based on stored consumption patterns, up-to-date inventory status, and emotional information.

[1672] Input: User consumption habits, refrigerator inventory, and emotional information.

[1673] Data Computing: Predicting ingredient needs using machine learning algorithms.

[1674] The generated shopping list and the optimal recipe are selected and sent to the user's device.

[1675] Output: Shopping list and suggested recipes sent to the user's device.

[1676] Step 5:

[1677] Virtual store shopping support

[1678] Users purchase ingredients at a virtual store based on a shopping list presented within the app.

[1679] Input: Shopping list and virtual store catalog information.

[1680] Data processing: Add selected ingredients to cart.

[1681] The server uses a recommendation system to support the virtual store shopping experience.

[1682] Output: Order completed in the virtual store and purchase notification sent to the user.

[1683] This system allows users to efficiently manage ingredients and purchase the ingredients they need without waste. It also suggests optimal recipes based on the user's emotional state, improving the quality and satisfaction of meals.

[1684] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1685] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1686] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1687] [Fourth embodiment]

[1688] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1689] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1690] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1691] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1692] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1693] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1694] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1695] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1696] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1697] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1698] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1699] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1700] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1701] The present invention relates to a system that learns a user's food consumption habits and refrigerator inventory status and proposes optimal shopping lists and recipes. Specific embodiments for implementing the system of the present invention will be described below.

[1702] System configuration

[1703] The system mainly includes the following components:

[1704] 1. Input Method

[1705] Using a dedicated application, users input their food consumption habits, cooking preferences, allergy information, etc. This information is used as the initial data for the system.

[1706] 2. How to store data in a database

[1707] The terminal transmits the data entered by the user to the server.

[1708] The server stores the received data in a database, which records detailed food consumption habits for each user.

[1709] 3. Filming Method

[1710] The device (refrigerator camera) periodically takes photos of the inventory status inside the refrigerator. For example, photos are taken automatically at set times, such as every morning or after meals.

[1711] 4. Means of receiving and analyzing image data

[1712] The terminal transmits the captured image data to the server.

[1713] The server analyzes the received image data using an image recognition algorithm to identify the inventory status within the refrigerator, and the analysis results are stored in a database.

[1714] 5. How to generate a shopping list

[1715] The server predicts the ingredients needed based on the user's food consumption patterns and the latest refrigerator inventory data, and generates a shopping list.

[1716] 6. Means of notifying the user terminal

[1717] The server sends the generated shopping list and suggested recipes to the user's device, where a dedicated application notifies and displays them.

[1718] 7. A way to suggest the best recipes

[1719] The server selects and suggests appropriate recipes to the user based on the user's food consumption habits and current refrigerator inventory.

[1720] Program processing

[1721] Next, the processing of the system program will be explained in natural language.

[1722] 1. User Initial Registration

[1723] Users enter their food consumption habits, cooking preferences, and allergy information through a dedicated application.

[1724] The terminal transmits these input data to the server.

[1725] The server receives the data, stores it in a database, and initializes a machine learning model to learn the user's food consumption patterns.

[1726] 2. Check the refrigerator inventory

[1727] The device (refrigerator camera) automatically takes pictures of the inside of the refrigerator at 6:00 every morning.

[1728] The terminal transmits the captured image data to the server.

[1729] The server uses image recognition algorithms to analyze the image data and determine the inventory status of the refrigerator, and the analysis results are updated in a database.

[1730] 3. Shopping list generation and notifications

[1731] The server predicts the ingredients needed at 6pm based on the user's food consumption patterns and the latest inventory status, and generates a shopping list.

[1732] The server sends the generated shopping list and recommended recipes to the user's device.

[1733] The device notifies the user through a dedicated app and displays shopping lists and recipes.

[1734] 4. User shopping and cooking

[1735] Users open the app, check their shopping list, and purchase the ingredients they need.

[1736] Users cook using recipes suggested by the app based on the ingredients they have purchased and the inventory they have in stock.

[1737] Specific examples

[1738] For example, if a user specifies that they consume milk and eggs every week, the system will learn this. While checking the refrigerator inventory each day, it will determine that there is one liter of milk and two eggs left. When the user goes shopping, the system will automatically generate a shopping list that includes "one liter of milk and six eggs" and notify the app. The system will also suggest recipes such as omelets and French toast based on the ingredients in the refrigerator.

[1739] In this way, the system of the present invention allows users to avoid forgetting to buy ingredients or wasting them, and allows them to shop and cook efficiently.

[1740] The processing flow will be explained below.

[1741] Step 1:

[1742] Users install a dedicated application and enter their food consumption habits, cooking preferences, and allergy information, specifically the ingredients they use each week and the dishes they make on specific days.

[1743] Step 2:

[1744] The device sends the user's input data to the server, including the frequency of food consumption, type of cuisine, and a list of ingredients that may cause allergies.

[1745] Step 3:

[1746] The server stores the received data in a database, where the data is managed separately for each user.

[1747] Step 4:

[1748] The server uses the stored data to initialize a machine learning model and learns the user's food consumption patterns. For example, the model can learn that the user uses eggs every Monday.

[1749] Step 5:

[1750] The device (refrigerator camera) automatically takes a photo of the inside of the refrigerator every morning at 6:00 or at a time specified by the user.

[1751] Step 6:

[1752] The device sends the captured image data to the server. This image must show the entire interior of the refrigerator.

[1753] Step 7:

[1754] The server then uses an image recognition algorithm to analyze the received image data and identify the types and quantities of food items in the refrigerator, such as whether there is one liter of milk left or two eggs.

[1755] Step 8:

[1756] The server updates the analysis results and stores them in a database, which is updated in real time.

[1757] Step 9:

[1758] The server generates a shopping list every day at 6pm based on the user's food consumption patterns and the latest inventory information. For example, one liter of milk and six eggs are added to the list.

[1759] Step 10:

[1760] The server generates a shopping list and selects the best recipes based on the user's preferences, such as omelet and French toast recipes.

[1761] Step 11:

[1762] The server sends the shopping list and recipes to the user's device.

[1763] Step 12:

[1764] The device will display a notification to the user through a dedicated application, and the user can open the app to view the displayed shopping list and recipes.

[1765] Step 13:

[1766] The user goes to the supermarket and buys the ingredients they need based on the shopping list, for example, adding 1 liter of milk and 6 eggs to their cart.

[1767] Step 14:

[1768] The app suggests recipes based on the ingredients purchased and the inventory in the refrigerator, allowing users to cook dishes such as omelets and French toast.

[1769] Example 1

[1770] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1771] Conventional refrigerator management systems require users to manually manage food inventory, which is time-consuming and can lead to forgotten purchases and wasteful consumption. It is also difficult to provide effective shopping lists and recipes that take into account the user's food consumption habits. This leads to inefficiencies in the user's lifestyle and wastes time and energy.

[1772] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1773] In this invention, the server includes input means for inputting a user's ingredient consumption habits and cooking preferences, means for receiving information from the input means and storing the information in a database, photographing means for automatically photographing images of the inside of the refrigerator periodically, means for receiving and analyzing image data obtained by the photographing means, means for generating a shopping list based on the analyzed inventory information and the stored consumption habits, means for notifying a user terminal of the generated shopping list, means for using a machine learning algorithm to predict optimal ingredients based on the suggested shopping list, and means for suggesting recipes based on the optimal ingredients and the user's consumption habits, thereby enabling users to efficiently manage ingredient consumption and inventory without hassle.

[1774] "User" refers to a person who uses the system to input their food consumption habits and cooking preferences and receive suggested shopping lists and recipes.

[1775] "Food consumption habits" refers to data that indicates the tendency and frequency of food ingredients consumed by a user during a specific period.

[1776] "Cooking preferences" refers to information about the user's favorite dishes and cooking methods.

[1777] "Input means" refers to an interface that allows a user to input their food consumption habits and cooking preferences into the system.

[1778] "Database" refers to an information system for storing information received from users, analysis results, inventory information, etc.

[1779] "Photographing means" refers to a device or equipment that automatically takes pictures of the inside of the refrigerator on a regular basis.

[1780] "Image data" refers to photographs and video data of the inside of the refrigerator captured by a photographing means.

[1781] "Analysis means" refers to a system or method that uses an image recognition algorithm to analyze captured image data and identify inventory information within the refrigerator.

[1782] "Inventory information" refers to data about the ingredients currently in the refrigerator and their quantities.

[1783] A "shopping list" is a list created by predicting the ingredients a user will need in the future.

[1784] "Notification means" refers to a system or method for sending the generated shopping list and suggested recipes to a user terminal.

[1785] "Machine learning algorithm" refers to a calculation method or program that learns from a user's food consumption habits and refrigerator inventory information to predict the best ingredients.

[1786] "Recipe suggestion means" refers to a system or method for providing a user with a recipe suitable for cooking based on optimal ingredients and the user's consumption habits.

[1787] The present invention relates to a system that learns a user's food consumption habits and refrigerator inventory status, and proposes optimal shopping lists and recipes. This system is implemented using the following hardware and software.

[1788] 1. Hardware Configuration

[1789] The system includes the following hardware:

[1790] User device: Mobile device such as a smartphone or tablet

[1791] Refrigerator camera: A camera that periodically takes pictures of the inside of a refrigerator

[1792] Server: A computer system that runs a database and image recognition and machine learning algorithms.

[1793] 2. Software Configuration

[1794] The system includes the following software:

[1795] Dedicated application: An app for users to input information about food consumption habits, cooking preferences, and allergies

[1796] Database Management Systems: Databases such as MySQL or PostgreSQL

[1797] Image recognition algorithm: Object detection algorithm using OpenCV and TensorFlow

[1798] Machine learning algorithms: predictive models using scikit-learn and TensorFlow

[1799] 3. System Operation

[1800] Onboarding a User

[1801] Users enter their food consumption habits, cooking preferences, allergy information, etc. through a dedicated application. This information is used as the initial data for the system.

[1802] The terminal transmits the data entered by the user to the server.

[1803] The server stores the received data in a database and initializes and starts training a machine learning model to learn the user's food consumption patterns.

[1804] Checking the inventory in the refrigerator

[1805] The device (refrigerator camera) automatically takes pictures of the inside of the refrigerator at a fixed time every day. This time can be set by the user.

[1806] The terminal transmits the captured image data to the server.

[1807] The server analyzes the image data using an image recognition algorithm to identify the inventory status of the refrigerator, and the analysis results are reflected in a database.

[1808] Shopping list generation and notifications

[1809] The server predicts the ingredients needed based on the user's ingredient consumption patterns and the latest inventory status, and generates a shopping list.

[1810] The server sends the generated shopping list to the user's device, and a dedicated app notifies the user and displays the list to them.

[1811] Recipe Suggestions

[1812] The server selects and suggests appropriate recipes based on the user's consumption habits and refrigerator inventory. Recipe suggestions are made via a dedicated app.

[1813] 4. Specific Examples

[1814] For example, if a user specifies that they "eat eggs and bread for breakfast every day," the system learns this and predicts the ingredients they will need based on their consumption patterns over the course of a week. A camera inside the refrigerator checks the inventory every day and generates a shopping list based on the analyzed results. When the user opens the app on their smartphone, a shopping list of "10 eggs and 1 bag of bread" is displayed. The system also suggests recipes such as "French toast" and "omelette" based on the inventory status in the refrigerator.

[1815] Prompt Sentence Examples

[1816] "Can you recommend a simple and delicious recipe for dinner? I have eggs and milk in the fridge."

[1817] In this way, the system of the present invention allows users to receive efficient support regarding the management and consumption of ingredients, preventing them from forgetting to buy food or wasting it.

[1818] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1819] Step 1: Onboarding users

[1820] Users open a dedicated app and enter data such as food consumption habits, cooking preferences, and allergy information.

[1821] Input: Food consumption habits, cooking preferences, allergy information

[1822] Specific action: The user enters the required information into a form within the app and presses the submit button.

[1823] The terminal receives input data and transmits it to the server.

[1824] Output: Sending user-entered data

[1825] The server stores the received data in a database.

[1826] Input: User's consumption habits, food preferences, and allergy information

[1827] Data processing: Store information in the database in JSON format and organize it by user ID.

[1828] Output: Database update

[1829] The server initializes the machine learning model with the user's consumption patterns and performs initial training.

[1830] Input: Initial database information

[1831] Data Computing: Model training with machine learning algorithms

[1832] Output: The model after initial training

[1833] Step 2: Take a photo of the refrigerator

[1834] The device (refrigerator camera) automatically takes pictures of the inside of the refrigerator at 6:00 every morning.

[1835] What it does: The camera's timer will activate and take images of each shelf inside the refrigerator.

[1836] Input: Real-time video of the inside of the refrigerator

[1837] Output: Captured image data

[1838] The terminal transmits the captured image data to the server.

[1839] Specific operation: Compresses image data and sends it to the server via an HTTP POST request.

[1840] Input: Captured image

[1841] Output: Image data sent to the server

[1842] Step 3: Sending and analyzing image data

[1843] The server analyzes the received image data.

[1844] Input: Image data sent from the device

[1845] Specific operation: The server uses Python's OpenCV library to apply image recognition algorithms.

[1846] Data processing: Perform object detection from image data to identify inventory.

[1847] Output: Inventory information as analysis result

[1848] The server stores the analysis results in a database and updates inventory data.

[1849] Input: Parsed inventory information

[1850] Data operation: Update the inventory table in the database.

[1851] Output: Updated database

[1852] Step 4: Create a shopping list

[1853] The server generates a shopping list based on the user's consumption patterns and the latest inventory information.

[1854] Input: Latest database information

[1855] Data Computing: Using machine learning models to predict food shortages.

[1856] Specific operation: Formalize the model's prediction results as a shopping list.

[1857] Output: Generated shopping list

[1858] Step 5: Shopping List and Recipe Notifications

[1859] The server notifies the user's terminal of the generated shopping list and recommended recipes.

[1860] Input: Shopping list, recommended recipes

[1861] Specific operation: Sends a push notification from the server and sends data to the API endpoint.

[1862] Output: Notification data sent to the user's device

[1863] The device notifies the user through a dedicated app and displays shopping lists and recipes.

[1864] Input: Notification data from the server

[1865] What it does: Your device will display a push notification and make your lists and recipes available in the app.

[1866] Output: User confirmation

[1867] Step 6: User Shopping and Cooking

[1868] Users open the app, check their shopping list and purchase the ingredients they need.

[1869] Action: Open the app, browse the list and make a purchase.

[1870] Input: Shopping List

[1871] Output: Purchased ingredients

[1872] Users cook using recipes suggested by the app based on the ingredients they have purchased and the inventory they have in stock.

[1873] Specific actions: Cook while looking at the recipe displayed on the app.

[1874] Input: Recommended Recipe

[1875] Output: Cooked food

[1876] (Application example 1)

[1877] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1878] Conventional refrigerator inventory management and food consumption management systems required users to check inventory and manually create shopping lists, which was time-consuming and often led to unnecessary food purchases or shortages. Furthermore, because there was no integration with delivery services, users had to go through the trouble of ordering separately. Furthermore, there was a lack of a mechanism for efficiently learning users' food consumption patterns, making it difficult to suggest appropriate recipes.

[1879] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1880] In this invention, the server includes an input means for inputting the user's ingredient consumption habits and cooking preferences, a photographing means for automatically taking periodic images of the inside of the refrigerator, and a means for automatically placing orders with an affiliated food delivery service. This automates refrigerator inventory management, reducing the user's workload and preventing shortages and unnecessary purchases of ingredients. Furthermore, the server automatically connects with the delivery service, allowing the user to quickly order the necessary ingredients. Furthermore, the server facilitates appropriate recipe suggestions based on the user's ingredient consumption patterns, enhancing user convenience.

[1881] A "user" is an individual or entity that uses the system to input their food consumption habits and cooking preferences.

[1882] "Input means" refers to a device or software that allows a user to input their food consumption habits and cooking preferences.

[1883] A "means for storing in a database" is a process and system that receives information from an input means and stores the information in a database.

[1884] The "photography means" refers to a camera and its control device that automatically captures images of the inside of the refrigerator on a regular basis.

[1885] The "analysis means" refers to hardware and software for receiving and analyzing image data obtained by the imaging means.

[1886] The "means for generating a shopping list" is a mechanism for creating a shopping list based on the analyzed inventory information and the stored consumption habits.

[1887] The "notification means" is a mechanism for transmitting the generated shopping list to the user terminal and informing the user.

[1888] The "means for suggesting recipes" is a system for suggesting optimal cooking recipes to a user based on a shopping list.

[1889] "Means for automatically placing orders with affiliated food delivery services" refers to a system that automatically sends an order to an affiliated food delivery service when necessary ingredients are in short supply.

[1890] This invention relates to a system that learns a user's food consumption habits and refrigerator inventory status, proposes optimal shopping lists and recipes, and even automatically orders the necessary ingredients. Specific embodiments for carrying out this invention are described below.

[1891] System configuration

[1892] The system includes the following components:

[1893] 1. Input Method

[1894] Users use a dedicated application to input their food consumption habits, cooking preferences, and allergy information.

[1895] 2. How to store data in a database

[1896] The terminal sends the input data from the user to the server.

[1897] The server stores the received data in a database, recording each user's detailed food consumption habits.

[1898] 3. Filming Method

[1899] A camera installed inside the refrigerator periodically takes pictures of the inventory inside the refrigerator.

[1900] 4. Analysis method

[1901] The terminal transmits the captured image data to the server.

[1902] The server uses image recognition algorithms to analyze the image data and identify the inventory status of the refrigerator, and the analysis results are stored in a database.

[1903] 5. How to generate a shopping list

[1904] The server predicts the ingredients needed based on the user's food consumption patterns and the latest refrigerator inventory data, and generates a shopping list.

[1905] 6. Means of Notification

[1906] The server sends the generated shopping list and suggested recipes to the user's device, where a dedicated application notifies and displays them.

[1907] 7. Proposed measures

[1908] The server selects and suggests appropriate recipes to users based on their consumption habits and inventory status.

[1909] 8. How to Order

[1910] The server automatically places orders with affiliated food delivery services.

[1911] Program processing

[1912] Next, we explain in natural language the program processing required to implement the system. The main hardware used is a camera installed inside the refrigerator, the user's smartphone, and a server. The software used is Python, OpenCV (image recognition library), Django (web framework), and TensorFlow (machine learning model).

[1913] Onboarding a User

[1914] Users input their food consumption habits, cooking preferences, and allergy information through a dedicated application. The device sends this data to the server, which receives it and stores it in a database. The server then initializes a machine learning model to learn the user's food consumption patterns.

[1915] Checking the inventory in the refrigerator

[1916] A camera installed inside the refrigerator automatically takes pictures of the inside of the refrigerator at 6:00 every morning. The device sends the captured image data to a server, which then analyzes the image data using an image recognition algorithm (OpenCV). The inventory status inside the refrigerator is identified and the results are updated in the database.

[1917] Shopping list generation and notifications

[1918] The server predicts what ingredients a user will need before dinner based on their food consumption patterns and the latest inventory status, and generates a shopping list. The generated shopping list and recommended recipes are sent to the user's device, and a dedicated app notifies the user.

[1919] Automated ordering and delivery

[1920] If a specific ingredient is in short supply, the server automatically places an order with a partner food delivery service, calls an API, and notifies the user of the order confirmation and delivery status.

[1921] Examples of concrete examples and prompts

[1922] For example, suppose a user initially registers their weekly consumption of milk and eggs. If the inventory in the refrigerator drops to 1 liter of milk and 2 eggs, the system will generate a shopping list for "1 liter of milk and 6 eggs," place an automatic order, and deliver the ingredients by 10:00 a.m. the next day. It will also suggest "omelette and French toast" as recipe suggestions.

[1923] Example prompt for a generative AI model:

[1924] You will help design an application that automatically orders ingredients and suggests the best grocery shopping list and recipes based on the user's food consumption habits and refrigerator inventory. Specifically, your functional requirements will be as follows:

[1925] 1. Initial User Registration

[1926] 2. Refrigerator inventory management and regular checks

[1927] 3. Learning food consumption patterns

[1928] 4. Optimal recipe suggestions

[1929] 5. Automated ordering and food delivery integration

[1930] Thank you very much for your cooperation.

[1931] In this way, the system of the present invention eliminates the need for users to perform cumbersome inventory checks or manually order ingredients, realizing convenient and efficient ingredient management.

[1932] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1933] Step 1:

[1934] The user inputs their food consumption habits and cooking preferences.

[1935] Specific operation: The user uses a dedicated application to input information such as food consumption habits, cooking preferences, and allergy information.

[1936] Input: User's food consumption habits, cooking preferences, allergy information, etc.

[1937] Output: User data stored in a dedicated application.

[1938] Step 2:

[1939] The terminal transmits the input data to the server.

[1940] Specific operation: The application sends the user's input data to the cloud server.

[1941] Input: User's food consumption habits, cooking preferences, allergy information, etc.

[1942] Output: User data stored on the server.

[1943] Step 3:

[1944] The server stores the received data in a database.

[1945] Specific operation: The server stores the received data in a database management system (e.g., MySQL).

[1946] Input: User's food consumption habits, cooking preferences, allergy information, etc.

[1947] Output: Detailed user data stored in a database.

[1948] Step 4:

[1949] A camera installed inside the refrigerator periodically takes pictures of the inventory inside the refrigerator.

[1950] Specific operation: The camera inside the refrigerator automatically takes a photo of the inside of the refrigerator at 6:00 every morning.

[1951] Input: Current state inside the refrigerator.

[1952] Output: Image data of the inside of the refrigerator.

[1953] Step 5:

[1954] The terminal transmits the captured image data to the server.

[1955] Specific operation: The camera inside the refrigerator or its control device sends the captured image data to a cloud server.

[1956] Input: Image data of the inside of the refrigerator.

[1957] Output: Image data stored on the server.

[1958] Step 6:

[1959] The server analyzes the image data using an image recognition algorithm.

[1960] Specific operation: The server uses an image recognition library such as OpenCV to analyze image data from inside the refrigerator and determine the inventory status.

[1961] Input: Image data of the inside of the refrigerator.

[1962] Output: Parsed inventory information.

[1963] Step 7:

[1964] The server stores and updates the analysis results in a database.

[1965] Specific operation: The server saves the parsed inventory information in the database and updates the existing inventory information.

[1966] Input: Parsed inventory information.

[1967] Output: The updated database.

[1968] Step 8:

[1969] The server predicts the ingredients needed and generates a shopping list based on the user's food consumption patterns and the latest refrigerator inventory data.

[1970] Specific operation: The server learns the user's past consumption patterns and compares them with current inventory data to generate a shopping list.

[1971] Input: User's food consumption patterns, latest refrigerator inventory data.

[1972] Output: Shopping list.

[1973] Step 9:

[1974] The server sends the generated shopping list and recommended recipes to the user's terminal.

[1975] Specific operation: The server sends the generated shopping list and recipes to a dedicated app and notifies it.

[1976] Input: shopping list, recommended recipes.

[1977] Output: The shopping list and recipes displayed on the user's device.

[1978] Step 10:

[1979] If the server runs out of a particular ingredient, it will automatically place an order with an affiliated food delivery service.

[1980] Specific operation: If the server is short on necessary ingredients, it calls the API of the partner delivery service and automatically places the order.

[1981] Input: Missing ingredient information.

[1982] Output: The order sent to the food delivery service.

[1983] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1984] The present invention relates to a system that learns a user's food consumption habits and refrigerator inventory status, and proposes optimal shopping lists and recipes, and further includes a function to recognize the user's emotions and adjust the system's operation based on the recognition results. Specific embodiments for implementing the system of the present invention are described below.

[1985] System configuration

[1986] The system mainly includes the following components:

[1987] 1. Input Method

[1988] Using a dedicated application, users input their food consumption habits, cooking preferences, allergy information, emotional state, etc. This information is used as the initial data for the system.

[1989] 2. How to store data in a database

[1990] The terminal transmits the data entered by the user to the server.

[1991] The server stores the received data in a database, which records detailed food consumption habits and emotional information for each user.

[1992] 3. Filming Method

[1993] The device (refrigerator camera) periodically takes photos of the inventory status inside the refrigerator. For example, photos are taken automatically at set times, such as every morning or after meals.

[1994] 4. Means of receiving and analyzing image data

[1995] The terminal transmits the captured image data to the server.

[1996] The server analyzes the received image data using an image recognition algorithm to identify the inventory status within the refrigerator, and the analysis results are stored in a database.

[1997] 5. Emotion Engine

[1998] The server includes an emotion engine that recognizes the user's emotional state through user input and interaction. The emotion engine analyzes emotions from text analysis, speech recognition, and user facial expressions.

[1999] 6. How to generate a shopping list

[2000] The server predicts the ingredients needed and generates a shopping list based on the user's food consumption patterns, the latest refrigerator inventory data, and their emotional state.

[2001] 7. Means of notifying the user terminal

[2002] The server sends the generated shopping list and suggested recipes to the user's device, where a dedicated application notifies and displays them.

[2003] 8. A way to suggest the best recipes

[2004] The server selects and suggests appropriate recipes to the user based on the user's food consumption habits, current refrigerator inventory, and emotional information.

[2005] Program processing

[2006] Next, the processing of the system program will be explained in natural language.

[2007] 1. User Initial Registration

[2008] Users enter their food consumption habits, cooking preferences, allergy information, and emotional state through a dedicated application.

[2009] The terminal transmits these input data to the server.

[2010] The server receives the data and stores it in a database, which is managed for each user and used as the initial data for the machine learning model and emotion engine.

[2011] 2. Check the refrigerator inventory

[2012] The device (refrigerator camera) automatically takes pictures of the inside of the refrigerator at a set time every morning.

[2013] The terminal transmits the captured image data to the server.

[2014] The server uses image recognition algorithms to analyze the image data and identify the inventory status of the refrigerator, and the analysis results are stored in a database.

[2015] 3. Emotion Recognition and Data Update

[2016] When a user uses the dedicated app, the emotion engine recognizes emotions from the user's input, voice, and facial expressions.

[2017] The server stores the emotional information analyzed by the emotion engine in a database and reflects it in the user's consumption patterns and recipe selection.

[2018] 4. Shopping list generation and notifications

[2019] The server predicts the ingredients needed based on the user's food consumption patterns, the latest inventory status, and emotional information, and generates a shopping list at a set time in the evening.

[2020] The server selects optimal recipes based on the user's preferences and emotional state along with the generated shopping list and sends them to the user's device.

[2021] The device displays notifications to the user through a dedicated app, allowing the user to check their shopping list and recipes.

[2022] 5. User shopping and cooking

[2023] Users can purchase the ingredients they need at the supermarket based on their shopping list. When making a purchase, the system also suggests ingredients and recipes that will boost their mood based on the "recommended mood" provided by the emotion engine.

[2024] The app suggests recipes based on the ingredients purchased and the inventory in the refrigerator, allowing users to cook. For example, they can make herbal tea to relax or a nutritious meal to energize.

[2025] The system of the present invention allows users to avoid forgetting to buy ingredients and avoiding waste, allowing them to shop and cook efficiently. It also suggests optimal recipes and ingredients based on the user's emotional state, improving the quality and satisfaction of meals.

[2026] The processing flow will be explained below.

[2027] Step 1:

[2028] Users install a dedicated application and input their food consumption habits, cooking preferences, allergy information, and emotional state, such as the ingredients they use each week, the dishes they make on specific days, and their current mood (e.g., energetic, tired, or wanting to relax).

[2029] Step 2:

[2030] The device sends the user's input data to the server, including the frequency of food consumption, type of cuisine, a list of ingredients that may cause allergies, and the user's emotional information.

[2031] Step 3:

[2032] The server stores the received data in a database, where it manages detailed information about each user's food consumption habits and emotions.

[2033] Step 4:

[2034] The server uses the saved data to initialize the machine learning model and emotion engine, learning the user's food consumption patterns and emotional patterns. For example, the model can register the user's habit of eating eggs every Monday and their emotional reactions to certain ingredients.

[2035] Step 5:

[2036] The device (refrigerator camera) automatically takes a photo of the inside of the refrigerator every morning at a set time or at a time specified by the user.

[2037] Step 6:

[2038] The device sends the captured image data to the server. This image must show the entire interior of the refrigerator.

[2039] Step 7:

[2040] The server then uses an image recognition algorithm to analyze the received image data and identify the types and quantities of food items in the refrigerator, such as whether there is one liter of milk left or two eggs.

[2041] Step 8:

[2042] The server updates the analysis results and stores them in a database, which is updated in real time.

[2043] Step 9:

[2044] The server generates a shopping list for a specific time in the evening based on the user's food consumption patterns, latest inventory information, and emotional information. For example, one liter of milk and six eggs are added to the list.

[2045] Step 10:

[2046] The server then generates a shopping list and selects the optimal recipe based on the user's preferences and emotional state. For example, if the user is looking to relax, it will suggest recipes using herbal teas and ingredients with relaxing effects.

[2047] Step 11:

[2048] The server sends the shopping list and recipes to the user's device.

[2049] Step 12:

[2050] The device will display a notification to the user through a dedicated application, and the user can open the app to view the displayed shopping list and recipes.

[2051] Step 13:

[2052] Users can purchase the ingredients they need at the supermarket based on their shopping list. At the time of purchase, the emotion engine also suggests ingredients and recipes that will boost their mood based on the "recommended mood" provided by the emotion engine.

[2053] Step 14:

[2054] The app suggests recipes based on the ingredients purchased and the stock in the refrigerator, allowing users to cook. For example, if a user is tired, they can choose a dish that will restore stamina or a dish that is easy to make.

[2055] In this way, this system incorporating an emotion engine helps users avoid forgetting to buy ingredients and avoiding waste, allowing them to shop and cook efficiently. Furthermore, since suggestions are made that take into account the user's emotional state, it can improve the quality and satisfaction of meals.

[2056] Example 2

[2057] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2058] In today's world, many people lead busy lives, and food waste due to forgetting to buy ingredients or expiry dates is a major problem. It is also not easy to find optimal recipes based on one's own food consumption habits and preferences. Furthermore, there are few systems that provide meal suggestions that take into account daily emotional states, and there is a need for systems that can help improve the quality and satisfaction of meals.

[2059] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[2060] In this invention, the server includes: means for a user to input information about food ingredient consumption habits, cooking preferences, allergy information, and emotional state; means for receiving information from the input means using a data transmission means and storing it in a database; means for automatically taking periodic images of the inside of the refrigerator; means for receiving image data obtained by the image taking means and analyzing it using an image recognition algorithm; means for generating a shopping list based on the analyzed inventory information and the consumption habits and emotional information stored in the database; means for notifying a user terminal of the generated shopping list and optimal recipes; and means for analyzing the user's emotional state using an emotion engine and reflecting the analysis results in system operation. This allows the user to use ingredients efficiently and improve the quality and satisfaction of their meals by receiving optimal recipes based on their emotional state.

[2061] "Food consumption habits" is information about how often and what types of food ingredients a user consumes within a certain period of time.

[2062] "Cooking preferences" is information relating to the types of dishes, characteristics of ingredients, and cooking methods that the user particularly likes.

[2063] "Allergy information" is information about ingredients or substances to which the user is allergic.

[2064] An "emotional state" is information about the sensations or moods a user is experiencing at a particular point in time.

[2065] "Data transmission means" is a communication means for transmitting input information to a server.

[2066] "Database" refers to a data storage system for storing a user's food consumption habits, cooking preferences, allergy information, and emotional state.

[2067] An "image recognition algorithm" is a computer program that analyzes captured image data and identifies objects and characters contained in the image.

[2068] An "emotion engine" is a system for analyzing a user's emotional state from input text, voice, facial expressions, etc.

[2069] A "shopping list" is a list of ingredients to be purchased that is generated based on the current inventory in the refrigerator, the user's ingredient consumption habits, and their emotional state.

[2070] A "recipe" is a description of the steps and ingredients needed to make a dish.

[2071] A "user terminal" is an electronic device that allows a user to access the system and input or receive information.

[2072] The "means for automatically taking images of the inside of the refrigerator" is a system that periodically takes photos of the inventory status inside the refrigerator using a camera.

[2073] A "machine learning algorithm" is a technology that learns a user's food consumption habits and emotional information and predicts future consumption patterns.

[2074] "Analysis results" are analytical data obtained by the server using image recognition algorithms and emotion engines.

[2075] The present invention relates to a system that learns a user's food consumption habits and refrigerator inventory status, and proposes optimal shopping lists and recipes, and further includes a function that recognizes the user's emotions and adjusts the system's operation based on the recognition results. Specific embodiments for carrying out the present invention will be described below.

[2076] The system is composed of hardware and software such as a dedicated application, an in-fridge camera, an image recognition algorithm, a machine learning algorithm, an emotion engine, a database, a server, and a user terminal.

[2077] First, the user installs a dedicated application and inputs initial information such as their food consumption habits, cooking preferences, allergy information, and emotional state. This information is sent from the user's terminal to the server using a data transmission means, and the server stores the received information in a database. For example, users might enter information such as "I eat chicken once a week," "I like Italian food," "I have a nut allergy," and "I want to relax."

[2078] To understand the inventory status in the refrigerator, the device (in-fridge camera) automatically takes pictures of the inside of the refrigerator at a set time every morning. The captured image data is sent from the device to a server. The server analyzes the image data using an image recognition algorithm such as Google Cloud Vision API to determine the inventory status in the refrigerator. For example, it may determine that there are three tomatoes, one bag of pasta, and two pieces of cheese in the refrigerator. The analysis results are stored in a database.

[2079] When a user uses the dedicated app, the emotion engine analyzes the user's input text, voice, and facial expressions in real time. For example, if a user inputs "I'm not feeling well," the text analysis identifies that emotion. The server stores the emotion engine's analysis results in a database and reflects them in the user's consumption patterns and recipe selection.

[2080] The server predicts the ingredients needed based on the user's food consumption patterns, the inventory status in the refrigerator, and emotional information. For example, if the user inputs "I use chicken twice a week" but there is no chicken in the refrigerator, the server will add more chicken. The server generates a shopping list at a set time in the evening. Along with the generated shopping list, it selects the optimal recipe based on the user's preferences and emotional state. For example, if the user inputs "I want to relax," a recipe for herbal tea with a relaxing effect will be selected. The server sends the shopping list and recipe to the user's device and displays a notification to the user through a dedicated app.

[2081] Based on the shopping list, the user purchases the ingredients they need at the supermarket. When making a purchase, the app also suggests ingredients and recipes that will boost their mood based on the "recommended mood" provided by the emotion engine. For example, it might suggest, "I feel like relaxing today, so why not try some cardamom tea?" The user then cooks a meal using recipes suggested by the app based on the ingredients they purchased and the inventory in their refrigerator. For example, they could make chicken and basil pasta using the tomatoes, pasta, and cheese in their refrigerator.

[2082] This system allows users to shop and cook efficiently, preventing them from forgetting to buy ingredients or wasting them. It also suggests optimal recipes and ingredients based on the user's emotional state, improving the quality and satisfaction of meals.

[2083] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2084] Step 1: Onboarding users

[2085] The user launches a dedicated application and inputs initial information such as food consumption habits, cooking preferences, allergy information, and emotional state.

[2086] For example, you can enter information such as "I eat chicken once a week," "I like Italian food," "I have a nut allergy," or "I want to relax."

[2087] Input data: food consumption habits, cooking preferences, allergy information, emotional state

[2088] The terminal transmits the input data to the server.

[2089] Data transmission: From user device to server

[2090] The server stores the received data in a database and manages it for each user.

[2091] Data storage: The server stores data in a database

[2092] Step 2: Check the refrigerator inventory

[2093] The device (refrigerator camera) automatically takes pictures of the inside of the refrigerator at a set time every morning.

[2094] For example, capture an image of the inside of your refrigerator at 8am.

[2095] Input data: Image data of the inside of the refrigerator

[2096] The terminal transmits the captured image data to the server.

[2097] Data transmission: From the camera inside the refrigerator to the server

[2098] The server uses image recognition algorithms to analyze the image data and determine the inventory status within the refrigerator.

[2099] Data analysis: Using image recognition algorithms such as Google Cloud Vision API

[2100] For example: "Identify that there are 3 tomatoes, 1 bag of pasta, and 2 cheeses in the refrigerator."

[2101] Analysis result: Inventory information (3 tomatoes, 1 bag of pasta, 2 cheeses)

[2102] The server stores the analysis results in a database.

[2103] Data storage: The server stores data in a database

[2104] Step 3: Emotion recognition and data update

[2105] When a user uses the dedicated app, the emotion engine analyzes the user's input text, voice, and facial expressions in real time.

[2106] For example, if a user types "I'm feeling down," text analysis will identify that emotion.

[2107] Input data: text, voice, facial expression data

[2108] The server obtains the analysis results of the emotion engine.

[2109] Data analysis: Emotion engine uses text analysis, speech recognition, and facial expression recognition

[2110] Example: "The user wants to relax."

[2111] Analysis results: Emotional state information

[2112] The server stores the analyzed emotional information in a database and reflects it in the user's consumption patterns and recipe selection.

[2113] Data storage: The server stores data in a database

[2114] Step 4: Create a shopping list

[2115] The server predicts the ingredients needed based on the user's food consumption patterns, the inventory status in the refrigerator, and emotional information.

[2116] For example, if a user inputs "I use chicken twice a week" but there is no chicken in the refrigerator, new chicken will be added.

[2117] Input data: food consumption patterns, refrigerator inventory, emotional information

[2118] The server generates a shopping list at a fixed time in the evening.

[2119] Data generation: Shopping list (e.g. chicken, basil)

[2120] Generated data: Shopping list (chicken, basil)

[2121] Step 5: Recipe suggestions and notifications

[2122] The server generates a shopping list and selects the best recipe based on the user's preferences and emotional state.

[2123] For example, if the user types "I want to relax," a recipe for a relaxing herbal tea will be selected.

[2124] Input data: shopping list, food consumption habits, emotional information

[2125] The server sends the shopping list and recipes to the user's device.

[2126] Data transmission: From the server to the user's device

[2127] The device will display a notification to the user through a dedicated app.

[2128] Notifications: Shopping Lists and Recipes

[2129] For example: "Your shopping list has been updated. Please purchase chicken and basil. My recommended recipe is chicken and basil pasta."

[2130] Step 6: User Shopping and Cooking

[2131] The user purchases the necessary ingredients at the supermarket based on the shopping list.

[2132] For example, buy chicken and basil according to the list.

[2133] The emotional engine suggests ingredients and recipes that will boost your mood based on your "recommended mood."

[2134] For example, suggest, "I'm feeling relaxed today, how about trying some cardamom tea?"

[2135] Users cook using recipes suggested by the app based on the ingredients they have purchased and what is in their refrigerator.

[2136] For example, make chicken and basil pasta using the tomatoes, pasta, and cheese in your fridge.

[2137] Through these steps, the system prevents users from forgetting to buy ingredients or wasting them, enabling them to shop and cook efficiently. It also improves the quality and satisfaction of meals by suggesting optimal recipes and ingredients based on the user's emotional state.

[2138] (Application example 2)

[2139] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2140] In today's busy daily lives, it is extremely difficult for users to effectively manage their food consumption habits and refrigerator inventory. In addition, there is a need for systems that provide optimal shopping lists and recipes while taking into account the user's emotional state while preparing meals. However, conventional systems are unable to incorporate information, including the user's emotional state, making it difficult to prevent unnecessary food purchases and food waste. Furthermore, the virtual store's purchasing experience is often insufficient, causing user stress. A system that solves these problems is needed.

[2141] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[2142] In this invention, the server includes a means for allowing a user to input information about their food ingredient consumption habits, cooking preferences, allergy information, and emotional state; a means for automatically taking periodic images of the inside of the refrigerator; a means for receiving and analyzing the image data obtained by the image taking means; a means for analyzing emotions from the user's input, voice, and facial expressions using an emotion recognition engine; and a means for supporting the shopping experience in the virtual store. This allows the user to significantly reduce food waste and shop and cook efficiently. Furthermore, the server can suggest ingredients and provide recipes based on the user's emotional state, further improving the shopping experience in the virtual store.

[2143] Definitions of important words

[2144] "User" refers to an individual who uses the system to manage ingredients and receive recipe suggestions.

[2145] "Food consumption habits" refers to the types, amounts, and frequency of food ingredients that a user consumes on a daily basis.

[2146] "Cooking preferences" refers to the types and seasonings of food that a user likes.

[2147] "Allergy information" refers to foods or ingredients to which the user is allergic.

[2148] "Emotional state" refers to the user's emotional condition, such as stress or relaxation.

[2149] "Input means" refers to an interface through which a user inputs information into a system.

[2150] "Photographing means" refers to a device or function that automatically photographs images inside the refrigerator.

[2151] "Image data" refers to a photograph of the inside of the refrigerator taken by a photographing means.

[2152] "Analysis means" refers to a device or function that includes an algorithm for analyzing image data and identifying the inventory status and ingredients in the refrigerator.

[2153] An "emotion recognition engine" refers to technology or software for analyzing emotions from user input, voice, and facial expressions.

[2154] A "shopping list" is a list of all the ingredients you need.

[2155] "User terminal" refers to a device through which a user interfaces with the system.

[2156] "Notification means" refers to a technique or device for notifying the user of the generated shopping list or recipe.

[2157] "Recipe suggestion means" refers to a device or function that suggests optimal recipes based on a shopping list.

[2158] A "virtual store" refers to a virtual shopping environment for purchasing food ingredients over the Internet.

[2159] "Supporting Means" refers to technology or devices that support the virtual store shopping experience.

[2160] The present invention relates to a system that learns a user's food consumption habits and refrigerator inventory status, recognizes the user's emotions, and proposes optimal shopping lists and recipes. Specific embodiments for implementing the system of the present invention are described below.

[2161] System configuration

[2162] The system mainly uses the following hardware and software:

[2163] Hardware

[2164] 1. Smartphone: A device where users input information.

[2165] 2. Refrigerator camera: A device that periodically takes pictures of the inside of the refrigerator.

[2166] 3. Server: A computer that stores, analyzes, and notifies data.

[2167] software

[2168] 1. TensorFlow: Machine learning framework.

[2169] 2. OpenCV: Image recognition library.

[2170] 3. Google Cloud Vision API: Image analysis solution.

[2171] 4. Azure Cognitive Services: Sentiment analysis service.

[2172] Specific processing of the program

[2173] The program of this system operates in the following manner.

[2174] 1. User Initial Registration

[2175] Users input their food consumption habits, cooking preferences, allergy information, and emotional state through a dedicated smartphone app.

[2176] The terminal sends this information to the server, which stores it in a database.

[2177] 2. Check the refrigerator inventory

[2178] The camera inside the refrigerator periodically takes pictures of the inside of the refrigerator.

[2179] The device sends the captured image data to a server, which then analyzes the image data using the Google Cloud Vision API and OpenCV to determine inventory status.

[2180] The analysis results are stored in a database.

[2181] 3. Emotion Recognition and Data Update

[2182] When a user uses the app, the emotion recognition engine analyzes emotions from the user's input, voice, and facial expressions. Emotion analysis is performed using Azure Cognitive Services.

[2183] The server stores the results of the sentiment analysis in a database and reflects them in food consumption patterns and recipe selection.

[2184] 4. Shopping list generation and notifications

[2185] The server generates a shopping list based on the user's consumption patterns, latest inventory status, and emotional information.

[2186] After generating the list, the most suitable recipe is selected and notified to the user's smartphone.

[2187] Users can access shopping lists and recipes through the app.

[2188] 5. Virtual store shopping support

[2189] Users can purchase ingredients suggested within the app at a virtual store.

[2190] The server provides a recommendation system to support the shopping experience in the virtual store.

[2191] Specific examples

[2192] For example, if a user types "I'm stressed today," the emotion analysis engine will determine the emotion "stress" and suggest herbal teas or relaxing dishes that will help reduce stress.

[2193] Example prompt sentence:

[2194] Write a Python program that recognizes emotions from text input and suggests ingredients and recipes based on them. Use the following libraries: TensorFlow, OpenCV, Google Cloud Vision API, and Azure Cognitive Services. If the user types "I'm tired today," suggest a corresponding recipe for a relaxing dish, such as chamomile tea or soup.

[2195] This system allows users to efficiently manage ingredients and purchase the ingredients they need without waste. It also suggests optimal recipes based on the user's emotional state, improving the quality and satisfaction of meals.

[2196] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2197] Program processing steps

[2198] Step 1:

[2199] Onboarding a User

[2200] Users launch the smartphone app and enter their food consumption habits, cooking preferences, allergy information, and emotional state.

[2201] Input: User's food consumption habits, cooking preferences, allergy information, and emotional state.

[2202] Data processing: Format user input information in JSON format.

[2203] The terminal transmits the input information to the server.

[2204] The server stores this data in a database.

[2205] Output: User information stored in the database.

[2206] Step 2:

[2207] Checking the inventory in the refrigerator

[2208] The camera inside the refrigerator automatically takes pictures of the inside of the refrigerator at a set time (e.g., every morning).

[2209] Input: Image data of the inside of the refrigerator.

[2210] Data processing: Image data is sent to the server in JPEG format.

[2211] The terminal transmits the captured image data to the server.

[2212] The server uses Google Cloud Vision API and OpenCV to analyze the image data and determine the inventory status in the refrigerator.

[2213] Data Computing: Image recognition algorithms are used to identify ingredients and generate inventory lists.

[2214] The analysis results are stored in a database.

[2215] Output: Refrigerator inventory information stored in a database.

[2216] Step 3:

[2217] Emotion Recognition and Data Update

[2218] While using a smartphone app, a user inputs their emotional state by typing or speaking.

[2219] Input: Text and / or audio data of the user's emotional state.

[2220] Data processing: Converting voice data into text and extracting sentiment keywords.

[2221] The terminal transmits the emotion data to the server.

[2222] The server uses Azure Cognitive Services to perform sentiment analysis and identify the user's emotional state.

[2223] Data Computing: Emotion recognition algorithms determine and store emotional states.

[2224] Output: User's emotional state information stored in a database.

[2225] Step 4:

[2226] Shopping list generation and notifications

[2227] The server generates a shopping list based on stored consumption patterns, up-to-date inventory status, and emotional information.

[2228] Input: User consumption habits, refrigerator inventory, and emotional information.

[2229] Data Computing: Predicting ingredient needs using machine learning algorithms.

[2230] The generated shopping list and the optimal recipe are selected and sent to the user's device.

[2231] Output: Shopping list and suggested recipes sent to the user's device.

[2232] Step 5:

[2233] Virtual store shopping support

[2234] Users purchase ingredients at a virtual store based on a shopping list presented within the app.

[2235] Input: Shopping list and virtual store catalog information.

[2236] Data processing: Add selected ingredients to cart.

[2237] The server uses a recommendation system to support the virtual store shopping experience.

[2238] Output: Order completed in the virtual store and purchase notification sent to the user.

[2239] This system allows users to efficiently manage ingredients and purchase the ingredients they need without waste. It also suggests optimal recipes based on the user's emotional state, improving the quality and satisfaction of meals.

[2240] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2241] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2242] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2243] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2244] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2245] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2246] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2247] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2248] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2249] The emotion identification model 59 inputs...

Claims

1. an input means for a user to input food ingredient consumption habits and cooking preferences; means for receiving information from said input means and storing it in a database; a photographing means for automatically photographing images of the inside of the refrigerator at regular intervals; a means for receiving and analyzing image data obtained by the imaging means; means for generating a shopping list based on the analyzed inventory information and stored consumption habits; means for notifying a user terminal of the generated shopping list; A means for suggesting an optimal recipe based on the shopping list; A system including:

2. 10. The system of claim 1, wherein the inventory information in the refrigerator is analyzed using an image recognition algorithm.

3. The system of claim 1 , wherein the system learns the user's food consumption habits using a machine learning algorithm.

Images