System

Abstract

A system is provided.SOLUTION: This food management device is provided with a means for recognizing the kind, quantity and best-before date of food, a means for transmitting the recognized food data, a means for preserving the transmitted data and tracking the best-before date or preservation period, a means for generating a recipe based on the taste, physical condition, season, weather and economical conditions of a user, and a means for transmitting the generated recipe. The system includes a notification means, a means for confirming a deficient food material and automatically placing an order, a means for analyzing the nutrition balance of the user and generating advice for supporting healthy eating habits, a means for transmitting the generated advice, and a means for collecting the feedback of the user and updating data.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] The problem that this invention aims to solve is to reduce the burden of food management in dual-income households and to support efficient and healthy eating habits. In particular, it addresses various issues that modern households face, such as managing food expiration dates, purchasing necessary ingredients, considering nutritional balance, and reducing food waste. [Means for solving the problem]

[0005] To solve this problem, the food management device of the present invention provides the following means. First, it has a means for recognizing the type, quantity, and expiration date of food. Then, it has a means for transmitting the recognized food data, and provides a means for saving the transmitted data and tracking the expiration date or storage period. Furthermore, it has a means for generating recipes based on the user's preferences, physical condition, season, weather, and economic situation, and a means for transmitting and notifying the user of the generated recipe. Next, it has a means for checking for missing ingredients and automatically ordering them. In addition, it provides a means for analyzing the user's nutritional balance and generating advice to support a healthy diet, and a means for transmitting the generated advice. Finally, it constructs a system that also includes a means for collecting user feedback and updating data.

[0006] A "food management device" is a device that has the function of recognizing the type, quantity, and expiration date of food, and also managing and processing this data.

[0007] "Means of recognition" refers to devices and sensor technology that detect and accurately identify the type, quantity, and expiration date of food.

[0008] "Transmitting means" refers to a communication device that transmits the recognized food data to a server or other system wirelessly or via wired communication.

[0009] "Storage means" refers to any device or software that records the food data transmitted and stores it in a database, if necessary.

[0010] "Tracking means" refers to devices or software that have the ability to monitor and manage expiration dates and shelf life based on stored food data.

[0011] "Means for generating recipes" refers to algorithms or software that suggest optimal dishes based on the user's preferences, physical condition, season, weather, and financial situation.

[0012] "Means for notification" refers to a notification system on a display or mobile device that notifies users of the generated recipes and advice.

[0013] "Means for automatic ordering" refers to a network or software that checks for shortages of ingredients and automatically sends purchase requests.

[0014] "Means of analysis" refers to an analysis system that analyzes users' eating habits and nutritional balance from data and supports a healthy diet.

[0015] "Means for generating advice" refers to algorithms or software that provide users with appropriate health advice or recipes based on the analysis results.

[0016] "Feedback collection methods" refers to the interfaces and software that receive user-provided information and use it to update data throughout the system. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] The present invention provides a system that uses a food management device to reduce the burden of food ingredient management in dual-income households and supports efficient and healthy eating habits. Hereinafter, embodiments of the present invention will be described in detail.

[0039] System Overview

[0040] This system uses various sensors and cameras installed inside the refrigerator to collect information such as food type, quantity, and expiration date, and sends this data to a server. The server analyzes this data to manage expiration dates and analyze the nutritional balance of food. It also generates optimal recipes based on the user's physical condition, preferences, season, weather, and economic situation, and notifies the user. It also automatically orders ingredients that are running low, preventing food waste and supporting healthy eating habits.

[0041] Food Recognition and Data Collection

[0042] Cameras and RFID readers installed inside the refrigerator recognize the type, quantity, and expiration date of food. This data is sent to a server via a communication device inside the refrigerator, and the server stores it in a database. For example, data such as "Milk (1 bottle, expiration date: October 10, 2023)" and "Eggs (6 eggs, expiration date: October 12, 2023)" may be acquired from the refrigerator.

[0043] Data storage and expiration date tracking

[0044] The data sent to the server is stored in a database, where expiration dates and storage periods are tracked. The server then notifies the user of food products that are approaching their expiration date. For example, if the expiration date of "milk" is approaching in two days, the server will send a notification to the device saying, "Milk's expiration date will expire in two days."

[0045] Recipe Suggestions

[0046] The server generates optimal recipes based on the user's preferences, physical condition, season, weather, and economic situation. For example, it generates a recipe for an "omelet" using milk and eggs based on the user's preferences and the ingredients currently in the refrigerator, and sends it to the device. The user can then refer to the recipe on the device and easily cook the dish.

[0047] Automatic ordering of necessary ingredients

[0048] The server analyzes the food data in the refrigerator and lists any ingredients that are running low. For example, if it detects that there are fewer than two eggs, the server generates an automatic ordering content based on the user's purchasing history and preferences. This order content is notified to the terminal, and if the user approves, an ordering request is sent to the online shopping site. After the order is completed, the user is notified that "the egg order has been completed."

[0049] Nutritional balance and health management

[0050] The server analyzes the user's eating habits and calculates nutritional balance. To support a healthy diet, it generates necessary health advice and recipes and notifies the device. For example, if the user has not eaten vegetables recently, it will suggest a salad recipe rich in green and yellow vegetables and notify the user. The user sends feedback to the server via their device, which updates the data throughout the system.

[0051] Specific examples

[0052] Next, the processing of the system will be described in detail using a specific example.

[0053] Example 1: Milk expiration date notification

[0054] The server retrieves information about "Milk (best before date: October 10, 2023)" from the database and confirms that the expiration date is two days later.

[0055] The server notifies the terminal that "The milk's expiration date will expire in two days."

[0056] The device displays this notification to the user.

[0057] Example 2: Omelette recipe suggestion

[0058] The server generates an omelet recipe based on the user's preferences and the information on milk and eggs in the refrigerator.

[0059] The server sends this recipe to the terminal.

[0060] The device notifies the user of the "omelette recipe."

[0061] Example 3: Automated egg ordering

[0062] The server determines that there are fewer than two eggs in the refrigerator.

[0063] The server generates an order for "eggs (pack of 10)" and sends an order request to the online shopping site.

[0064] The terminal notifies the user that "your egg order has been completed."

[0065] In this way, the present invention realizes efficient food management and support for a healthy diet using a food management device. This system reduces the burden of food ingredient management for dual-income households and provides a healthy diet that takes nutritional balance into consideration.

[0066] The processing flow will be explained below.

[0067] Step 1:

[0068] The refrigerator detects when the door is opened and closed, activating the internal camera and RFID reader. This recognizes data such as the type of food, quantity, and expiration date. For example, it obtains information such as "Milk (1 bottle, expiration date: October 10, 2023)" and "Eggs (6 eggs, expiration date: October 12, 2023)."

[0069] Step 2:

[0070] The refrigerator sends the food data it recognizes to the server, including the type of food, quantity, and expiration date.

[0071] Step 3:

[0072] The server stores the received food data in a database. The data is structured as information such as the type, quantity, expiration date, and storage period of each food item.

[0073] Step 4:

[0074] The server periodically checks the database and lists foods that are close to their expiration date. For example, it detects that the expiration date of "Milk (expiration date: October 10, 2023)" is just two days away.

[0075] Step 5:

[0076] The server prepares notifications for food items that are nearing their expiration date. For items with an expiration date within two days, the server prepares to notify the user.

[0077] Step 6:

[0078] The server sends the notification content to the terminal. For example, it sends notification data saying "The expiration date of the milk will be reached in two days."

[0079] Step 7:

[0080] The device displays the received notification to the user. For example, a notification pops up on the smartphone saying "The expiration date of the milk will expire in two days."

[0081] Step 8:

[0082] The server generates recipes based on the user's preferences, physical condition, season, weather, and economic situation, and analyzes various collected data to select the optimal recipe.

[0083] Step 9:

[0084] The server sends the generated recipe to the terminal. For example, it sends a "recipe for an omelet using milk and eggs."

[0085] Step 10:

[0086] The device notifies the user of the received recipe. For example, the smartphone displays "An omelet recipe has been suggested."

[0087] Step 11:

[0088] The server analyzes the food data in the refrigerator and lists the ingredients that are running low. For example, it might determine that there are fewer than two eggs.

[0089] Step 12:

[0090] The server generates automatic ordering based on the ingredients that are in short supply. Taking into account the user's purchasing history and preferences, it prepares an order request for "Eggs (pack of 10)."

[0091] Step 13:

[0092] The server sends the order to the terminal and asks the user for approval, for example, sending a confirmation message saying "Do you want to automatically order eggs?"

[0093] Step 14:

[0094] The terminal displays the received order confirmation to the user, who can then select "Accept" or "Cancel" in response to the confirmation message.

[0095] Step 15:

[0096] The user responds with "approval" to the order confirmation. The response data is sent from the terminal to the server.

[0097] Step 16:

[0098] The server receives the user's approval and sends an order request to the online shopping site. For example, it sends a request to order "eggs (pack of 10)."

[0099] Step 17:

[0100] The server sends a notification of order completion to the terminal, for example, data such as "Your egg order has been completed."

[0101] Step 18:

[0102] The device will notify the user that their eggs have been ordered. The user can then check the status of their order.

[0103] Step 19:

[0104] The server collects data on the user's eating habits and analyzes their nutritional balance. For example, it obtains information such as, "I've been eating less vegetables recently."

[0105] Step 20:

[0106] The server generates health advice and recipes based on the analysis results, such as a recipe for a salad rich in green and yellow vegetables.

[0107] Step 21:

[0108] The server sends the generated advice or recipe to the terminal, for example, sending a "salad recipe."

[0109] Step 22:

[0110] The device will notify the user of the advice or recipes it has received, and the user can view the notification and incorporate the advice or recipes.

[0111] Step 23:

[0112] If the user actually uses the suggested advice or recipe, feedback is sent via the device.

[0113] Step 24:

[0114] The server receives the feedback submitted by the user and updates the database, which improves the accuracy of the system and allows for a more personalized service.

[0115] Example 1

[0116] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0117] The present invention relates to a system that reduces the burden of food ingredient management for dual-income households and supports efficient and healthy eating habits. Conventional food management systems often require manual management of food expiration dates and analysis of nutritional balance, which can be time-consuming and labor-intensive for dual-income households. Furthermore, automatic food ordering and recipe suggestions may not be performed appropriately, potentially resulting in food waste and nutritional imbalances. A solution to these issues is needed.

[0118] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0119] In this invention, the server includes means for recognizing the type, quantity, and expiration date of food, means for transmitting the recognized food data, means for storing the transmitted data and tracking the expiration date or storage period, means for generating recipes based on the user's preferences, physical condition, season, weather conditions, and economic situation, means for transmitting and notifying the user of the generated recipes, means for checking for missing ingredients and automatically ordering, means for analyzing the user's nutritional balance and generating suggestions to support a healthy diet, means for transmitting the generated suggestions, means including a communication device for collecting food data, and means for collecting user feedback and updating the data. This reduces the burden of food management for dual-income households and makes it possible to support an efficient and healthy diet.

[0120] A "food management device" is a device that recognizes and manages information such as food type, quantity, and expiration date.

[0121] "Means for recognizing the type, quantity, and expiration date of food" refers to means for identifying the type, quantity, and expiration date of food using a camera, RFID reader, etc.

[0122] The "means for transmitting recognized food data" is a communication device for transmitting information about the recognized food to a server or database.

[0123] "Means for storing submitted data and tracking expiration dates or shelf life" refers to means for storing submitted food data in a database and managing and tracking expiration dates or shelf life.

[0124] "Means for generating recipes based on the user's preferences, physical condition, season, weather conditions, and economic situation" refers to a program or system that generates optimal recipes based on the user's personal information and external conditions.

[0125] The "means for transmitting and notifying the generated recipe" is a communication and notification system for transmitting and notifying the generated recipe to the user's terminal.

[0126] The "means of checking for shortages of ingredients and automatically ordering" is a system that analyzes food data in the refrigerator, identifies shortages of food, and automatically orders them.

[0127] The "means for analyzing a user's nutritional balance and generating suggestions to support a healthy diet" is a system that analyzes a user's eating habit data and generates suggestions to support a healthy diet from the perspective of nutritional balance.

[0128] The "means for transmitting the generated suggestions" is a communication system for transmitting the generated health suggestions to the user's terminal.

[0129] The "means including a communication device for collecting food data" is a system including a communication device for collecting food information and transmitting it to a server.

[0130] "Means for collecting user feedback and updating data" refers to means that have the function of receiving feedback from users and updating data throughout the system based on that feedback.

[0131] MODE FOR CARRYING OUT THE INVENTION

[0132] The present invention provides a system that uses a food management device to reduce the burden of food ingredient management in dual-income households and supports efficient and healthy eating habits. Hereinafter, embodiments of the present invention will be described in detail.

[0133] System Overview

[0134] This system uses various sensors and cameras installed inside the refrigerator to collect information such as food type, quantity, and expiration date, and sends this data to a server. The server analyzes this data to manage expiration dates and analyze the nutritional balance of food. It also generates optimal recipes based on the user's physical condition, preferences, season, weather conditions, and economic situation, and notifies the user. It also automatically orders ingredients that are running low, preventing food waste and supporting healthy eating habits.

[0135] Food Recognition and Data Collection

[0136] Cameras and RFID readers installed inside the refrigerator recognize the type, quantity, and expiration date of food. This data is sent to a server via a communication device inside the refrigerator, and the server stores it in a database. For example, data such as "Milk (1 bottle, expiration date: October 10, 2023)" and "Eggs (6 eggs, expiration date: October 12, 2023)" may be acquired from the refrigerator.

[0137] Data storage and expiration date tracking

[0138] The data sent to the server is stored in a database, where expiration dates and storage periods are tracked. The server then notifies the user of food products that are approaching their expiration date. For example, if the expiration date of "milk" is approaching in two days, the server will send a notification to the device saying, "Milk's expiration date will expire in two days."

[0139] Recipe Suggestions

[0140] The server generates optimal recipes based on the user's preferences, physical condition, season, weather conditions, and economic situation. For example, it generates a recipe for an "omelet" using milk and eggs based on the user's preferences and the ingredients currently in the refrigerator, and sends it to the device. The user can then refer to the recipe on the device and easily cook the dish.

[0141] Automatic ordering of necessary ingredients

[0142] The server analyzes the food data in the refrigerator and lists any ingredients that are running low. For example, if it detects that there are fewer than two eggs, the server generates an automatic ordering content based on the user's purchasing history and preferences. This order content is notified to the terminal, and if the user approves, an ordering request is sent to the e-commerce site. After the order is completed, the server sends a notification to the user that "the egg order has been completed."

[0143] Nutritional balance and health management

[0144] The server analyzes the user's eating habits and calculates nutritional balance. To support a healthy diet, it generates necessary health advice and recipes and notifies the device. For example, if the user has not eaten vegetables recently, it will suggest a salad recipe rich in green and yellow vegetables and notify the user. The user sends feedback to the server via their device, which updates the data throughout the system.

[0145] Specific examples

[0146] Next, the processing of the system will be described in detail using a specific example.

[0147] Example 1: Milk expiration date notification

[0148] The server retrieves information about "Milk (best before date: October 10, 2023)" from the database and confirms that the expiration date is two days later.

[0149] The server notifies the terminal that "The milk's expiration date will expire in two days."

[0150] The device displays this notification to the user.

[0151] Example 2: Omelette recipe suggestion

[0152] The server generates an omelet recipe based on the user's preferences and the information on milk and eggs in the refrigerator.

[0153] The server sends this recipe to the terminal.

[0154] The device notifies the user of the "omelette recipe."

[0155] Example 3: Automated egg ordering

[0156] The server determines that there are fewer than two eggs in the refrigerator.

[0157] The server generates an order for "Eggs (pack of 10)" and sends the order request to the e-commerce site.

[0158] The terminal notifies the user that "your egg order has been completed."

[0159] Prompt Sentence Examples

[0160] Below is an example of a prompt sentence to input to the generative AI model.

[0161] "Please analyze the food in my refrigerator and its expiration date and suggest the best recipes."

[0162] "Analyze the ingredients that are in short supply and use an automated ordering system to generate orders for the ingredients that are needed."

[0163] "Based on the user's eating habits data, create nutritional advice to support healthy eating habits."

[0164] In this way, the present invention realizes efficient food management and support for a healthy diet using a food management device. This system reduces the burden of food ingredient management for dual-income households and provides a healthy diet that takes nutritional balance into consideration.

[0165] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0166] Step 1: Collect food data

[0167] Input: Food in the fridge

[0168] Output: Food type, quantity, expiration date data

[0169] What happens:

[0170] A camera installed inside the refrigerator takes pictures of the food, and an RFID reader reads the data from the RFID tags.

[0171] The communication device inside the refrigerator acquires the image data from the camera and the tag data from the RFID reader.

[0172] The data obtained includes the type of food (e.g., "milk" or "egg"), quantity (e.g., "1 bottle" or "6 pieces"), and expiration date (e.g., "October 10, 2023" or "October 12, 2023").

[0173] The communication device transmits this data to the server.

[0174] Step 2: Data storage and analysis

[0175] Input: Food type, quantity, expiration date data

[0176] Output: Saved food data, analysis results

[0177] What happens:

[0178] The server receives the food data transmitted from the refrigerator.

[0179] The received data includes the type of food, quantity, expiration date, etc.

[0180] The server stores this data in a database (e.g., "Milk (1 bottle, expiration date: October 10, 2023)" and "Eggs (6 eggs, expiration date: October 12, 2023)").

[0181] The server analyzes the stored data to identify current food inventory and foods that are close to their expiration date.

[0182] Step 3: Best before date notification

[0183] Input: Food data in the database

[0184] Output: Best before date notification message

[0185] What happens:

[0186] The server periodically scans the food data in the database to identify foods that are approaching their expiration date.

[0187] For example, find "milk" whose expiration date is just two days away.

[0188] The server generates a notification message saying "Milk expires in 2 days."

[0189] Send this message to the user's terminal.

[0190] The device will display this notification to the user.

[0191] Step 4: Recipe suggestions

[0192] Input: User preferences, physical condition, season, weather conditions, economic situation, food data in refrigerator

[0193] Output: Recipe suggestion message

[0194] What happens:

[0195] The server collects data on the user's preferences, physical condition, season, weather conditions, and economic situation.

[0196] It also retrieves food data from the refrigerator and checks the current ingredients.

[0197] For example, suppose you have milk and eggs in your refrigerator.

[0198] The server uses this information to input prompts into the generative AI model (e.g., "There is currently milk and eggs in the refrigerator. Please suggest some recipes using these.").

[0199] The generative AI model generates a recipe for "omelette" based on prompts.

[0200] The server sends this generated recipe to the user's terminal, and the user is notified of the "omelette recipe."

[0201] Step 5: Automatically order the ingredients you need

[0202] Input: Food data in the refrigerator

[0203] Output: Order completion notification message

[0204] What happens:

[0205] The server analyzes the database and detects that there is a shortage of food in the refrigerator.

[0206] For example, identify a decrease to "less than two eggs."

[0207] The server generates an order (e.g., "pack of 10 eggs") based on the user's past purchasing history and preferences.

[0208] The server sends the order details to the electronic commerce site as an order request.

[0209] After the order is completed, the server sends a notification to the user's terminal saying "Egg order completed."

[0210] Step 6: Nutritional Balance and Health Management

[0211] Input: User's dietary habits data

[0212] Output: Health suggestion message

[0213] What happens:

[0214] The server periodically analyzes the user's eating habits data.

[0215] Based on the analysis results, nutritional balance is calculated and any missing nutrients are identified.

[0216] For example, it detects when a user has not eaten vegetables recently.

[0217] The server generates recipes for salads rich in green vegetables to support a healthy diet.

[0218] This generated recipe is sent to the user's device and notified to the user.

[0219] Users send feedback to the server through their devices, and data is updated throughout the system.

[0220] (Application example 1)

[0221] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0222] In busy households, such as dual-income households, it is difficult to manage food and ensure a balanced diet. Even when shopping at brick-and-mortar stores, it can be time-consuming to know what to buy and whether there are any foods nearing their expiration date. Furthermore, there is a need for a method to efficiently procure ingredients while maintaining a healthy diet. A system that solves these issues and supports food management and healthy eating habits at home is needed.

[0223] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0224] In this invention, the server includes means for recognizing the type, quantity, and expiration date of food, means for transmitting the recognized food data, means for storing the transmitted data and tracking the expiration date or storage period, means for generating recipes based on the user's preferences, physical condition, season, weather, and economic situation, means for transmitting and notifying the user of the generated recipes, means for checking for missing ingredients and automatically ordering, means for analyzing the user's nutritional balance and generating advice to support a healthy diet, means for transmitting the generated advice, means for collecting user feedback and updating data, means for displaying ingredient information in a physical store via smart glasses and presenting a list of ingredients that need to be replenished, and means for presenting recipes and special sale information generated based on the user's profile and external environmental information in real time, thereby enabling more efficient food management and supporting a healthy diet.

[0225] A "food management device" is a device for recognizing and managing information such as the type, quantity, and expiration date of food.

[0226] A "sensor" is a device that detects a physical quantity or state and outputs it as an electrical signal.

[0227] A "camera" is a device that can capture video or still images and store and transmit them as digital data.

[0228] A "communication device" is a device for sending and receiving data to and from other devices and servers.

[0229] A "server" is a computer system that stores, analyzes, and processes data and provides information in response to requests from other devices.

[0230] "User preferences" refers to information that indicates the user's taste and cooking preferences.

[0231] "Physical condition" is information that indicates the user's health condition and physical condition.

[0232] "Season" refers to the time of year brought about by changes in the climate.

[0233] "Weather" is information indicating weather conditions in a particular area and time.

[0234] "Financial status" is information that indicates the user's financial status, such as income, expenses, and budget.

[0235] A "recipe" is a detailed list of instructions and ingredients needed to make a particular dish.

[0236] "Notification" means sending messages or alerts to inform users.

[0237] "Auto ordering" is the process of automatically ordering necessary items and ingredients.

[0238] "Nutritional balance" refers to a state in which the user's diet properly incorporates the necessary nutrients.

[0239] "Advice" is information that provides advice or recommendations to the user.

[0240] "Feedback" refers to opinions and evaluations provided by users to the system.

[0241] "Smart glasses" are wearable devices that have the ability to display visual information.

[0242] A "brick and mortar store" is a physical commercial establishment where users can visit and purchase products.

[0243] "Food information" refers to data such as food type, quantity, expiration date, and storage conditions.

[0244] A "profile" is a data set that compiles personal information and characteristics about a user.

[0245] "External environment information" refers to information that is not directly managed by the user, such as weather, season, and surrounding conditions.

[0246] "Special sale information" is information about products being sold at a price lower than the regular price.

[0247] The present invention is a system for supporting home food management and healthy eating habits. The purpose of the present invention is to streamline food management in physical stores through smart glasses and improve the user's shopping experience. Specific embodiments for implementing the present invention are described in detail below.

[0248] System Overview

[0249] This system uses various sensors and cameras installed inside the refrigerator to collect information such as food type, quantity, and expiration date, and sends this data to a server. The server analyzes this data to manage expiration dates and analyze the nutritional balance of food. In addition, when a user wearing smart glasses purchases ingredients in a physical store, the system displays real-time information on ingredients, special offers, recipe suggestions, and more.

[0250] Hardware and Software

[0251] Hardware:

[0252] Cameras and sensors inside the refrigerator: Used to collect food data.

[0253] Smart glasses: Using "Google Glass" and "Vuzix Blade" as examples.

[0254] Server: We use AWS and Azure for data analysis and notifications.

[0255] software:

[0256] Data collection software: Collects data from cameras and sensors inside the refrigerator and sends it to a server.

[0257] Data analysis software: Food data, user information, and external environmental information are analyzed on the server.

[0258] User Interface: An application for display on smart glasses.

[0259] Specific processing

[0260] 1. Food Recognition and Data Collection:

[0261] Cameras and RFID readers installed inside the refrigerator identify the type, quantity and expiration date of food and send this data to a server, which stores it in a database and tracks expiration dates.

[0262] 2. Information display through smart glasses:

[0263] When users shop in a physical store, the smart glasses display real-time information about the ingredients in their refrigerator. For example, if milk or eggs are low in stock, the glasses notify the user. They also display sales and coupon information to support their purchasing behavior.

[0264] 3. Recipe and nutritional suggestions:

[0265] The server uses a generative AI model to generate appropriate recipes based on the user's preferences, physical condition, season, weather, and economic situation. The generated recipes are then sent to the user via smart glasses. The server also analyzes nutritional balance and sends advice to support healthy eating.

[0266] Specific examples

[0267] A specific example of the system's operation is shown below.

[0268] Example 1: Shopping assistance in physical stores

[0269] The user wears smart glasses and goes shopping in a physical store. The smart glasses' display shows real-time information about the food inventory in the refrigerator. A notification such as "You're low on milk and need to buy more" is displayed, along with information about special sales and coupons available in the store.

[0270] Example 2: Recipe suggestions

[0271] When a user is using the smart glasses at home, the server generates an "omelette" recipe based on the user's current health condition, preferences, and information about ingredients in the refrigerator, and notifies the smart glasses. The following prompt sentence is used for the generating AI model:

[0272] Example prompt for a generative AI model:

[0273] "The user has a bottle of milk and two eggs in the refrigerator. It's fall, and the user prefers a balanced diet. Based on this, please suggest some suitable recipes."

[0274] In this way, the present invention can utilize smart glasses to comprehensively support food management in brick-and-mortar stores and improve users' quality of life.

[0275] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0276] Step 1:

[0277] Food data collection

[0278] The server receives input from cameras and RFID readers installed inside the refrigerator. This input includes the type of food, quantity, and expiration date. The server analyzes this data and stores it in a database for each food item. For example, information such as "Milk (Quantity: 1 bottle, Best before date: October 10, 2023)" may be collected.

[0279] Step 2:

[0280] Best before date tracking

[0281] The server periodically analyzes the expiration dates of the food items stored in the database and lists the food items that are approaching their expiration date. For example, it lists the food items whose expiration date is two days away and prepares a notification such as "The expiration date will expire in two days." This notification is sent to the user's device.

[0282] Step 3:

[0283] Detecting missing ingredients

[0284] The server analyzes the food data in the database and detects ingredients that are low in stock. For example, if it determines that there are fewer than two eggs, it adds them to a list of ingredients that are in short supply. The server notifies the user of the list of ingredients that are in short supply, and clearly indicates which ingredients need to be replenished.

[0285] Step 4:

[0286] Preparing for automatic ordering

[0287] The server generates an automatic order for missing ingredients based on the user's purchasing history and preferences. For example, it prepares an order for "eggs (pack of 10)" and prepares to send a request to an online shopping site. This order preparation data is notified to the user and they are asked to approve it.

[0288] Step 5:

[0289] Recipe Generation

[0290] Based on the input data of the user's preferences, physical condition, season, weather, and economic situation, the server uses a generative AI model to generate the optimal recipe. The generative AI model is input with the following prompt:

[0291] "The user has a bottle of milk and two eggs in the refrigerator. It's fall, and the user prefers a balanced diet. Based on this, please suggest some suitable recipes."

[0292] The generated recipe, for example an "omelette" recipe, is completed and sent to the user terminal.

[0293] Step 6:

[0294] Information display on smart glasses

[0295] When a user wears the smart glasses and goes shopping in a physical store, the device displays real-time information about ingredients in the refrigerator and a list of ingredients that are running low. For example, a notification will appear on the smart glasses saying, "You're low on milk and need to buy more." In addition, the device will display information about special sales and coupons in the store, allowing users to shop efficiently.

[0296] Step 7:

[0297] Nutritional balance analysis and advice delivery

[0298] The server analyzes the user's eating habits, calculates nutritional balance, and generates advice to support a healthy diet. For example, it generates advice such as "You haven't been eating enough vegetables recently," and notifies the user's device. This advice collects user feedback and serves as the basis for updating data throughout the system.

[0299] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0300] The present invention provides a system that uses a food management device combined with an emotion engine to reduce the burden of food ingredient management for dual-income households and support efficient and healthy eating habits. The following describes in detail the embodiments of the present invention.

[0301] System Overview

[0302] This system uses sensors and cameras installed inside the refrigerator to collect information such as food type, quantity, and expiration date, and sends this data to a server. The server analyzes this data to manage expiration dates and analyze the nutritional balance of food. It also generates optimal recipes based on the user's physical condition, preferences, season, weather, economic situation, and emotions, and notifies the user. It also automatically orders ingredients that are running low, preventing food waste and supporting healthy eating habits.

[0303] Food Recognition and Data Collection

[0304] Cameras and RFID readers installed inside the refrigerator recognize the type, quantity, and expiration date of food. This data is sent to a server via a communication device inside the refrigerator, and the server stores it in a database. For example, data such as "Milk (1 bottle, expiration date: October 10, 2023)" and "Eggs (6 eggs, expiration date: October 12, 2023)" may be acquired from the refrigerator.

[0305] Data storage and expiration date tracking

[0306] The data sent to the server is stored in a database, where expiration dates and storage periods are tracked. The server then notifies the user of food products that are approaching their expiration date. For example, if the expiration date of "milk" is approaching in two days, the server will send a notification to the device saying, "Milk's expiration date will expire in two days."

[0307] Recipe Suggestions

[0308] The server generates optimal recipes based on the user's preferences, physical condition, season, weather, economic situation, and emotional data obtained by the emotion engine. For example, it generates a recipe for an "omelette" using "milk" and "eggs" based on the user's preferences and the ingredients currently in the refrigerator, and sends it to the device. The user can then refer to the recipe on their device and easily cook the dish.

[0309] Automatic ordering of necessary ingredients

[0310] The server analyzes the food data in the refrigerator and lists any ingredients that are running low. For example, if it detects that there are fewer than two eggs, the server generates an automatic ordering content based on the user's purchasing history and preferences. This order content is notified to the terminal, and if the user approves, an ordering request is sent to the online shopping site. After the order is completed, the user is notified that "the egg order has been completed."

[0311] Nutritional balance and health management

[0312] The server analyzes the user's eating habits and calculates nutritional balance. To support a healthy diet, it generates necessary health advice and recipes and notifies the device. For example, if the user has not eaten vegetables recently, it will suggest a salad recipe rich in green and yellow vegetables and notify the user. The user sends feedback to the server via their device, which updates the data throughout the system.

[0313] Further suggestions from the emotion engine

[0314] The emotion engine recognizes the user's emotions and further personalizes recipes and health advice based on this. For example, if the user is feeling stressed, it will suggest relaxing recipes that will help reduce stress.

[0315] Specific examples

[0316] Example 1: Emotion-based recipe suggestions

[0317] The server collects the user's emotional state through an emotion engine. For example, if the user is "feeling stressed," emotion data is acquired.

[0318] The server generates relaxation recipes, including chamomile tea, which is effective in reducing stress.

[0319] The server sends this relaxation recipe to the terminal.

[0320] The device will display a notification to the user saying, "A suggestion has been made for making chamomile tea."

[0321] Example 2: Emotion-based health advice

[0322] The server obtains emotional data from the emotion engine, such as "feeling tired" by the user.

[0323] The server generates advice including "vitamin-rich dishes" that are effective in relieving fatigue.

[0324] The server sends this advice to the terminal.

[0325] The device will display a notification to the user saying, "A dish that is effective in relieving fatigue has been suggested."

[0326] In this way, the present invention realizes efficient food management and support for healthy eating habits using a food management device with an emotion engine built in. This system reduces the burden of food ingredient management for dual-income households and provides a healthy diet that takes nutritional balance into consideration.

[0327] The processing flow will be explained below.

[0328] Step 1:

[0329] The refrigerator detects when the door is opened and closed, activating the internal camera and RFID reader. This recognizes data such as the type of food, quantity, and expiration date. For example, it obtains information such as "Milk (1 bottle, expiration date: October 10, 2023)" and "Eggs (6 eggs, expiration date: October 12, 2023)."

[0330] Step 2:

[0331] The refrigerator sends the food data it recognizes to the server, including the type of food, quantity, and expiration date.

[0332] Step 3:

[0333] The server stores the received food data in a database. The data is structured as information such as the type, quantity, expiration date, and storage period of each food item.

[0334] Step 4:

[0335] The server periodically checks the database and lists foods that are close to their expiration date. For example, it detects that the expiration date of "Milk (expiration date: October 10, 2023)" is just two days away.

[0336] Step 5:

[0337] The server prepares notifications for food items that are nearing their expiration date. For items with an expiration date within two days, the server prepares to notify the user.

[0338] Step 6:

[0339] The server sends the notification content to the terminal. For example, it sends notification data saying "The expiration date of the milk will be reached in two days."

[0340] Step 7:

[0341] The device displays the received notification to the user. For example, a notification pops up on the smartphone saying "The expiration date of the milk will expire in two days."

[0342] Step 8:

[0343] The server generates optimal recipes based on the user's preferences, physical condition, season, weather, economic situation, and emotional data obtained by the emotion engine. For example, it generates a recipe for "omelette with milk and eggs" based on the user's preferences and the ingredients currently in the refrigerator.

[0344] Step 9:

[0345] The server sends the generated recipe to the terminal. For example, it sends a "recipe for an omelet using milk and eggs."

[0346] Step 10:

[0347] The device notifies the user of the received recipe, for example, "An omelet recipe has been suggested."

[0348] Step 11:

[0349] The server analyzes the food data in the refrigerator and lists the ingredients that are running low. For example, it might determine that there are fewer than two eggs.

[0350] Step 12:

[0351] The server generates automatic ordering based on the ingredients that are in short supply. Taking into account the user's purchasing history and preferences, it prepares an order request for "Eggs (pack of 10)."

[0352] Step 13:

[0353] The server sends the order to the terminal and asks the user for approval, for example, sending a confirmation message saying "Do you want to automatically order eggs?"

[0354] Step 14:

[0355] The terminal displays the received order confirmation to the user, who can then select "Accept" or "Cancel" in response to the confirmation message.

[0356] Step 15:

[0357] The user responds with "approval" to the order confirmation. The response data is sent from the terminal to the server.

[0358] Step 16:

[0359] The server receives the user's approval and sends an order request to the online shopping site. For example, it sends a request to order "eggs (pack of 10)."

[0360] Step 17:

[0361] The server sends a notification of order completion to the terminal, for example, data such as "Your egg order has been completed."

[0362] Step 18:

[0363] The device will notify the user that their eggs have been ordered. The user can then check the status of their order.

[0364] Step 19:

[0365] The server collects data on the user's eating habits and analyzes their nutritional balance. For example, it obtains information such as, "I've been eating less vegetables recently."

[0366] Step 20:

[0367] The server generates health advice and recipes based on the analysis results, such as a recipe for a salad rich in green and yellow vegetables.

[0368] Step 21:

[0369] The server sends the generated advice or recipe to the terminal, for example, sending a "salad recipe."

[0370] Step 22:

[0371] The device will notify the user of the advice or recipes it has received, and the user can view the notification and incorporate the advice or recipes.

[0372] Step 23:

[0373] The server collects the user's emotional state through the emotion engine. For example, if the user says "I'm feeling stressed," the emotion data is acquired.

[0374] Step 24:

[0375] The server analyzes the emotional data and generates recipes and advice based on the user's emotional state. For example, it generates a relaxation recipe that includes chamomile tea, which is effective in reducing stress.

[0376] Step 25:

[0377] The server sends the generated relaxation recipe to the device, for example, "How to make chamomile tea."

[0378] Step 26:

[0379] The device notifies the user of the relaxation recipe it has received, for example, by displaying "A chamomile tea recipe has been suggested."

[0380] Step 27:

[0381] The server receives the feedback submitted by the user and updates the database, which improves the accuracy of the system and allows for a more personalized service.

[0382] Example 2

[0383] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0384] In dual-income households, it is difficult to maintain an efficient and healthy diet while reducing the burden of food management. There is also a need for optimal dietary recommendations based on the user's emotions and physical condition, but existing systems have difficulty comprehensively considering these factors. Therefore, there is a need for a system that automates the management of food types, quantities, and expiration dates, and also provides personalized recipes and advice based on the user's individual condition.

[0385] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0386] A means of recognizing the type, quantity, and expiration date of food;

[0387] means for transmitting the recognized food data;

[0388] A means for storing the transmitted data and tracking expiration dates or shelf life;

[0389] A means for generating recipes based on the user's preferences, physical condition, season, weather, and economic situation;

[0390] A means for transmitting and notifying the generated recipe;

[0391] A way to check for shortages of ingredients and automatically order them,

[0392] A means to analyze the user's nutritional balance and generate advice to support a healthy diet;

[0393] means for transmitting the generated advice;

[0394] A means of collecting user feedback and updating the data;

[0395] A means to recognize user emotions using an emotion engine and personalize recipes and advice based on this;

[0396] A means to use a generative AI model to generate prompts based on data on food in the refrigerator and user sentiment data, and to suggest recipes and advice based on those prompts;

[0397] A means to transmit the entire food recognition and management process to a server through a secure communication protocol

[0398] This will reduce the burden of food management for dual-income households and enable efficient and healthy eating habits. It will also enable personalized suggestions that take into account the user's emotions and physical condition.

[0399] A "food management device" is a device that manages information such as the type, quantity, and expiration date of food, and provides this information to users.

[0400] The "recognition means" refers to a method or device for detecting and processing information such as the type, quantity, and expiration date of food.

[0401] A "means for transmitting" is a method or device for transmitting the recognized information to another device or system.

[0402] The "storing means" refers to a method or device for storing the transmitted data in a recording medium such as a database.

[0403] A "tracking means" is a method or device for monitoring stored data and updating information or taking action based on certain conditions.

[0404] A "recipe generator" is a method or device for suggesting cooking steps and ingredients based on a user's specific requirements.

[0405] "Notification means" refers to a method or device for notifying the user of the generated information.

[0406] The "automatic ordering means" is a method or device for detecting shortages of ingredients and automatically carrying out the ordering procedure.

[0407] The "means for generating advice" is a method or device for generating suggestions to support a healthy diet based on the user's nutritional balance and health status.

[0408] A "means for collecting feedback" is a method or device for gathering user responses and opinions and updating the system.

[0409] An "emotion engine" is a technology that recognizes a user's emotional state and generates personalized suggestions based on that data.

[0410] A "generative AI model" is an artificial intelligence model that automatically generates text or suggestions based on input data.

[0411] A "prompt" is text containing a question or command that is input to a generative AI model.

[0412] A "secure communication protocol" is a communication protocol that protects data transmission and prevents unauthorized access by third parties.

[0413] MODE FOR CARRYING OUT THE INVENTION

[0414] The present invention provides a system that uses a food management device combined with an emotion engine to reduce the burden of food ingredient management for dual-income households and support efficient and healthy eating habits. Specific embodiments for implementing the present invention are described below.

[0415] System Overview

[0416] This system uses sensors and cameras installed inside the refrigerator to collect information such as food type, quantity, and expiration date, and sends this data to a server. The server analyzes this data to manage expiration dates and analyze the nutritional balance of food. It also generates optimal recipes based on the user's physical condition, preferences, season, weather, economic situation, and emotions, and notifies the user. It also automatically orders ingredients that are running low, preventing food waste and supporting healthy eating habits.

[0417] Food Recognition and Data Collection

[0418] Cameras and RFID readers installed inside the refrigerator recognize the type, quantity, and expiration date of food. This data is sent to a server via a communication device inside the refrigerator, and the server stores it in a database. For example, data such as "Milk (1 bottle, expiration date: October 10, 2023)" and "Eggs (6 eggs, expiration date: October 12, 2023)" may be acquired from the refrigerator. In this case, the camera captures high-resolution images inside the refrigerator, and the RFID reader reads the RFID tags on the food items and obtains the information contained therein.

[0419] Data storage and expiration date tracking

[0420] The data sent to the server is stored in a database, where expiration dates and storage periods are tracked. The server then notifies the user of food products that are approaching their expiration date. For example, if the expiration date of "milk" is approaching in two days, the server will send a notification to the device saying, "Milk's expiration date will expire in two days."

[0421] Recipe Suggestions

[0422] The server generates optimal recipes based on the user's preferences, physical condition, season, weather, economic situation, and emotional data obtained by the emotion engine. Using a generative AI model, the server generates prompts based on the food data in the refrigerator and the user's emotional data, and suggests recipes and advice based on these. For example, it generates a recipe for an "omelette" using the "milk" and "eggs" in the refrigerator and sends it to the device. The user can then refer to the recipe on the device and cook the food. Examples of prompts are as follows:

[0423] "There is one bottle of milk and six eggs in the refrigerator. The user is feeling stressed. Based on this, please suggest the best recipe for the user."

[0424] Automatic ordering of necessary ingredients

[0425] The server analyzes the food data in the refrigerator and lists the ingredients that are running low. For example, if there are fewer than two eggs, the server generates an automatic ordering content based on the user's purchasing history and preferences. This order content is notified to the terminal, and if the user approves, an ordering request is sent to the online shopping site. After the order is completed, the user is notified that "the egg order has been completed."

[0426] Nutritional balance and health management

[0427] The server analyzes the user's eating habits and calculates nutritional balance. To support a healthy diet, it generates necessary health advice and recipes and notifies the device. For example, if the user has not eaten vegetables recently, it will suggest a salad recipe rich in green and yellow vegetables and notify the user. The user sends feedback to the server via their device, which updates the data throughout the system.

[0428] Further suggestions from the emotion engine

[0429] The emotion engine recognizes the user's emotions and further personalizes recipes and health advice based on this. For example, if the user is feeling stressed, it will suggest relaxing recipes that will help reduce stress.

[0430] Specific examples

[0431] Example 1: Emotion-based recipe suggestions

[0432] The server collects the user's emotional state through an emotion engine. For example, if the user is "feeling stressed," emotion data is acquired.

[0433] The server generates relaxation recipes, including chamomile tea, which is effective in reducing stress.

[0434] The server sends this relaxation recipe to the terminal.

[0435] The device will display a notification to the user saying, "A suggestion has been made for making chamomile tea."

[0436] Example 2: Emotion-based health advice

[0437] The server obtains emotional data from the emotion engine, such as "feeling tired" by the user.

[0438] The server generates advice including "vitamin-rich dishes" that are effective in relieving fatigue.

[0439] The server sends this advice to the terminal.

[0440] The device will display a notification to the user saying, "A dish that is effective in relieving fatigue has been suggested."

[0441] In this way, users can utilize a system that supports efficient food management and healthy eating habits based on food data and emotional data in their refrigerator.

[0442] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0443] Step 1:

[0444] A camera installed in the refrigerator takes pictures of the food items, and an RFID reader retrieves the RFID tag information for each food item. The input is the food items in the refrigerator, and the output is the food images and RFID tag data. The camera captures high-resolution images, and the RFID reader reads the food information (e.g., type, quantity, expiration date).

[0445] Step 2:

[0446] A communication device in the refrigerator transmits the acquired data (food images and RFID tag data) to a server. The input is the captured image and the read RFID data, and the output is the data to be sent to the server. The communication device transmits the data using a secure protocol (e.g., HTTPS).

[0447] Step 3:

[0448] The server analyzes the received data and stores the food type, quantity, and expiration date in a database. The input is the transmitted image and RFID data, and the output is the food data stored in the database. The server uses an image recognition algorithm (e.g., convolutional neural network) to identify the food type, match it with the RFID tag information, and update the database.

[0449] Step 4:

[0450] The server periodically scans the stored food data to detect foods approaching their expiration date. The input is the stored database information, and the output is a list of foods approaching their expiration date. The server performs batch processing and filters foods within a certain number of days of their expiration date using JSON or SQL queries.

[0451] Step 5:

[0452] The server generates notification messages about food products that are about to expire and sends them to the device. The input is the filtered expiration warning data, and the output is the notification message. The server generates a specific message such as "Milk will expire in 2 days" and sends it to the device via the push notification API.

[0453] Step 6:

[0454] The server collects data on the user's preferences, physical condition, season, weather, economic situation, and emotions. The input is the user's profile data and data from the emotion engine, and the output is the integrated information of these data. Information is sent periodically from the device, and the emotion engine obtains emotion data in real time.

[0455] Step 7:

[0456] The server compares the food data in the refrigerator with the user's individual data and generates the optimal recipe. The input is the food data, the user's profile information, and emotion data, and the output is a recipe generated by a generative AI model. Using a generative AI model (e.g., GPT-3), input a prompt sentence in the following format:

[0457] "There is one bottle of milk and six eggs in the refrigerator. The user is feeling stressed. Based on this, please suggest the best recipe for the user."

[0458] Step 8:

[0459] The server sends the generated recipe to the terminal and asks for feedback in a prepared form. The input is the generated recipe data, and the output is recipe information in HTML format and a feedback form. The recipe details are sent to the terminal, and the user can check the details on the terminal.

[0460] Step 9:

[0461] The server analyzes the food data in the refrigerator and lists the ingredients that are missing. The input is the stored food data, and the output is the list of ingredients that are missing.

[0462] Step 10:

[0463] The server generates automatic ordering information based on the user's purchasing history and preferences and notifies the user. The input is a list of missing ingredients and purchasing history data, and the output is the automatic ordering information and notification message. Notifications are sent to the user using a push notification API.

[0464] Step 11:

[0465] If the user approves, the server sends an order request to the online shopping site. The input is the approved order details, and the output is an order request to the online shopping site. An API is used to add the product to the shopping site's cart and start the order process.

[0466] Step 12:

[0467] The server notifies the terminal that the order has been placed. The input is the confirmation of the order being placed from the online shopping site, and the output is a notification message to the user. The order completion message is sent using the push notification API.

[0468] Step 13:

[0469] The server analyzes the user's dietary habits and calculates the nutritional balance. The input is the user's dietary habits and the output is the calculated nutritional balance. The server performs the nutrient analysis using SQL queries or dedicated algorithms.

[0470] Step 14:

[0471] The server identifies nutrient deficiencies and generates health advice. The input is the calculated nutritional balance, and the output is health advice. For example, it generates a recipe for a salad rich in green and yellow vegetables.

[0472] Step 15:

[0473] The server sends the health advice to the terminal and notifies the user. The input is the generated health advice and the output is the notification message. The notification is sent to the terminal and the user can check it.

[0474] Step 16:

[0475] The server obtains the user's emotional data in real time through the emotion engine. The input is the user's emotional state, and the output is emotional data. The user's emotional state is detected through micro-facial expression and voice analysis.

[0476] Step 17:

[0477] The server further personalizes recipes and health suggestions based on the acquired emotional data. The input is emotional data, and the output is personalized suggestions. For example, if you are feeling stressed, it generates a recipe for "chamomile tea," which has a relaxing effect.

[0478] Step 18:

[0479] The server sends sentiment-based suggestions to the device and notifies the user. The input is a personalized suggestion, and the output is a notification message. A notification is sent to the device, allowing the user to confirm the suggestion.

[0480] (Application example 2)

[0481] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0482] In today's dual-income households, managing food ingredients and maintaining a healthy diet is a significant burden. Furthermore, emotional states such as stress and fatigue often affect the entire lifestyle, making appropriate responses necessary. While current food management systems can manage ingredients and provide nutritional balance recommendations to a certain extent, they lack security advice and alert functions tailored to the user's emotional state. Therefore, there is a need for a comprehensive food management and health management system that takes emotional states into account.

[0483] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for recognizing the type, quantity, and expiration date of food products; means for transmitting the recognized food product data; means for saving the transmitted data and tracking the expiration date or shelf life; means for generating recipes based on the user's preferences, physical condition, season, weather, economic situation, and emotional state; means for transmitting and notifying the user of the generated recipes; means for checking for missing ingredients and automatically ordering them; means for analyzing the user's nutritional balance and generating advice to support a healthy diet; means for transmitting the generated advice; means for collecting user feedback and updating data; means for analyzing the user's emotional state and providing security advice and alerts based on the analysis; and means for recording fluctuations in the emotional state and generating a daily report. This enables food ingredient management and maintaining a healthy diet, as well as appropriate responses according to the user's emotional state, thereby improving the overall quality of life.

[0484] A "food management device" is a device that recognizes the type, quantity, and expiration date of food, and manages, transmits, and stores this data.

[0485] "Emotional state" refers to the user's mental and emotional state, and includes categories such as stress, fatigue, and relaxation.

[0486] "User preferences" refers to the user's preferred choices and tendencies regarding ingredients and dishes.

[0487] "Security Advice" is safety advice or suggestions provided based on the user's emotional state.

[0488] "Alerts" refer to warnings or notifications based on the user's emotional state or the state of the system.

[0489] "Recipe generation" is the process of creating cooking instructions based on the user's preferences, physical condition, season, weather, financial situation, and emotional state.

[0490] "Automatic ordering" refers to a system that checks for shortages of ingredients and automatically orders the necessary ingredients.

[0491] "Nutritional balance" refers to the harmony of nutrients that a user should consume in their diet.

[0492] "Feedback" refers to the opinions and thoughts provided by users to the system, and the system's data is updated based on them.

[0493] "Data transmission" refers to the process of transmitting the recognized food data to the server.

[0494] "Expiration date tracking" is the act of managing the expiration date and storage period of food based on the transmitted data and notifying the user appropriately.

[0495] "Supporting a healthy diet" means analyzing the user's nutritional balance and providing advice and suggestions for maintaining a proper diet.

[0496] "Report generation" is the process of recording fluctuations in emotional state and creating a summary report at the end of the day.

[0497] "Data updating" is the act of keeping the data in the system up to date based on user feedback.

[0498] System Overview

[0499] An embodiment of this invention includes a food management device and a security management system incorporating an emotion engine. The food management device uses sensors and cameras installed in the refrigerator to collect information such as food type, quantity, and expiration date, and sends the data to a server. The server analyzes this data to manage expiration dates and analyze the nutritional balance of food. It also recognizes the user's emotional state and provides appropriate recipes and security advice based on that information.

[0500] Hardware and Software

[0501] The system uses the following hardware and software:

[0502] Hardware:

[0503] Cameras and RFID readers inside refrigerators

[0504] Smartphone or tablet

[0505] Security systems (cameras, alarms)

[0506] software:

[0507] OpenCV: Image processing from camera

[0508] EmotionRecognizer (virtual emotion engine): Recognizes user emotions

[0509] Python: General application logic

[0510] Data processing and calculation methods

[0511] Food Recognition and Data Collection

[0512] Cameras and RFID readers identify the type, quantity and expiration date of food and transmit this data through a communication device inside the refrigerator to a server, which stores it in a database and tracks expiration dates and storage times.

[0513] Recognition of emotional states

[0514] It uses the smartphone's camera and microphone to capture the user's facial expressions and voice, which are then analyzed by an emotion engine. If the emotional state is recognized as stress or fatigue, it triggers a security alarm and issues a safety alert.

[0515] Recipe and advice generation

[0516] The server generates appropriate recipes based on the user's preferences, physical condition, season, weather, financial situation, and emotional state. It also analyzes the user's emotional state and provides security advice as needed. For example, if the user is feeling extremely stressed, it will suggest cooking or resting methods that will help reduce stress.

[0517] Daily report generation

[0518] It records fluctuations in emotional state and automatically generates a report at the end of each day summarizing the user's emotional state and corresponding behavior, including security and health management advice for the next day.

[0519] Prompt Sentence Examples

[0520] The detailed realization of this application also utilizes a generative AI model, with the following prompts as examples:

[0521] Please create a program that meets the following requirements as an example of a security service that applies a system that incorporates an emotion engine into a food management device for dual-income households to a smartphone app.

[0522] Recognizes user emotions (stress, fatigue, etc.) using a camera

[0523] Trigger security alerts based on emotions

[0524] Providing security advice

[0525] Generate daily reports based on stress levels

[0526] The application is written in Python and uses the OpenCV library.

[0527] In this way, the present invention uses a food management device incorporating an emotion engine to achieve efficient food management and support for a healthy diet, and also performs security management according to the user's emotional state, thereby improving the overall quality of life.

[0528] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0529] Step 1:

[0530] Recognizing food types, quantities, and expiration dates

[0531] A camera and RFID reader installed inside the refrigerator recognize the type, quantity, and expiration date of food. The recognized data is acquired as the food name, quantity, and expiration date and sent to the server via the communication device inside the refrigerator. This allows the server to obtain food inventory information.

[0532] Step 2:

[0533] Data storage and expiration date tracking

[0534] The server stores the submitted food data in a database, while simultaneously running algorithms that track expiration dates and shelf life to identify foods approaching their expiry date, which are then listed for possible user notification along with updates to the database.

[0535] Step 3:

[0536] Recognizing the user's emotional state

[0537] The user captures their voice and facial expressions using the smartphone's camera and microphone. The emotion engine analyzes this input data and recognizes their emotional state, such as stress, fatigue, or relaxation. The recognized emotional state is then sent to the server.

[0538] Step 4:

[0539] Generating recipes and security advice

[0540] The server generates appropriate recipes based on the user's emotional state, preferences, physical condition, season, weather, and economic situation. It may also provide security advice based on the user's emotional state. The generated recipes and advice are sent to the user's smartphone.

[0541] Step 5:

[0542] Automated ordering

[0543] The server analyzes the food data in the refrigerator and lists any ingredients that are missing. Based on this information, it generates an automatic order based on the user's purchasing history and preferences, and sends an order request to the online shopping site. Once the order is complete, the user is notified.

[0544] Step 6:

[0545] Analyzing the user's nutritional balance and generating advice

[0546] The server analyzes the user's eating habits and calculates nutritional balance. Based on this, it generates advice and recipes for maintaining a healthy diet and notifies the user, including salad recipes that encourage the intake of appropriate amounts of green and yellow vegetables.

[0547] Step 7:

[0548] Recording emotional state fluctuations and generating daily reports

[0549] The server records the user's emotional fluctuations throughout the day and compiles the data throughout the day. Based on this data, a summary report of the day is automatically generated, including security and health management advice for the next day. The report is then sent to the user's device.

[0550] Step 8:

[0551] Gathering feedback and updating the system

[0552] Users can send feedback on the recipes and advice provided to the server from their device, and the server receives this feedback and updates the system's database to improve the accuracy of future suggestions and advice.

[0553] This allows servers, terminals, and users to work together to achieve efficient food management, healthy living, and security management.

[0554] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0555] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0556] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0557] [Second embodiment]

[0558] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0559] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0560] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0561] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0562] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0563] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0564] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0565] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0566] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0567] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0568] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0569] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0570] The present invention provides a system that uses a food management device to reduce the burden of food ingredient management in dual-income households and supports efficient and healthy eating habits. Hereinafter, an embodiment of the present invention will be described in detail.

[0571] System Overview

[0572] This system uses various sensors and cameras installed inside the refrigerator to collect information such as food type, quantity, and expiration date, and sends this data to a server. The server analyzes this data to manage expiration dates and analyze the nutritional balance of food. It also generates optimal recipes based on the user's physical condition, preferences, season, weather, and economic situation, and notifies the user. It also automatically orders ingredients that are running low, preventing food waste and supporting healthy eating habits.

[0573] Food Recognition and Data Collection

[0574] Cameras and RFID readers installed inside the refrigerator recognize the type, quantity, and expiration date of food. This data is sent to a server via a communication device inside the refrigerator, and the server stores it in a database. For example, data such as "Milk (1 bottle, expiration date: October 10, 2023)" and "Eggs (6 eggs, expiration date: October 12, 2023)" may be acquired from the refrigerator.

[0575] Data storage and expiration date tracking

[0576] The data sent to the server is stored in a database, where expiration dates and storage periods are tracked. The server then notifies the user of food products that are approaching their expiration date. For example, if the expiration date of "milk" is approaching in two days, the server will send a notification to the device saying, "Milk's expiration date will expire in two days."

[0577] Recipe Suggestions

[0578] The server generates optimal recipes based on the user's preferences, physical condition, season, weather, and economic situation. For example, it generates a recipe for an "omelet" using milk and eggs based on the user's preferences and the ingredients currently in the refrigerator, and sends it to the device. The user can then refer to the recipe on the device and easily cook the dish.

[0579] Automatic ordering of necessary ingredients

[0580] The server analyzes the food data in the refrigerator and lists any ingredients that are running low. For example, if it detects that there are fewer than two eggs, the server generates an automatic ordering content based on the user's purchasing history and preferences. This order content is notified to the terminal, and if the user approves, an ordering request is sent to the online shopping site. After the order is completed, the user is notified that "the egg order has been completed."

[0581] Nutritional balance and health management

[0582] The server analyzes the user's eating habits and calculates nutritional balance. To support a healthy diet, it generates necessary health advice and recipes and notifies the device. For example, if the user has not eaten vegetables recently, it will suggest a salad recipe rich in green and yellow vegetables and notify the user. The user sends feedback to the server via their device, which updates the data throughout the system.

[0583] Specific examples

[0584] Next, the processing of the system will be described in detail using a specific example.

[0585] Example 1: Milk expiration date notification

[0586] The server retrieves information about "Milk (best before date: October 10, 2023)" from the database and confirms that the expiration date is two days later.

[0587] The server notifies the terminal that "The milk's expiration date will expire in two days."

[0588] The device displays this notification to the user.

[0589] Example 2: Omelette recipe suggestion

[0590] The server generates an omelet recipe based on the user's preferences and the information on milk and eggs in the refrigerator.

[0591] The server sends this recipe to the terminal.

[0592] The device notifies the user of the "omelette recipe."

[0593] Example 3: Automated egg ordering

[0594] The server determines that there are fewer than two eggs in the refrigerator.

[0595] The server generates an order for "eggs (pack of 10)" and sends an order request to the online shopping site.

[0596] The terminal notifies the user that "your egg order has been completed."

[0597] In this way, the present invention realizes efficient food management and support for a healthy diet using a food management device. This system reduces the burden of food ingredient management for dual-income households and provides a healthy diet that takes nutritional balance into consideration.

[0598] The processing flow will be explained below.

[0599] Step 1:

[0600] The refrigerator detects when the door is opened and closed, activating the internal camera and RFID reader. This recognizes data such as the type of food, quantity, and expiration date. For example, it obtains information such as "Milk (1 bottle, expiration date: October 10, 2023)" and "Eggs (6 eggs, expiration date: October 12, 2023)."

[0601] Step 2:

[0602] The refrigerator sends the food data it recognizes to the server, including the type of food, quantity, and expiration date.

[0603] Step 3:

[0604] The server stores the received food data in a database. The data is structured as information such as the type, quantity, expiration date, and storage period of each food item.

[0605] Step 4:

[0606] The server periodically checks the database and lists foods that are close to their expiration date. For example, it detects that the expiration date of "Milk (expiration date: October 10, 2023)" is just two days away.

[0607] Step 5:

[0608] The server prepares notifications for food items that are nearing their expiration date. For items with an expiration date within two days, the server prepares to notify the user.

[0609] Step 6:

[0610] The server sends the notification content to the terminal. For example, it sends notification data saying "The expiration date of the milk will be reached in two days."

[0611] Step 7:

[0612] The device displays the received notification to the user. For example, a notification pops up on the smartphone saying "The expiration date of the milk will expire in two days."

[0613] Step 8:

[0614] The server generates recipes based on the user's preferences, physical condition, season, weather, and economic situation, and analyzes various collected data to select the optimal recipe.

[0615] Step 9:

[0616] The server sends the generated recipe to the terminal. For example, it sends a "recipe for an omelet using milk and eggs."

[0617] Step 10:

[0618] The device notifies the user of the received recipe. For example, the smartphone displays "An omelet recipe has been suggested."

[0619] Step 11:

[0620] The server analyzes the food data in the refrigerator and lists the ingredients that are running low. For example, it might determine that there are fewer than two eggs.

[0621] Step 12:

[0622] The server generates automatic ordering based on the ingredients that are in short supply. Taking into account the user's purchasing history and preferences, it prepares an order request for "Eggs (pack of 10)."

[0623] Step 13:

[0624] The server sends the order to the terminal and asks the user for approval, for example, sending a confirmation message saying "Do you want to automatically order eggs?"

[0625] Step 14:

[0626] The terminal displays the received order confirmation to the user, who can then select "Accept" or "Cancel" in response to the confirmation message.

[0627] Step 15:

[0628] The user responds with "approval" to the order confirmation. The response data is sent from the terminal to the server.

[0629] Step 16:

[0630] The server receives the user's approval and sends an order request to the online shopping site. For example, it sends a request to order "eggs (pack of 10)."

[0631] Step 17:

[0632] The server sends a notification of order completion to the terminal, for example, data such as "Your egg order has been completed."

[0633] Step 18:

[0634] The device will notify the user that their eggs have been ordered. The user can then check the status of their order.

[0635] Step 19:

[0636] The server collects data on the user's eating habits and analyzes their nutritional balance. For example, it obtains information such as, "I've been eating less vegetables recently."

[0637] Step 20:

[0638] The server generates health advice and recipes based on the analysis results, such as a recipe for a salad rich in green and yellow vegetables.

[0639] Step 21:

[0640] The server sends the generated advice or recipe to the terminal, for example, sending a "salad recipe."

[0641] Step 22:

[0642] The device will notify the user of the advice or recipes it has received, and the user can view the notification and incorporate the advice or recipes.

[0643] Step 23:

[0644] If the user actually uses the suggested advice or recipe, feedback is sent via the device.

[0645] Step 24:

[0646] The server receives the feedback submitted by the user and updates the database, which improves the accuracy of the system and allows for a more personalized service.

[0647] Example 1

[0648] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0649] The present invention relates to a system that reduces the burden of food ingredient management for dual-income households and supports efficient and healthy eating habits. Conventional food management systems often require manual management of food expiration dates and analysis of nutritional balance, which can be time-consuming and labor-intensive for dual-income households. Furthermore, automatic food ordering and recipe suggestions may not be performed appropriately, potentially resulting in food waste and nutritional imbalances. A solution to these issues is needed.

[0650] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0651] In this invention, the server includes means for recognizing the type, quantity, and expiration date of food, means for transmitting the recognized food data, means for storing the transmitted data and tracking the expiration date or storage period, means for generating recipes based on the user's preferences, physical condition, season, weather conditions, and economic situation, means for transmitting and notifying the user of the generated recipes, means for checking for missing ingredients and automatically ordering, means for analyzing the user's nutritional balance and generating suggestions to support a healthy diet, means for transmitting the generated suggestions, means including a communication device for collecting food data, and means for collecting user feedback and updating the data. This reduces the burden of food management for dual-income households and makes it possible to support an efficient and healthy diet.

[0652] A "food management device" is a device that recognizes and manages information such as food type, quantity, and expiration date.

[0653] "Means for recognizing the type, quantity, and expiration date of food" refers to means for identifying the type, quantity, and expiration date of food using a camera, RFID reader, etc.

[0654] The "means for transmitting recognized food data" is a communication device for transmitting information about the recognized food to a server or database.

[0655] "Means for storing submitted data and tracking expiration dates or shelf life" refers to means for storing submitted food data in a database and managing and tracking expiration dates or shelf life.

[0656] "Means for generating recipes based on the user's preferences, physical condition, season, weather conditions, and economic situation" refers to a program or system that generates optimal recipes based on the user's personal information and external conditions.

[0657] The "means for transmitting and notifying the generated recipe" is a communication and notification system for transmitting and notifying the generated recipe to the user's terminal.

[0658] The "means of checking for shortages of ingredients and automatically ordering" is a system that analyzes food data in the refrigerator, identifies shortages of food, and automatically orders them.

[0659] The "means for analyzing a user's nutritional balance and generating suggestions to support a healthy diet" is a system that analyzes a user's eating habit data and generates suggestions to support a healthy diet from the perspective of nutritional balance.

[0660] The "means for transmitting the generated suggestions" is a communication system for transmitting the generated health suggestions to the user's terminal.

[0661] The "means including a communication device for collecting food data" is a system including a communication device for collecting food information and transmitting it to a server.

[0662] "Means for collecting user feedback and updating data" refers to means that have the function of receiving feedback from users and updating data throughout the system based on that feedback.

[0663] MODE FOR CARRYING OUT THE INVENTION

[0664] The present invention provides a system that uses a food management device to reduce the burden of food ingredient management in dual-income households and supports efficient and healthy eating habits. Hereinafter, embodiments of the present invention will be described in detail.

[0665] System Overview

[0666] This system uses various sensors and cameras installed inside the refrigerator to collect information such as food type, quantity, and expiration date, and sends this data to a server. The server analyzes this data to manage expiration dates and analyze the nutritional balance of food. It also generates optimal recipes based on the user's physical condition, preferences, season, weather conditions, and economic situation, and notifies the user. It also automatically orders ingredients that are running low, preventing food waste and supporting healthy eating habits.

[0667] Food Recognition and Data Collection

[0668] Cameras and RFID readers installed inside the refrigerator recognize the type, quantity, and expiration date of food. This data is sent to a server via a communication device inside the refrigerator, and the server stores it in a database. For example, data such as "Milk (1 bottle, expiration date: October 10, 2023)" and "Eggs (6 eggs, expiration date: October 12, 2023)" may be acquired from the refrigerator.

[0669] Data storage and expiration date tracking

[0670] The data sent to the server is stored in a database, where expiration dates and storage periods are tracked. The server then notifies the user of food products that are approaching their expiration date. For example, if the expiration date of "milk" is approaching in two days, the server will send a notification to the device saying, "Milk's expiration date will expire in two days."

[0671] Recipe Suggestions

[0672] The server generates optimal recipes based on the user's preferences, physical condition, season, weather conditions, and economic situation. For example, it generates a recipe for an "omelet" using milk and eggs based on the user's preferences and the ingredients currently in the refrigerator, and sends it to the device. The user can then refer to the recipe on the device and easily cook the dish.

[0673] Automatic ordering of necessary ingredients

[0674] The server analyzes the food data in the refrigerator and lists any ingredients that are running low. For example, if it detects that there are fewer than two eggs, the server generates an automatic ordering content based on the user's purchasing history and preferences. This order content is notified to the terminal, and if the user approves, an ordering request is sent to the e-commerce site. After the order is completed, the server sends a notification to the user that "the egg order has been completed."

[0675] Nutritional balance and health management

[0676] The server analyzes the user's eating habits and calculates nutritional balance. To support a healthy diet, it generates necessary health advice and recipes and notifies the device. For example, if the user has not eaten vegetables recently, it will suggest a salad recipe rich in green and yellow vegetables and notify the user. The user sends feedback to the server via their device, which updates the data throughout the system.

[0677] Specific examples

[0678] Next, the processing of the system will be described in detail using a specific example.

[0679] Example 1: Milk expiration date notification

[0680] The server retrieves information about "Milk (best before date: October 10, 2023)" from the database and confirms that the expiration date is two days later.

[0681] The server notifies the terminal that "The milk's expiration date will expire in two days."

[0682] The device displays this notification to the user.

[0683] Example 2: Omelette recipe suggestion

[0684] The server generates an omelet recipe based on the user's preferences and the information on milk and eggs in the refrigerator.

[0685] The server sends this recipe to the terminal.

[0686] The device notifies the user of the "omelette recipe."

[0687] Example 3: Automated egg ordering

[0688] The server determines that there are fewer than two eggs in the refrigerator.

[0689] The server generates an order for "Eggs (pack of 10)" and sends the order request to the e-commerce site.

[0690] The terminal notifies the user that "your egg order has been completed."

[0691] Prompt Sentence Examples

[0692] Below is an example of a prompt sentence to input to the generative AI model.

[0693] "Please analyze the food in my refrigerator and its expiration date and suggest the best recipes."

[0694] "Analyze the ingredients that are in short supply and use an automated ordering system to generate orders for the ingredients that are needed."

[0695] "Based on the user's eating habits data, create nutritional advice to support healthy eating habits."

[0696] In this way, the present invention realizes efficient food management and support for a healthy diet using a food management device. This system reduces the burden of food ingredient management for dual-income households and provides a healthy diet that takes nutritional balance into consideration.

[0697] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0698] Step 1: Collect food data

[0699] Input: Food in the fridge

[0700] Output: Food type, quantity, expiration date data

[0701] What happens:

[0702] A camera installed inside the refrigerator takes pictures of the food, and an RFID reader reads the data from the RFID tags.

[0703] The communication device inside the refrigerator acquires the image data from the camera and the tag data from the RFID reader.

[0704] The data obtained includes the type of food (e.g., "milk" or "egg"), quantity (e.g., "1 bottle" or "6 pieces"), and expiration date (e.g., "October 10, 2023" or "October 12, 2023").

[0705] The communication device transmits this data to the server.

[0706] Step 2: Data storage and analysis

[0707] Input: Food type, quantity, expiration date data

[0708] Output: Saved food data, analysis results

[0709] What happens:

[0710] The server receives the food data transmitted from the refrigerator.

[0711] The received data includes the type of food, quantity, expiration date, etc.

[0712] The server stores this data in a database (e.g., "Milk (1 bottle, expiration date: October 10, 2023)" and "Eggs (6 eggs, expiration date: October 12, 2023)").

[0713] The server analyzes the stored data to identify current food inventory and foods that are close to their expiration date.

[0714] Step 3: Best before date notification

[0715] Input: Food data in the database

[0716] Output: Best before date notification message

[0717] What happens:

[0718] The server periodically scans the food data in the database to identify foods that are approaching their expiration date.

[0719] For example, find "milk" whose expiration date is just two days away.

[0720] The server generates a notification message saying "Milk expires in 2 days."

[0721] Send this message to the user's terminal.

[0722] The device will display this notification to the user.

[0723] Step 4: Recipe suggestions

[0724] Input: User preferences, physical condition, season, weather conditions, economic situation, food data in refrigerator

[0725] Output: Recipe suggestion message

[0726] What happens:

[0727] The server collects data on the user's preferences, physical condition, season, weather conditions, and economic situation.

[0728] It also retrieves food data from the refrigerator and checks the current ingredients.

[0729] For example, suppose you have milk and eggs in your refrigerator.

[0730] The server uses this information to input prompts into the generative AI model (e.g., "There is currently milk and eggs in the refrigerator. Please suggest some recipes using these.").

[0731] The generative AI model generates a recipe for "omelette" based on prompts.

[0732] The server sends this generated recipe to the user's terminal, and the user is notified of the "omelette recipe."

[0733] Step 5: Automatically order the ingredients you need

[0734] Input: Food data in the refrigerator

[0735] Output: Order completion notification message

[0736] What happens:

[0737] The server analyzes the database and detects that there is a shortage of food in the refrigerator.

[0738] For example, identify a decrease to "less than two eggs."

[0739] The server generates an order (e.g., "pack of 10 eggs") based on the user's past purchasing history and preferences.

[0740] The server sends the order details to the electronic commerce site as an order request.

[0741] After the order is completed, the server sends a notification to the user's terminal saying "Egg order completed."

[0742] Step 6: Nutritional Balance and Health Management

[0743] Input: User's dietary habits data

[0744] Output: Health suggestion message

[0745] What happens:

[0746] The server periodically analyzes the user's eating habits data.

[0747] Based on the analysis results, nutritional balance is calculated and any missing nutrients are identified.

[0748] For example, it detects when a user has not eaten vegetables recently.

[0749] The server generates recipes for salads rich in green vegetables to support a healthy diet.

[0750] This generated recipe is sent to the user's device and notified to the user.

[0751] Users send feedback to the server through their devices, and data is updated throughout the system.

[0752] (Application example 1)

[0753] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0754] In busy households, such as dual-income households, it is difficult to manage food and ensure a balanced diet. Even when shopping at brick-and-mortar stores, it can be time-consuming to know what to buy and whether there are any foods nearing their expiration date. Furthermore, there is a need for a method to efficiently procure ingredients while maintaining a healthy diet. A system that solves these issues and supports food management and healthy eating habits at home is needed.

[0755] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0756] In this invention, the server includes means for recognizing the type, quantity, and expiration date of food, means for transmitting the recognized food data, means for storing the transmitted data and tracking the expiration date or storage period, means for generating recipes based on the user's preferences, physical condition, season, weather, and economic situation, means for transmitting and notifying the user of the generated recipes, means for checking for missing ingredients and automatically ordering, means for analyzing the user's nutritional balance and generating advice to support a healthy diet, means for transmitting the generated advice, means for collecting user feedback and updating data, means for displaying ingredient information in a physical store via smart glasses and presenting a list of ingredients that need to be replenished, and means for presenting recipes and special sale information generated based on the user's profile and external environmental information in real time, thereby enabling more efficient food management and supporting a healthy diet.

[0757] A "food management device" is a device for recognizing and managing information such as the type, quantity, and expiration date of food.

[0758] A "sensor" is a device that detects a physical quantity or state and outputs it as an electrical signal.

[0759] A "camera" is a device that can capture video or still images and store and transmit them as digital data.

[0760] A "communication device" is a device for sending and receiving data to and from other devices and servers.

[0761] A "server" is a computer system that stores, analyzes, and processes data and provides information in response to requests from other devices.

[0762] "User preferences" refers to information that indicates the user's taste and cooking preferences.

[0763] "Physical condition" is information that indicates the user's health condition and physical condition.

[0764] "Season" refers to the time of year brought about by changes in the climate.

[0765] "Weather" is information indicating weather conditions in a particular area and time.

[0766] "Financial status" is information that indicates the user's financial status, such as income, expenses, and budget.

[0767] A "recipe" is a detailed list of instructions and ingredients needed to make a particular dish.

[0768] "Notification" means sending messages or alerts to inform users.

[0769] "Auto ordering" is the process of automatically ordering necessary items and ingredients.

[0770] "Nutritional balance" refers to a state in which the user's diet properly incorporates the necessary nutrients.

[0771] "Advice" is information that provides advice or recommendations to the user.

[0772] "Feedback" refers to opinions and evaluations provided by users to the system.

[0773] "Smart glasses" are wearable devices that have the ability to display visual information.

[0774] A "brick and mortar store" is a physical commercial establishment where users can visit and purchase products.

[0775] "Food information" refers to data such as food type, quantity, expiration date, and storage conditions.

[0776] A "profile" is a data set that compiles personal information and characteristics about a user.

[0777] "External environment information" refers to information that is not directly managed by the user, such as weather, season, and surrounding conditions.

[0778] "Special sale information" is information about products being sold at a price lower than the regular price.

[0779] The present invention is a system for supporting home food management and healthy eating habits. The purpose of the present invention is to streamline food management in physical stores through smart glasses and improve the user's shopping experience. Specific embodiments for implementing the present invention are described in detail below.

[0780] System Overview

[0781] This system uses various sensors and cameras installed inside the refrigerator to collect information such as food type, quantity, and expiration date, and sends this data to a server. The server analyzes this data to manage expiration dates and analyze the nutritional balance of food. In addition, when a user wearing smart glasses purchases ingredients in a physical store, the system displays real-time information on ingredients, special offers, recipe suggestions, and more.

[0782] Hardware and Software

[0783] Hardware:

[0784] Cameras and sensors inside the refrigerator: Used to collect food data.

[0785] Smart glasses: Using "Google Glass" and "Vuzix Blade" as examples.

[0786] Server: We use AWS and Azure for data analysis and notifications.

[0787] software:

[0788] Data collection software: Collects data from cameras and sensors inside the refrigerator and sends it to a server.

[0789] Data analysis software: Food data, user information, and external environmental information are analyzed on the server.

[0790] User Interface: An application for display on smart glasses.

[0791] Specific processing

[0792] 1. Food Recognition and Data Collection:

[0793] Cameras and RFID readers installed inside the refrigerator identify the type, quantity and expiration date of food and send this data to a server, which stores it in a database and tracks expiration dates.

[0794] 2. Information display through smart glasses:

[0795] When users shop in a physical store, the smart glasses display real-time information about the ingredients in their refrigerator. For example, if milk or eggs are low in stock, the glasses notify the user. They also display sales and coupon information to support their purchasing behavior.

[0796] 3. Recipe and nutritional suggestions:

[0797] The server uses a generative AI model to generate appropriate recipes based on the user's preferences, physical condition, season, weather, and economic situation. The generated recipes are then sent to the user via smart glasses. The server also analyzes nutritional balance and sends advice to support healthy eating.

[0798] Specific examples

[0799] A specific example of the system's operation is shown below.

[0800] Example 1: Shopping assistance in physical stores

[0801] The user wears smart glasses and goes shopping in a physical store. The smart glasses' display shows real-time information about the food inventory in the refrigerator. A notification such as "You're low on milk and need to buy more" is displayed, along with information about special sales and coupons available in the store.

[0802] Example 2: Recipe suggestions

[0803] When a user is using the smart glasses at home, the server generates an "omelette" recipe based on the user's current health condition, preferences, and information about ingredients in the refrigerator, and notifies the smart glasses. The following prompt sentence is used for the generating AI model:

[0804] Example prompt for a generative AI model:

[0805] "The user has a bottle of milk and two eggs in the refrigerator. It's fall, and the user prefers a balanced diet. Based on this, please suggest some suitable recipes."

[0806] In this way, the present invention can utilize smart glasses to comprehensively support food management in brick-and-mortar stores and improve users' quality of life.

[0807] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0808] Step 1:

[0809] Food data collection

[0810] The server receives input from cameras and RFID readers installed inside the refrigerator. This input includes the type of food, quantity, and expiration date. The server analyzes this data and stores it in a database for each food item. For example, information such as "Milk (Quantity: 1 bottle, Best before date: October 10, 2023)" may be collected.

[0811] Step 2:

[0812] Best before date tracking

[0813] The server periodically analyzes the expiration dates of the food items stored in the database and lists the food items that are approaching their expiration date. For example, it lists the food items whose expiration date is two days away and prepares a notification such as "The expiration date will expire in two days." This notification is sent to the user's device.

[0814] Step 3:

[0815] Detecting missing ingredients

[0816] The server analyzes the food data in the database and detects ingredients that are low in stock. For example, if it determines that there are fewer than two eggs, it adds them to a list of ingredients that are in short supply. The server notifies the user of the list of ingredients that are in short supply, and clearly indicates which ingredients need to be replenished.

[0817] Step 4:

[0818] Preparing for automatic ordering

[0819] The server generates an automatic order for missing ingredients based on the user's purchasing history and preferences. For example, it prepares an order for "eggs (pack of 10)" and prepares to send a request to an online shopping site. This order preparation data is notified to the user and they are asked to approve it.

[0820] Step 5:

[0821] Recipe Generation

[0822] Based on the input data of the user's preferences, physical condition, season, weather, and economic situation, the server uses a generative AI model to generate the optimal recipe. The generative AI model is input with the following prompt:

[0823] "The user has a bottle of milk and two eggs in the refrigerator. It's fall, and the user prefers a balanced diet. Based on this, please suggest some suitable recipes."

[0824] The generated recipe, for example an "omelette" recipe, is completed and sent to the user terminal.

[0825] Step 6:

[0826] Information display on smart glasses

[0827] When a user wears the smart glasses and goes shopping in a physical store, the device displays real-time information about ingredients in the refrigerator and a list of ingredients that are running low. For example, a notification will appear on the smart glasses saying, "You're low on milk and need to buy more." In addition, the device will display information about special sales and coupons in the store, allowing users to shop efficiently.

[0828] Step 7:

[0829] Nutritional balance analysis and advice delivery

[0830] The server analyzes the user's eating habits, calculates nutritional balance, and generates advice to support a healthy diet. For example, it generates advice such as "You haven't been eating enough vegetables recently," and notifies the user's device. This advice collects user feedback and serves as the basis for updating data throughout the system.

[0831] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0832] The present invention provides a system that uses a food management device combined with an emotion engine to reduce the burden of food ingredient management for dual-income households and support efficient and healthy eating habits. The following describes in detail the embodiments of the present invention.

[0833] System Overview

[0834] This system uses sensors and cameras installed inside the refrigerator to collect information such as food type, quantity, and expiration date, and sends this data to a server. The server analyzes this data to manage expiration dates and analyze the nutritional balance of food. It also generates optimal recipes based on the user's physical condition, preferences, season, weather, economic situation, and emotions, and notifies the user. It also automatically orders ingredients that are running low, preventing food waste and supporting healthy eating habits.

[0835] Food Recognition and Data Collection

[0836] Cameras and RFID readers installed inside the refrigerator recognize the type, quantity, and expiration date of food. This data is sent to a server via a communication device inside the refrigerator, and the server stores it in a database. For example, data such as "Milk (1 bottle, expiration date: October 10, 2023)" and "Eggs (6 eggs, expiration date: October 12, 2023)" may be acquired from the refrigerator.

[0837] Data storage and expiration date tracking

[0838] The data sent to the server is stored in a database, where expiration dates and storage periods are tracked. The server then notifies the user of food products that are approaching their expiration date. For example, if the expiration date of "milk" is approaching in two days, the server will send a notification to the device saying, "Milk's expiration date will expire in two days."

[0839] Recipe Suggestions

[0840] The server generates optimal recipes based on the user's preferences, physical condition, season, weather, economic situation, and emotional data obtained by the emotion engine. For example, it generates a recipe for an "omelette" using "milk" and "eggs" based on the user's preferences and the ingredients currently in the refrigerator, and sends it to the device. The user can then refer to the recipe on their device and easily cook the dish.

[0841] Automatic ordering of necessary ingredients

[0842] The server analyzes the food data in the refrigerator and lists any ingredients that are running low. For example, if it detects that there are fewer than two eggs, the server generates an automatic ordering content based on the user's purchasing history and preferences. This order content is notified to the terminal, and if the user approves, an ordering request is sent to the online shopping site. After the order is completed, the user is notified that "the egg order has been completed."

[0843] Nutritional balance and health management

[0844] The server analyzes the user's eating habits and calculates nutritional balance. To support a healthy diet, it generates necessary health advice and recipes and notifies the device. For example, if the user has not eaten vegetables recently, it will suggest a salad recipe rich in green and yellow vegetables and notify the user. The user sends feedback to the server via their device, which updates the data throughout the system.

[0845] Further suggestions from the emotion engine

[0846] The emotion engine recognizes the user's emotions and further personalizes recipes and health advice based on this. For example, if the user is feeling stressed, it will suggest relaxing recipes that will help reduce stress.

[0847] Specific examples

[0848] Example 1: Emotion-based recipe suggestions

[0849] The server collects the user's emotional state through an emotion engine. For example, if the user is "feeling stressed," emotion data is acquired.

[0850] The server generates relaxation recipes, including chamomile tea, which is effective in reducing stress.

[0851] The server sends this relaxation recipe to the terminal.

[0852] The device will display a notification to the user saying, "A suggestion has been made for making chamomile tea."

[0853] Example 2: Emotion-based health advice

[0854] The server obtains emotional data from the emotion engine, such as "feeling tired" by the user.

[0855] The server generates advice including "vitamin-rich dishes" that are effective in relieving fatigue.

[0856] The server sends this advice to the terminal.

[0857] The device will display a notification to the user saying, "A dish that is effective in relieving fatigue has been suggested."

[0858] In this way, the present invention realizes efficient food management and support for healthy eating habits using a food management device with an emotion engine built in. This system reduces the burden of food ingredient management for dual-income households and provides a healthy diet that takes nutritional balance into consideration.

[0859] The processing flow will be explained below.

[0860] Step 1:

[0861] The refrigerator detects when the door is opened and closed, activating the internal camera and RFID reader. This recognizes data such as the type of food, quantity, and expiration date. For example, it obtains information such as "Milk (1 bottle, expiration date: October 10, 2023)" and "Eggs (6 eggs, expiration date: October 12, 2023)."

[0862] Step 2:

[0863] The refrigerator sends the food data it recognizes to the server, including the type of food, quantity, and expiration date.

[0864] Step 3:

[0865] The server stores the received food data in a database. The data is structured as information such as the type, quantity, expiration date, and storage period of each food item.

[0866] Step 4:

[0867] The server periodically checks the database and lists foods that are close to their expiration date. For example, it detects that the expiration date of "Milk (expiration date: October 10, 2023)" is just two days away.

[0868] Step 5:

[0869] The server prepares notifications for food items that are nearing their expiration date. For items with an expiration date within two days, the server prepares to notify the user.

[0870] Step 6:

[0871] The server sends the notification content to the terminal. For example, it sends notification data saying "The expiration date of the milk will be reached in two days."

[0872] Step 7:

[0873] The device displays the received notification to the user. For example, a notification pops up on the smartphone saying "The expiration date of the milk will expire in two days."

[0874] Step 8:

[0875] The server generates optimal recipes based on the user's preferences, physical condition, season, weather, economic situation, and emotional data obtained by the emotion engine. For example, it generates a recipe for "omelette with milk and eggs" based on the user's preferences and the ingredients currently in the refrigerator.

[0876] Step 9:

[0877] The server sends the generated recipe to the terminal. For example, it sends a "recipe for an omelet using milk and eggs."

[0878] Step 10:

[0879] The device notifies the user of the received recipe, for example, "An omelet recipe has been suggested."

[0880] Step 11:

[0881] The server analyzes the food data in the refrigerator and lists the ingredients that are running low. For example, it might determine that there are fewer than two eggs.

[0882] Step 12:

[0883] The server generates automatic ordering based on the ingredients that are in short supply. Taking into account the user's purchasing history and preferences, it prepares an order request for "Eggs (pack of 10)."

[0884] Step 13:

[0885] The server sends the order to the terminal and asks the user for approval, for example, sending a confirmation message saying "Do you want to automatically order eggs?"

[0886] Step 14:

[0887] The terminal displays the received order confirmation to the user, who can then select "Accept" or "Cancel" in response to the confirmation message.

[0888] Step 15:

[0889] The user responds with "approval" to the order confirmation. The response data is sent from the terminal to the server.

[0890] Step 16:

[0891] The server receives the user's approval and sends an order request to the online shopping site. For example, it sends a request to order "eggs (pack of 10)."

[0892] Step 17:

[0893] The server sends a notification of order completion to the terminal, for example, data such as "Your egg order has been completed."

[0894] Step 18:

[0895] The device will notify the user that their eggs have been ordered. The user can then check the status of their order.

[0896] Step 19:

[0897] The server collects data on the user's eating habits and analyzes their nutritional balance. For example, it obtains information such as, "I've been eating less vegetables recently."

[0898] Step 20:

[0899] The server generates health advice and recipes based on the analysis results, such as a recipe for a salad rich in green and yellow vegetables.

[0900] Step 21:

[0901] The server sends the generated advice or recipe to the terminal, for example, sending a "salad recipe."

[0902] Step 22:

[0903] The device will notify the user of the advice or recipes it has received, and the user can view the notification and incorporate the advice or recipes.

[0904] Step 23:

[0905] The server collects the user's emotional state through the emotion engine. For example, if the user says "I'm feeling stressed," the emotion data is acquired.

[0906] Step 24:

[0907] The server analyzes the emotional data and generates recipes and advice based on the user's emotional state. For example, it generates a relaxation recipe that includes chamomile tea, which is effective in reducing stress.

[0908] Step 25:

[0909] The server sends the generated relaxation recipe to the device, for example, "How to make chamomile tea."

[0910] Step 26:

[0911] The device notifies the user of the relaxation recipe it has received, for example, by displaying "A chamomile tea recipe has been suggested."

[0912] Step 27:

[0913] The server receives the feedback submitted by the user and updates the database, which improves the accuracy of the system and allows for a more personalized service.

[0914] Example 2

[0915] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0916] In dual-income households, it is difficult to maintain an efficient and healthy diet while reducing the burden of food management. There is also a need for optimal dietary recommendations based on the user's emotions and physical condition, but existing systems have difficulty comprehensively considering these factors. Therefore, there is a need for a system that automates the management of food types, quantities, and expiration dates, and also provides personalized recipes and advice based on the user's individual condition.

[0917] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0918] A means of recognizing the type, quantity, and expiration date of food;

[0919] means for transmitting the recognized food data;

[0920] A means for storing the transmitted data and tracking expiration dates or shelf life;

[0921] A means for generating recipes based on the user's preferences, physical condition, season, weather, and economic situation;

[0922] A means for transmitting and notifying the generated recipe;

[0923] A way to check for shortages of ingredients and automatically order them,

[0924] A means to analyze the user's nutritional balance and generate advice to support a healthy diet;

[0925] means for transmitting the generated advice;

[0926] A means of collecting user feedback and updating the data;

[0927] A means to recognize user emotions using an emotion engine and personalize recipes and advice based on this;

[0928] A means to use a generative AI model to generate prompts based on data on food in the refrigerator and user sentiment data, and to suggest recipes and advice based on those prompts;

[0929] A means to transmit the entire food recognition and management process to a server through a secure communication protocol

[0930] This will reduce the burden of food management for dual-income households and enable efficient and healthy eating habits. It will also enable personalized suggestions that take into account the user's emotions and physical condition.

[0931] A "food management device" is a device that manages information such as the type, quantity, and expiration date of food, and provides this information to users.

[0932] The "recognition means" refers to a method or device for detecting and processing information such as the type, quantity, and expiration date of food.

[0933] A "means for transmitting" is a method or device for transmitting the recognized information to another device or system.

[0934] The "storing means" refers to a method or device for storing the transmitted data in a recording medium such as a database.

[0935] A "tracking means" is a method or device for monitoring stored data and updating information or taking action based on certain conditions.

[0936] A "recipe generator" is a method or device for suggesting cooking steps and ingredients based on a user's specific requirements.

[0937] "Notification means" refers to a method or device for notifying the user of the generated information.

[0938] The "automatic ordering means" is a method or device for detecting shortages of ingredients and automatically carrying out the ordering procedure.

[0939] The "means for generating advice" is a method or device for generating suggestions to support a healthy diet based on the user's nutritional balance and health status.

[0940] A "means for collecting feedback" is a method or device for gathering user responses and opinions and updating the system.

[0941] An "emotion engine" is a technology that recognizes a user's emotional state and generates personalized suggestions based on that data.

[0942] A "generative AI model" is an artificial intelligence model that automatically generates text or suggestions based on input data.

[0943] A "prompt" is text containing a question or command that is input to a generative AI model.

[0944] A "secure communication protocol" is a communication protocol that protects data transmission and prevents unauthorized access by third parties.

[0945] MODE FOR CARRYING OUT THE INVENTION

[0946] The present invention provides a system that uses a food management device combined with an emotion engine to reduce the burden of food ingredient management for dual-income households and support efficient and healthy eating habits. Specific embodiments for implementing the present invention are described below.

[0947] System Overview

[0948] This system uses sensors and cameras installed inside the refrigerator to collect information such as food type, quantity, and expiration date, and sends this data to a server. The server analyzes this data to manage expiration dates and analyze the nutritional balance of food. It also generates optimal recipes based on the user's physical condition, preferences, season, weather, economic situation, and emotions, and notifies the user. It also automatically orders ingredients that are running low, preventing food waste and supporting healthy eating habits.

[0949] Food Recognition and Data Collection

[0950] Cameras and RFID readers installed inside the refrigerator recognize the type, quantity, and expiration date of food. This data is sent to a server via a communication device inside the refrigerator, and the server stores it in a database. For example, data such as "Milk (1 bottle, expiration date: October 10, 2023)" and "Eggs (6 eggs, expiration date: October 12, 2023)" may be acquired from the refrigerator. In this case, the camera captures high-resolution images inside the refrigerator, and the RFID reader reads the RFID tags on the food items and obtains the information contained therein.

[0951] Data storage and expiration date tracking

[0952] The data sent to the server is stored in a database, where expiration dates and storage periods are tracked. The server then notifies the user of food products that are approaching their expiration date. For example, if the expiration date of "milk" is approaching in two days, the server will send a notification to the device saying, "Milk's expiration date will expire in two days."

[0953] Recipe Suggestions

[0954] The server generates optimal recipes based on the user's preferences, physical condition, season, weather, economic situation, and emotional data obtained by the emotion engine. Using a generative AI model, the server generates prompts based on the food data in the refrigerator and the user's emotional data, and suggests recipes and advice based on these. For example, it generates a recipe for an "omelette" using the "milk" and "eggs" in the refrigerator and sends it to the device. The user can then refer to the recipe on the device and cook the food. Examples of prompts are as follows:

[0955] "There is one bottle of milk and six eggs in the refrigerator. The user is feeling stressed. Based on this, please suggest the best recipe for the user."

[0956] Automatic ordering of necessary ingredients

[0957] The server analyzes the food data in the refrigerator and lists the ingredients that are running low. For example, if there are fewer than two eggs, the server generates an automatic ordering content based on the user's purchasing history and preferences. This order content is notified to the terminal, and if the user approves, an ordering request is sent to the online shopping site. After the order is completed, the user is notified that "the egg order has been completed."

[0958] Nutritional balance and health management

[0959] The server analyzes the user's eating habits and calculates nutritional balance. To support a healthy diet, it generates necessary health advice and recipes and notifies the device. For example, if the user has not eaten vegetables recently, it will suggest a salad recipe rich in green and yellow vegetables and notify the user. The user sends feedback to the server via their device, which updates the data throughout the system.

[0960] Further suggestions from the emotion engine

[0961] The emotion engine recognizes the user's emotions and further personalizes recipes and health advice based on this. For example, if the user is feeling stressed, it will suggest relaxing recipes that will help reduce stress.

[0962] Specific examples

[0963] Example 1: Emotion-based recipe suggestions

[0964] The server collects the user's emotional state through an emotion engine. For example, if the user is "feeling stressed," emotion data is acquired.

[0965] The server generates relaxation recipes, including chamomile tea, which is effective in reducing stress.

[0966] The server sends this relaxation recipe to the terminal.

[0967] The device will display a notification to the user saying, "A suggestion has been made for making chamomile tea."

[0968] Example 2: Emotion-based health advice

[0969] The server obtains emotional data from the emotion engine, such as "feeling tired" by the user.

[0970] The server generates advice including "vitamin-rich dishes" that are effective in relieving fatigue.

[0971] The server sends this advice to the terminal.

[0972] The device will display a notification to the user saying, "A dish that is effective in relieving fatigue has been suggested."

[0973] In this way, users can utilize a system that supports efficient food management and healthy eating habits based on food data and emotional data in their refrigerator.

[0974] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0975] Step 1:

[0976] A camera installed in the refrigerator takes pictures of the food items, and an RFID reader retrieves the RFID tag information for each food item. The input is the food items in the refrigerator, and the output is the food images and RFID tag data. The camera captures high-resolution images, and the RFID reader reads the food information (e.g., type, quantity, expiration date).

[0977] Step 2:

[0978] A communication device in the refrigerator transmits the acquired data (food images and RFID tag data) to a server. The input is the captured image and the read RFID data, and the output is the data to be sent to the server. The communication device transmits the data using a secure protocol (e.g., HTTPS).

[0979] Step 3:

[0980] The server analyzes the received data and stores the food type, quantity, and expiration date in a database. The input is the transmitted image and RFID data, and the output is the food data stored in the database. The server uses an image recognition algorithm (e.g., convolutional neural network) to identify the food type, match it with the RFID tag information, and update the database.

[0981] Step 4:

[0982] The server periodically scans the stored food data to detect foods approaching their expiration date. The input is the stored database information, and the output is a list of foods approaching their expiration date. The server performs batch processing and filters foods within a certain number of days of their expiration date using JSON or SQL queries.

[0983] Step 5:

[0984] The server generates notification messages about food products that are about to expire and sends them to the device. The input is the filtered expiration warning data, and the output is the notification message. The server generates a specific message such as "Milk will expire in 2 days" and sends it to the device via the push notification API.

[0985] Step 6:

[0986] The server collects data on the user's preferences, physical condition, season, weather, economic situation, and emotions. The input is the user's profile data and data from the emotion engine, and the output is the integrated information of these data. Information is sent periodically from the device, and the emotion engine obtains emotion data in real time.

[0987] Step 7:

[0988] The server compares the food data in the refrigerator with the user's individual data and generates the optimal recipe. The input is the food data, the user's profile information, and emotion data, and the output is a recipe generated by a generative AI model. Using a generative AI model (e.g., GPT-3), input a prompt sentence in the following format:

[0989] "There is one bottle of milk and six eggs in the refrigerator. The user is feeling stressed. Based on this, please suggest the best recipe for the user."

[0990] Step 8:

[0991] The server sends the generated recipe to the terminal and asks for feedback in a prepared form. The input is the generated recipe data, and the output is recipe information in HTML format and a feedback form. The recipe details are sent to the terminal, and the user can check the details on the terminal.

[0992] Step 9:

[0993] The server analyzes the food data in the refrigerator and lists the ingredients that are missing. The input is the stored food data, and the output is the list of ingredients that are missing.

[0994] Step 10:

[0995] The server generates automatic ordering information based on the user's purchasing history and preferences and notifies the user. The input is a list of missing ingredients and purchasing history data, and the output is the automatic ordering information and notification message. Notifications are sent to the user using a push notification API.

[0996] Step 11:

[0997] If the user approves, the server sends an order request to the online shopping site. The input is the approved order details, and the output is an order request to the online shopping site. An API is used to add the product to the shopping site's cart and start the order process.

[0998] Step 12:

[0999] The server notifies the terminal that the order has been placed. The input is the confirmation of the order being placed from the online shopping site, and the output is a notification message to the user. The order completion message is sent using the push notification API.

[1000] Step 13:

[1001] The server analyzes the user's dietary habits and calculates the nutritional balance. The input is the user's dietary habits and the output is the calculated nutritional balance. The server performs the nutrient analysis using SQL queries or dedicated algorithms.

[1002] Step 14:

[1003] The server identifies nutrient deficiencies and generates health advice. The input is the calculated nutritional balance, and the output is health advice. For example, it generates a recipe for a salad rich in green and yellow vegetables.

[1004] Step 15:

[1005] The server sends the health advice to the terminal and notifies the user. The input is the generated health advice and the output is the notification message. The notification is sent to the terminal and the user can check it.

[1006] Step 16:

[1007] The server obtains the user's emotional data in real time through the emotion engine. The input is the user's emotional state, and the output is emotional data. The user's emotional state is detected through micro-facial expression and voice analysis.

[1008] Step 17:

[1009] The server further personalizes recipes and health suggestions based on the acquired emotional data. The input is emotional data, and the output is personalized suggestions. For example, if you are feeling stressed, it generates a recipe for "chamomile tea," which has a relaxing effect.

[1010] Step 18:

[1011] The server sends sentiment-based suggestions to the device and notifies the user. The input is a personalized suggestion, and the output is a notification message. A notification is sent to the device, allowing the user to confirm the suggestion.

[1012] (Application example 2)

[1013] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1014] In today's dual-income households, managing food ingredients and maintaining a healthy diet is a significant burden. Furthermore, emotional states such as stress and fatigue often affect the entire lifestyle, making appropriate responses necessary. While current food management systems can manage ingredients and provide nutritional balance recommendations to a certain extent, they lack security advice and alert functions tailored to the user's emotional state. Therefore, there is a need for a comprehensive food management and health management system that takes emotional states into account.

[1015] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for recognizing the type, quantity, and expiration date of food products; means for transmitting the recognized food product data; means for saving the transmitted data and tracking the expiration date or shelf life; means for generating recipes based on the user's preferences, physical condition, season, weather, economic situation, and emotional state; means for transmitting and notifying the user of the generated recipes; means for checking for missing ingredients and automatically ordering them; means for analyzing the user's nutritional balance and generating advice to support a healthy diet; means for transmitting the generated advice; means for collecting user feedback and updating data; means for analyzing the user's emotional state and providing security advice and alerts based on the analysis; and means for recording fluctuations in the emotional state and generating a daily report. This enables food ingredient management and maintaining a healthy diet, as well as appropriate responses according to the user's emotional state, thereby improving the overall quality of life.

[1016] A "food management device" is a device that recognizes the type, quantity, and expiration date of food, and manages, transmits, and stores this data.

[1017] "Emotional state" refers to the user's mental and emotional state, and includes categories such as stress, fatigue, and relaxation.

[1018] "User preferences" refers to the user's preferred choices and tendencies regarding ingredients and dishes.

[1019] "Security Advice" is safety advice or suggestions provided based on the user's emotional state.

[1020] "Alerts" refer to warnings or notifications based on the user's emotional state or the state of the system.

[1021] "Recipe generation" is the process of creating cooking instructions based on the user's preferences, physical condition, season, weather, financial situation, and emotional state.

[1022] "Automatic ordering" refers to a system that checks for shortages of ingredients and automatically orders the necessary ingredients.

[1023] "Nutritional balance" refers to the harmony of nutrients that a user should consume in their diet.

[1024] "Feedback" refers to the opinions and thoughts provided by users to the system, and the system's data is updated based on them.

[1025] "Data transmission" refers to the process of transmitting the recognized food data to the server.

[1026] "Expiration date tracking" is the act of managing the expiration date and storage period of food based on the transmitted data and notifying the user appropriately.

[1027] "Supporting a healthy diet" means analyzing the user's nutritional balance and providing advice and suggestions for maintaining a proper diet.

[1028] "Report generation" is the process of recording fluctuations in emotional state and creating a summary report at the end of the day.

[1029] "Data updating" is the act of keeping the data in the system up to date based on user feedback.

[1030] System Overview

[1031] An embodiment of this invention includes a food management device and a security management system incorporating an emotion engine. The food management device uses sensors and cameras installed in the refrigerator to collect information such as food type, quantity, and expiration date, and sends the data to a server. The server analyzes this data to manage expiration dates and analyze the nutritional balance of food. It also recognizes the user's emotional state and provides appropriate recipes and security advice based on that information.

[1032] Hardware and Software

[1033] The system uses the following hardware and software:

[1034] Hardware:

[1035] Cameras and RFID readers inside refrigerators

[1036] Smartphone or tablet

[1037] Security systems (cameras, alarms)

[1038] software:

[1039] OpenCV: Image processing from camera

[1040] EmotionRecognizer (virtual emotion engine): Recognizes user emotions

[1041] Python: General application logic

[1042] Data processing and calculation methods

[1043] Food Recognition and Data Collection

[1044] Cameras and RFID readers identify the type, quantity and expiration date of food and transmit this data through a communication device inside the refrigerator to a server, which stores it in a database and tracks expiration dates and storage times.

[1045] Recognition of emotional states

[1046] It uses the smartphone's camera and microphone to capture the user's facial expressions and voice, which are then analyzed by an emotion engine. If the emotional state is recognized as stress or fatigue, it triggers a security alarm and issues a safety alert.

[1047] Recipe and advice generation

[1048] The server generates appropriate recipes based on the user's preferences, physical condition, season, weather, financial situation, and emotional state. It also analyzes the user's emotional state and provides security advice as needed. For example, if the user is feeling extremely stressed, it will suggest cooking or resting methods that will help reduce stress.

[1049] Daily report generation

[1050] It records fluctuations in emotional state and automatically generates a report at the end of each day summarizing the user's emotional state and corresponding behavior, including security and health management advice for the next day.

[1051] Prompt Sentence Examples

[1052] The detailed realization of this application also utilizes a generative AI model, with the following prompts as examples:

[1053] Please create a program that meets the following requirements as an example of a security service that applies a system that incorporates an emotion engine into a food management device for dual-income households to a smartphone app.

[1054] Recognizes user emotions (stress, fatigue, etc.) using a camera

[1055] Trigger security alerts based on emotions

[1056] Providing security advice

[1057] Generate daily reports based on stress levels

[1058] The application is written in Python and uses the OpenCV library.

[1059] In this way, the present invention uses a food management device incorporating an emotion engine to achieve efficient food management and support for a healthy diet, and also performs security management according to the user's emotional state, thereby improving the overall quality of life.

[1060] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1061] Step 1:

[1062] Recognizing food types, quantities, and expiration dates

[1063] A camera and RFID reader installed inside the refrigerator recognize the type, quantity, and expiration date of food. The recognized data is acquired as the food name, quantity, and expiration date and sent to the server via the communication device inside the refrigerator. This allows the server to obtain food inventory information.

[1064] Step 2:

[1065] Data storage and expiration date tracking

[1066] The server stores the submitted food data in a database, while simultaneously running algorithms that track expiration dates and shelf life to identify foods approaching their expiry date, which are then listed for possible user notification along with updates to the database.

[1067] Step 3:

[1068] Recognizing the user's emotional state

[1069] The user captures their voice and facial expressions using the smartphone's camera and microphone. The emotion engine analyzes this input data and recognizes their emotional state, such as stress, fatigue, or relaxation. The recognized emotional state is then sent to the server.

[1070] Step 4:

[1071] Generating recipes and security advice

[1072] The server generates appropriate recipes based on the user's emotional state, preferences, physical condition, season, weather, and economic situation. It may also provide security advice based on the user's emotional state. The generated recipes and advice are sent to the user's smartphone.

[1073] Step 5:

[1074] Automated ordering

[1075] The server analyzes the food data in the refrigerator and lists any ingredients that are missing. Based on this information, it generates an automatic order based on the user's purchasing history and preferences, and sends an order request to the online shopping site. Once the order is complete, the user is notified.

[1076] Step 6:

[1077] Analyzing the user's nutritional balance and generating advice

[1078] The server analyzes the user's eating habits and calculates nutritional balance. Based on this, it generates advice and recipes for maintaining a healthy diet and notifies the user, including salad recipes that encourage the intake of appropriate amounts of green and yellow vegetables.

[1079] Step 7:

[1080] Recording emotional state fluctuations and generating daily reports

[1081] The server records the user's emotional fluctuations throughout the day and compiles the data throughout the day. Based on this data, a summary report of the day is automatically generated, including security and health management advice for the next day. The report is then sent to the user's device.

[1082] Step 8:

[1083] Gathering feedback and updating the system

[1084] Users can send feedback on the recipes and advice provided to the server from their device, and the server receives this feedback and updates the system's database to improve the accuracy of future suggestions and advice.

[1085] This allows servers, terminals, and users to work together to achieve efficient food management, healthy living, and security management.

[1086] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1087] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1088] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1089] [Third embodiment]

[1090] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1091] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1092] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1093] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1094] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1095] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1096] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1097] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1098] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1099] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1100] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1101] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1102] The present invention provides a system that uses a food management device to reduce the burden of food ingredient management in dual-income households and supports efficient and healthy eating habits. Hereinafter, an embodiment of the present invention will be described in detail.

[1103] System Overview

[1104] This system uses various sensors and cameras installed inside the refrigerator to collect information such as food type, quantity, and expiration date, and sends this data to a server. The server analyzes this data to manage expiration dates and analyze the nutritional balance of food. It also generates optimal recipes based on the user's physical condition, preferences, season, weather, and economic situation, and notifies the user. It also automatically orders ingredients that are running low, preventing food waste and supporting healthy eating habits.

[1105] Food Recognition and Data Collection

[1106] Cameras and RFID readers installed inside the refrigerator recognize the type, quantity, and expiration date of food. This data is sent to a server via a communication device inside the refrigerator, and the server stores it in a database. For example, data such as "Milk (1 bottle, expiration date: October 10, 2023)" and "Eggs (6 eggs, expiration date: October 12, 2023)" may be acquired from the refrigerator.

[1107] Data storage and expiration date tracking

[1108] The data sent to the server is stored in a database, where expiration dates and storage periods are tracked. The server then notifies the user of food products that are approaching their expiration date. For example, if the expiration date of "milk" is approaching in two days, the server will send a notification to the device saying, "Milk's expiration date will expire in two days."

[1109] Recipe Suggestions

[1110] The server generates optimal recipes based on the user's preferences, physical condition, season, weather, and economic situation. For example, it generates a recipe for an "omelet" using milk and eggs based on the user's preferences and the ingredients currently in the refrigerator, and sends it to the device. The user can then refer to the recipe on the device and easily cook the dish.

[1111] Automatic ordering of necessary ingredients

[1112] The server analyzes the food data in the refrigerator and lists any ingredients that are running low. For example, if it detects that there are fewer than two eggs, the server generates an automatic ordering content based on the user's purchasing history and preferences. This order content is notified to the terminal, and if the user approves, an ordering request is sent to the online shopping site. After the order is completed, the user is notified that "the egg order has been completed."

[1113] Nutritional balance and health management

[1114] The server analyzes the user's eating habits and calculates nutritional balance. To support a healthy diet, it generates necessary health advice and recipes and notifies the device. For example, if the user has not eaten vegetables recently, it will suggest a salad recipe rich in green and yellow vegetables and notify the user. The user sends feedback to the server via their device, which updates the data throughout the system.

[1115] Specific examples

[1116] Next, the processing of the system will be described in detail using a specific example.

[1117] Example 1: Milk expiration date notification

[1118] The server retrieves information about "Milk (best before date: October 10, 2023)" from the database and confirms that the expiration date is two days later.

[1119] The server notifies the terminal that "The milk's expiration date will expire in two days."

[1120] The device displays this notification to the user.

[1121] Example 2: Omelette recipe suggestion

[1122] The server generates an omelet recipe based on the user's preferences and the information on milk and eggs in the refrigerator.

[1123] The server sends this recipe to the terminal.

[1124] The device notifies the user of the "omelette recipe."

[1125] Example 3: Automated egg ordering

[1126] The server determines that there are fewer than two eggs in the refrigerator.

[1127] The server generates an order for "eggs (pack of 10)" and sends an order request to the online shopping site.

[1128] The terminal notifies the user that "your egg order has been completed."

[1129] In this way, the present invention realizes efficient food management and support for a healthy diet using a food management device. This system reduces the burden of food ingredient management for dual-income households and provides a healthy diet that takes nutritional balance into consideration.

[1130] The processing flow will be explained below.

[1131] Step 1:

[1132] The refrigerator detects when the door is opened and closed, activating the internal camera and RFID reader. This recognizes data such as the type of food, quantity, and expiration date. For example, it obtains information such as "Milk (1 bottle, expiration date: October 10, 2023)" and "Eggs (6 eggs, expiration date: October 12, 2023)."

[1133] Step 2:

[1134] The refrigerator sends the food data it recognizes to the server, including the type of food, quantity, and expiration date.

[1135] Step 3:

[1136] The server stores the received food data in a database. The data is structured as information such as the type, quantity, expiration date, and storage period of each food item.

[1137] Step 4:

[1138] The server periodically checks the database and lists foods that are close to their expiration date. For example, it detects that the expiration date of "Milk (expiration date: October 10, 2023)" is just two days away.

[1139] Step 5:

[1140] The server prepares notifications for food items that are nearing their expiration date. For items with an expiration date within two days, the server prepares to notify the user.

[1141] Step 6:

[1142] The server sends the notification content to the terminal. For example, it sends notification data saying "The expiration date of the milk will be reached in two days."

[1143] Step 7:

[1144] The device displays the received notification to the user. For example, a notification pops up on the smartphone saying "The expiration date of the milk will expire in two days."

[1145] Step 8:

[1146] The server generates recipes based on the user's preferences, physical condition, season, weather, and economic situation, and analyzes various collected data to select the optimal recipe.

[1147] Step 9:

[1148] The server sends the generated recipe to the terminal. For example, it sends a "recipe for an omelet using milk and eggs."

[1149] Step 10:

[1150] The device notifies the user of the received recipe. For example, the smartphone displays "An omelet recipe has been suggested."

[1151] Step 11:

[1152] The server analyzes the food data in the refrigerator and lists the ingredients that are running low. For example, it might determine that there are fewer than two eggs.

[1153] Step 12:

[1154] The server generates automatic ordering based on the ingredients that are in short supply. Taking into account the user's purchasing history and preferences, it prepares an order request for "Eggs (pack of 10)."

[1155] Step 13:

[1156] The server sends the order to the terminal and asks the user for approval, for example, sending a confirmation message saying "Do you want to automatically order eggs?"

[1157] Step 14:

[1158] The terminal displays the received order confirmation to the user, who can then select "Accept" or "Cancel" in response to the confirmation message.

[1159] Step 15:

[1160] The user responds with "approval" to the order confirmation. The response data is sent from the terminal to the server.

[1161] Step 16:

[1162] The server receives the user's approval and sends an order request to the online shopping site. For example, it sends a request to order "eggs (pack of 10)."

[1163] Step 17:

[1164] The server sends a notification of order completion to the terminal, for example, data such as "Your egg order has been completed."

[1165] Step 18:

[1166] The device will notify the user that their eggs have been ordered. The user can then check the status of their order.

[1167] Step 19:

[1168] The server collects data on the user's eating habits and analyzes their nutritional balance. For example, it obtains information such as, "I've been eating less vegetables recently."

[1169] Step 20:

[1170] The server generates health advice and recipes based on the analysis results, such as a recipe for a salad rich in green and yellow vegetables.

[1171] Step 21:

[1172] The server sends the generated advice or recipe to the terminal, for example, sending a "salad recipe."

[1173] Step 22:

[1174] The device will notify the user of the advice or recipes it has received, and the user can view the notification and incorporate the advice or recipes.

[1175] Step 23:

[1176] If the user actually uses the suggested advice or recipe, feedback is sent via the device.

[1177] Step 24:

[1178] The server receives the feedback submitted by the user and updates the database, which improves the accuracy of the system and allows for a more personalized service.

[1179] Example 1

[1180] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1181] The present invention relates to a system that reduces the burden of food ingredient management for dual-income households and supports efficient and healthy eating habits. Conventional food management systems often require manual management of food expiration dates and analysis of nutritional balance, which can be time-consuming and labor-intensive for dual-income households. Furthermore, automatic food ordering and recipe suggestions may not be performed appropriately, potentially resulting in food waste and nutritional imbalances. A solution to these issues is needed.

[1182] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1183] In this invention, the server includes means for recognizing the type, quantity, and expiration date of food, means for transmitting the recognized food data, means for storing the transmitted data and tracking the expiration date or storage period, means for generating recipes based on the user's preferences, physical condition, season, weather conditions, and economic situation, means for transmitting and notifying the user of the generated recipes, means for checking for missing ingredients and automatically ordering, means for analyzing the user's nutritional balance and generating suggestions to support a healthy diet, means for transmitting the generated suggestions, means including a communication device for collecting food data, and means for collecting user feedback and updating the data. This reduces the burden of food management for dual-income households and makes it possible to support an efficient and healthy diet.

[1184] A "food management device" is a device that recognizes and manages information such as food type, quantity, and expiration date.

[1185] "Means for recognizing the type, quantity, and expiration date of food" refers to means for identifying the type, quantity, and expiration date of food using a camera, RFID reader, etc.

[1186] The "means for transmitting recognized food data" is a communication device for transmitting information about the recognized food to a server or database.

[1187] "Means for storing submitted data and tracking expiration dates or shelf life" refers to means for storing submitted food data in a database and managing and tracking expiration dates or shelf life.

[1188] "Means for generating recipes based on the user's preferences, physical condition, season, weather conditions, and economic situation" refers to a program or system that generates optimal recipes based on the user's personal information and external conditions.

[1189] The "means for transmitting and notifying the generated recipe" is a communication and notification system for transmitting and notifying the generated recipe to the user's terminal.

[1190] The "means of checking for shortages of ingredients and automatically ordering" is a system that analyzes food data in the refrigerator, identifies shortages of food, and automatically orders them.

[1191] The "means for analyzing a user's nutritional balance and generating suggestions to support a healthy diet" is a system that analyzes a user's eating habit data and generates suggestions to support a healthy diet from the perspective of nutritional balance.

[1192] The "means for transmitting the generated suggestions" is a communication system for transmitting the generated health suggestions to the user's terminal.

[1193] The "means including a communication device for collecting food data" is a system including a communication device for collecting food information and transmitting it to a server.

[1194] "Means for collecting user feedback and updating data" refers to means that has the function of receiving feedback from users and updating data throughout the system based on that feedback.

[1195] MODE FOR CARRYING OUT THE INVENTION

[1196] The present invention provides a system that uses a food management device to reduce the burden of food ingredient management in dual-income households and supports efficient and healthy eating habits. Hereinafter, an embodiment of the present invention will be described in detail.

[1197] System Overview

[1198] This system uses various sensors and cameras installed inside the refrigerator to collect information such as food type, quantity, and expiration date, and sends this data to a server. The server analyzes this data to manage expiration dates and analyze the nutritional balance of food. It also generates optimal recipes based on the user's physical condition, preferences, season, weather conditions, and economic situation, and notifies the user. It also automatically orders ingredients that are running low, preventing food waste and supporting healthy eating habits.

[1199] Food Recognition and Data Collection

[1200] Cameras and RFID readers installed inside the refrigerator recognize the type, quantity, and expiration date of food. This data is sent to a server via a communication device inside the refrigerator, and the server stores it in a database. For example, data such as "Milk (1 bottle, expiration date: October 10, 2023)" and "Eggs (6 eggs, expiration date: October 12, 2023)" may be acquired from the refrigerator.

[1201] Data storage and expiration date tracking

[1202] The data sent to the server is stored in a database, where expiration dates and storage periods are tracked. The server then notifies the user of food products that are approaching their expiration date. For example, if the expiration date of "milk" is approaching in two days, the server will send a notification to the device saying, "Milk's expiration date will expire in two days."

[1203] Recipe Suggestions

[1204] The server generates optimal recipes based on the user's preferences, physical condition, season, weather conditions, and economic situation. For example, it generates a recipe for an "omelet" using milk and eggs based on the user's preferences and the ingredients currently in the refrigerator, and sends it to the device. The user can then refer to the recipe on the device and easily cook the dish.

[1205] Automatic ordering of necessary ingredients

[1206] The server analyzes the food data in the refrigerator and lists any ingredients that are running low. For example, if it detects that there are fewer than two eggs, the server generates an automatic ordering content based on the user's purchasing history and preferences. This order content is notified to the terminal, and if the user approves, an ordering request is sent to the e-commerce site. After the order is completed, the server sends a notification to the user that "the egg order has been completed."

[1207] Nutritional balance and health management

[1208] The server analyzes the user's eating habits and calculates nutritional balance. To support a healthy diet, it generates necessary health advice and recipes and notifies the device. For example, if the user has not eaten vegetables recently, it will suggest a salad recipe rich in green and yellow vegetables and notify the user. The user sends feedback to the server via their device, which updates the data throughout the system.

[1209] Specific examples

[1210] Next, the processing of the system will be described in detail using a specific example.

[1211] Example 1: Milk expiration date notification

[1212] The server retrieves information about "Milk (best before date: October 10, 2023)" from the database and confirms that the expiration date is two days later.

[1213] The server notifies the terminal that "The milk's expiration date will expire in two days."

[1214] The device displays this notification to the user.

[1215] Example 2: Omelette recipe suggestion

[1216] The server generates an omelet recipe based on the user's preferences and the information on milk and eggs in the refrigerator.

[1217] The server sends this recipe to the terminal.

[1218] The device notifies the user of the "omelette recipe."

[1219] Example 3: Automated egg ordering

[1220] The server determines that there are fewer than two eggs in the refrigerator.

[1221] The server generates an order for "Eggs (pack of 10)" and sends the order request to the e-commerce site.

[1222] The terminal notifies the user that "your egg order has been completed."

[1223] Prompt Sentence Examples

[1224] Below is an example of a prompt sentence to input to the generative AI model.

[1225] "Please analyze the food in my refrigerator and its expiration date and suggest the best recipes."

[1226] "Analyze the ingredients that are in short supply and use an automated ordering system to generate orders for the ingredients that are needed."

[1227] "Based on the user's eating habits data, create nutritional advice to support healthy eating habits."

[1228] In this way, the present invention realizes efficient food management and support for a healthy diet using a food management device. This system reduces the burden of food ingredient management for dual-income households and provides a healthy diet that takes nutritional balance into consideration.

[1229] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1230] Step 1: Collect food data

[1231] Input: Food in the fridge

[1232] Output: Food type, quantity, expiration date data

[1233] What happens:

[1234] A camera installed inside the refrigerator takes pictures of the food, and an RFID reader reads the data from the RFID tags.

[1235] The communication device inside the refrigerator acquires the image data from the camera and the tag data from the RFID reader.

[1236] The data obtained includes the type of food (e.g., "milk" or "egg"), quantity (e.g., "1 bottle" or "6 pieces"), and expiration date (e.g., "October 10, 2023" or "October 12, 2023").

[1237] The communication device transmits this data to the server.

[1238] Step 2: Data storage and analysis

[1239] Input: Food type, quantity, expiration date data

[1240] Output: Saved food data, analysis results

[1241] What happens:

[1242] The server receives the food data transmitted from the refrigerator.

[1243] The received data includes the type of food, quantity, expiration date, etc.

[1244] The server stores this data in a database (e.g., "Milk (1 bottle, expiration date: October 10, 2023)" and "Eggs (6 eggs, expiration date: October 12, 2023)").

[1245] The server analyzes the stored data to identify current food inventory and foods that are close to their expiration date.

[1246] Step 3: Best before date notification

[1247] Input: Food data in the database

[1248] Output: Best before date notification message

[1249] What happens:

[1250] The server periodically scans the food data in the database to identify foods that are approaching their expiration date.

[1251] For example, find "milk" whose expiration date is just two days away.

[1252] The server generates a notification message saying "Milk expires in 2 days."

[1253] Send this message to the user's terminal.

[1254] The device will display this notification to the user.

[1255] Step 4: Recipe suggestions

[1256] Input: User preferences, physical condition, season, weather conditions, economic situation, food data in refrigerator

[1257] Output: Recipe suggestion message

[1258] What happens:

[1259] The server collects data on the user's preferences, physical condition, season, weather conditions, and economic situation.

[1260] It also retrieves food data from the refrigerator and checks the current ingredients.

[1261] For example, suppose you have milk and eggs in your refrigerator.

[1262] The server uses this information to input prompts into the generative AI model (e.g., "There is currently milk and eggs in the refrigerator. Please suggest some recipes using these.").

[1263] The generative AI model generates a recipe for "omelette" based on prompts.

[1264] The server sends this generated recipe to the user's terminal, and the user is notified of the "omelette recipe."

[1265] Step 5: Automatically order the ingredients you need

[1266] Input: Food data in the refrigerator

[1267] Output: Order completion notification message

[1268] What happens:

[1269] The server analyzes the database and detects that there is a shortage of food in the refrigerator.

[1270] For example, identify a decrease to "less than two eggs."

[1271] The server generates an order (e.g., "pack of 10 eggs") based on the user's past purchasing history and preferences.

[1272] The server sends the order details to the electronic commerce site as an order request.

[1273] After the order is completed, the server sends a notification to the user's terminal saying "Egg order completed."

[1274] Step 6: Nutritional Balance and Health Management

[1275] Input: User's dietary habits data

[1276] Output: Health suggestion message

[1277] What happens:

[1278] The server periodically analyzes the user's eating habits data.

[1279] Based on the analysis results, nutritional balance is calculated and any missing nutrients are identified.

[1280] For example, it detects when a user has not eaten vegetables recently.

[1281] The server generates recipes for salads rich in green vegetables to support a healthy diet.

[1282] This generated recipe is sent to the user's device and notified to the user.

[1283] Users send feedback to the server through their devices, and data is updated throughout the system.

[1284] (Application example 1)

[1285] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1286] In busy households, such as dual-income households, it is difficult to manage food and ensure a balanced diet. Even when shopping at brick-and-mortar stores, it can be time-consuming to know what to buy and whether there are any foods nearing their expiration date. Furthermore, there is a need for a method to efficiently procure ingredients while maintaining a healthy diet. A system that solves these issues and supports food management and healthy eating habits at home is needed.

[1287] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1288] In this invention, the server includes means for recognizing the type, quantity, and expiration date of food, means for transmitting the recognized food data, means for storing the transmitted data and tracking the expiration date or storage period, means for generating recipes based on the user's preferences, physical condition, season, weather, and economic situation, means for transmitting and notifying the user of the generated recipes, means for checking for missing ingredients and automatically ordering, means for analyzing the user's nutritional balance and generating advice to support a healthy diet, means for transmitting the generated advice, means for collecting user feedback and updating data, means for displaying ingredient information in a physical store via smart glasses and presenting a list of ingredients that need to be replenished, and means for presenting recipes and special sale information generated based on the user's profile and external environment information in real time, thereby enabling more efficient food management and supporting a healthy diet.

[1289] A "food management device" is a device for recognizing and managing information such as the type, quantity, and expiration date of food.

[1290] A "sensor" is a device that detects a physical quantity or state and outputs it as an electrical signal.

[1291] A "camera" is a device that can capture video or still images and store and transmit them as digital data.

[1292] A "communication device" is a device for sending and receiving data to and from other devices and servers.

[1293] A "server" is a computer system that stores, analyzes, and processes data and provides information in response to requests from other devices.

[1294] "User preferences" refers to information that indicates the user's taste and cooking preferences.

[1295] "Physical condition" is information that indicates the user's health condition and physical condition.

[1296] "Season" refers to the time of year brought about by changes in the climate.

[1297] "Weather" is information indicating weather conditions in a particular area and time.

[1298] "Financial status" is information that indicates the user's financial status, such as income, expenses, and budget.

[1299] A "recipe" is a detailed list of instructions and ingredients needed to make a particular dish.

[1300] "Notification" means sending messages or alerts to inform users.

[1301] "Auto ordering" is the process of automatically ordering necessary items and ingredients.

[1302] "Nutritional balance" refers to a state in which the user's diet properly incorporates the necessary nutrients.

[1303] "Advice" is information that provides advice or recommendations to the user.

[1304] "Feedback" refers to opinions and evaluations provided by users to the system.

[1305] "Smart glasses" are wearable devices that have the ability to display visual information.

[1306] A "brick and mortar store" is a physical commercial establishment where users can visit and purchase products.

[1307] "Food information" refers to data such as food type, quantity, expiration date, and storage conditions.

[1308] A "profile" is a data set that compiles personal information and characteristics about a user.

[1309] "External environment information" refers to information that is not directly managed by the user, such as weather, season, and surrounding conditions.

[1310] "Special sale information" is information about products being sold at a price lower than the regular price.

[1311] The present invention is a system for supporting home food management and healthy eating habits. The purpose of the present invention is to streamline food management in physical stores through smart glasses and improve the user's shopping experience. Specific embodiments for implementing the present invention are described in detail below.

[1312] System Overview

[1313] This system uses various sensors and cameras installed inside the refrigerator to collect information such as food type, quantity, and expiration date, and sends this data to a server. The server analyzes this data to manage expiration dates and analyze the nutritional balance of food. In addition, when a user wearing smart glasses purchases ingredients in a physical store, the system displays real-time information on necessary ingredients, special sales, recipe suggestions, and more.

[1314] Hardware and Software

[1315] Hardware:

[1316] Cameras and sensors inside the refrigerator: Used to collect food data.

[1317] Smart glasses: Using "Google Glass" and "Vuzix Blade" as examples.

[1318] Server: We use AWS and Azure for data analysis and notifications.

[1319] software:

[1320] Data collection software: Collects data from cameras and sensors inside the refrigerator and sends it to a server.

[1321] Data analysis software: Food data, user information, and external environmental information are analyzed on the server.

[1322] User Interface: An application for display on smart glasses.

[1323] Specific processing

[1324] 1. Food Recognition and Data Collection:

[1325] Cameras and RFID readers installed inside the refrigerator identify the type, quantity and expiration date of food and send this data to a server, which stores it in a database and tracks expiration dates.

[1326] 2. Information display through smart glasses:

[1327] When users shop in a physical store, the smart glasses display real-time information about the ingredients in their refrigerator. For example, if milk or eggs are low in stock, the glasses notify the user. They also display sales and coupon information to support their purchasing behavior.

[1328] 3. Recipe and nutritional suggestions:

[1329] The server uses a generative AI model to generate appropriate recipes based on the user's preferences, physical condition, season, weather, and economic situation. The generated recipes are then sent to the user via smart glasses. The server also analyzes nutritional balance and sends advice to support healthy eating.

[1330] Specific examples

[1331] A specific example of the system's operation is shown below.

[1332] Example 1: Shopping assistance in physical stores

[1333] The user wears smart glasses and goes shopping in a physical store. The smart glasses' display shows real-time information about the food inventory in the refrigerator. A notification such as "You're low on milk and need to buy more" is displayed, along with information about special sales and coupons available in the store.

[1334] Example 2: Recipe suggestions

[1335] When a user is using the smart glasses at home, the server generates an "omelette" recipe based on the user's current health condition, preferences, and information about ingredients in the refrigerator, and notifies the smart glasses. The following prompt sentence is used for the generating AI model:

[1336] Example prompt for a generative AI model:

[1337] "The user has a bottle of milk and two eggs in the refrigerator. It's fall, and the user prefers a balanced diet. Based on this, please suggest some suitable recipes."

[1338] In this way, the present invention can utilize smart glasses to comprehensively support food management in brick-and-mortar stores and improve users' quality of life.

[1339] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1340] Step 1:

[1341] Food data collection

[1342] The server receives input from cameras and RFID readers installed inside the refrigerator. This input includes the type of food, quantity, and expiration date. The server analyzes this data and stores it in a database for each food item. For example, information such as "Milk (Quantity: 1 bottle, Best before date: October 10, 2023)" may be collected.

[1343] Step 2:

[1344] Best before date tracking

[1345] The server periodically analyzes the expiration dates of the food items stored in the database and lists the food items that are approaching their expiration date. For example, it lists the food items whose expiration date is two days away and prepares a notification such as "The expiration date will expire in two days." This notification is sent to the user's device.

[1346] Step 3:

[1347] Detecting missing ingredients

[1348] The server analyzes the food data in the database and detects ingredients that are low in stock. For example, if it determines that there are fewer than two eggs, it adds them to a list of ingredients that are in short supply. The server notifies the user of the list of ingredients that are in short supply, and clearly indicates which ingredients need to be replenished.

[1349] Step 4:

[1350] Preparing for automatic ordering

[1351] The server generates an automatic order for missing ingredients based on the user's purchasing history and preferences. For example, it prepares an order for "eggs (pack of 10)" and prepares to send a request to an online shopping site. This order preparation data is notified to the user and they are asked to approve it.

[1352] Step 5:

[1353] Recipe Generation

[1354] Based on the input data of the user's preferences, physical condition, season, weather, and economic situation, the server uses a generative AI model to generate the optimal recipe. The generative AI model is input with the following prompt:

[1355] "The user has a bottle of milk and two eggs in the refrigerator. It's fall, and the user prefers a balanced diet. Based on this, please suggest some suitable recipes."

[1356] The generated recipe, for example an "omelette" recipe, is completed and sent to the user terminal.

[1357] Step 6:

[1358] Information display on smart glasses

[1359] When a user wears the smart glasses and goes shopping in a physical store, the device displays real-time information about ingredients in the refrigerator and a list of ingredients that are running low. For example, a notification will appear on the smart glasses saying, "You're low on milk and need to buy more." In addition, the device will display information about special sales and coupons in the store, allowing users to shop efficiently.

[1360] Step 7:

[1361] Nutritional balance analysis and advice delivery

[1362] The server analyzes the user's eating habits, calculates nutritional balance, and generates advice to support a healthy diet. For example, it generates advice such as "You haven't been eating enough vegetables recently," and notifies the user's device. This advice collects user feedback and serves as the basis for updating data throughout the system.

[1363] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1364] The present invention provides a system that uses a food management device combined with an emotion engine to reduce the burden of food ingredient management for dual-income households and support efficient and healthy eating habits. The following describes in detail the embodiments of the present invention.

[1365] System Overview

[1366] This system uses sensors and cameras installed inside the refrigerator to collect information such as food type, quantity, and expiration date, and sends this data to a server. The server analyzes this data to manage expiration dates and analyze the nutritional balance of food. It also generates optimal recipes based on the user's physical condition, preferences, season, weather, economic situation, and emotions, and notifies the user. It also automatically orders ingredients that are running low, preventing food waste and supporting healthy eating habits.

[1367] Food Recognition and Data Collection

[1368] Cameras and RFID readers installed inside the refrigerator recognize the type, quantity, and expiration date of food. This data is sent to a server via a communication device inside the refrigerator, and the server stores it in a database. For example, data such as "Milk (1 bottle, expiration date: October 10, 2023)" and "Eggs (6 eggs, expiration date: October 12, 2023)" may be acquired from the refrigerator.

[1369] Data storage and expiration date tracking

[1370] The data sent to the server is stored in a database, where expiration dates and storage periods are tracked. The server then notifies the user of food products that are approaching their expiration date. For example, if the expiration date of "milk" is approaching in two days, the server will send a notification to the device saying, "Milk's expiration date will expire in two days."

[1371] Recipe Suggestions

[1372] The server generates optimal recipes based on the user's preferences, physical condition, season, weather, economic situation, and emotional data obtained by the emotion engine. For example, it generates a recipe for an "omelette" using "milk" and "eggs" based on the user's preferences and the ingredients currently in the refrigerator, and sends it to the device. The user can then refer to the recipe on their device and easily cook the dish.

[1373] Automatic ordering of necessary ingredients

[1374] The server analyzes the food data in the refrigerator and lists any ingredients that are running low. For example, if it detects that there are fewer than two eggs, the server generates an automatic ordering content based on the user's purchasing history and preferences. This order content is notified to the terminal, and if the user approves, an ordering request is sent to the online shopping site. After the order is completed, the user is notified that "the egg order has been completed."

[1375] Nutritional balance and health management

[1376] The server analyzes the user's eating habits and calculates nutritional balance. To support a healthy diet, it generates necessary health advice and recipes and notifies the device. For example, if the user has not eaten vegetables recently, it will suggest a salad recipe rich in green and yellow vegetables and notify the user. The user sends feedback to the server via their device, which updates the data throughout the system.

[1377] Further suggestions from the emotion engine

[1378] The emotion engine recognizes the user's emotions and further personalizes recipes and health advice based on this. For example, if the user is feeling stressed, it will suggest relaxing recipes that will help reduce stress.

[1379] Specific examples

[1380] Example 1: Emotion-based recipe suggestions

[1381] The server collects the user's emotional state through an emotion engine. For example, if the user is "feeling stressed," emotion data is acquired.

[1382] The server generates relaxation recipes, including chamomile tea, which is effective in reducing stress.

[1383] The server sends this relaxation recipe to the terminal.

[1384] The device will display a notification to the user saying, "A suggestion has been made for making chamomile tea."

[1385] Example 2: Emotion-based health advice

[1386] The server obtains emotional data from the emotion engine, such as "feeling tired" by the user.

[1387] The server generates advice including "vitamin-rich dishes" that are effective in relieving fatigue.

[1388] The server sends this advice to the terminal.

[1389] The device will display a notification to the user saying, "A dish that is effective in relieving fatigue has been suggested."

[1390] In this way, the present invention realizes efficient food management and support for healthy eating habits using a food management device with an emotion engine built in. This system reduces the burden of food ingredient management for dual-income households and provides a healthy diet that takes nutritional balance into consideration.

[1391] The processing flow will be explained below.

[1392] Step 1:

[1393] The refrigerator detects when the door is opened and closed, activating the internal camera and RFID reader. This recognizes data such as the type of food, quantity, and expiration date. For example, it obtains information such as "Milk (1 bottle, expiration date: October 10, 2023)" and "Eggs (6 eggs, expiration date: October 12, 2023)."

[1394] Step 2:

[1395] The refrigerator sends the food data it recognizes to the server, including the type of food, quantity, and expiration date.

[1396] Step 3:

[1397] The server stores the received food data in a database. The data is structured as information such as the type, quantity, expiration date, and storage period of each food item.

[1398] Step 4:

[1399] The server periodically checks the database and lists foods that are close to their expiration date. For example, it detects that the expiration date of "Milk (expiration date: October 10, 2023)" is just two days away.

[1400] Step 5:

[1401] The server prepares notifications for food items that are nearing their expiration date. For items with an expiration date within two days, the server prepares to notify the user.

[1402] Step 6:

[1403] The server sends the notification content to the terminal. For example, it sends notification data saying "The expiration date of the milk will be reached in two days."

[1404] Step 7:

[1405] The device displays the received notification to the user. For example, a notification pops up on the smartphone saying "The expiration date of the milk will expire in two days."

[1406] Step 8:

[1407] The server generates optimal recipes based on the user's preferences, physical condition, season, weather, economic situation, and emotional data obtained by the emotion engine. For example, it generates a recipe for "omelette with milk and eggs" based on the user's preferences and the ingredients currently in the refrigerator.

[1408] Step 9:

[1409] The server sends the generated recipe to the terminal. For example, it sends a "recipe for an omelet using milk and eggs."

[1410] Step 10:

[1411] The device notifies the user of the received recipe, for example, "An omelet recipe has been suggested."

[1412] Step 11:

[1413] The server analyzes the food data in the refrigerator and lists the ingredients that are running low. For example, it might determine that there are fewer than two eggs.

[1414] Step 12:

[1415] The server generates automatic ordering based on the ingredients that are in short supply. Taking into account the user's purchasing history and preferences, it prepares an order request for "Eggs (pack of 10)."

[1416] Step 13:

[1417] The server sends the order to the terminal and asks the user for approval, for example, sending a confirmation message saying "Do you want to automatically order eggs?"

[1418] Step 14:

[1419] The terminal displays the received order confirmation to the user, who can then select "Accept" or "Cancel" in response to the confirmation message.

[1420] Step 15:

[1421] The user responds with "approval" to the order confirmation. The response data is sent from the terminal to the server.

[1422] Step 16:

[1423] The server receives the user's approval and sends an order request to the online shopping site. For example, it sends a request to order "eggs (pack of 10)."

[1424] Step 17:

[1425] The server sends a notification of order completion to the terminal, for example, data such as "Your egg order has been completed."

[1426] Step 18:

[1427] The device will notify the user that their eggs have been ordered. The user can then check the status of their order.

[1428] Step 19:

[1429] The server collects data on the user's eating habits and analyzes their nutritional balance. For example, it obtains information such as, "I've been eating less vegetables recently."

[1430] Step 20:

[1431] The server generates health advice and recipes based on the analysis results, such as a recipe for a salad rich in green and yellow vegetables.

[1432] Step 21:

[1433] The server sends the generated advice or recipe to the terminal, for example, sending a "salad recipe."

[1434] Step 22:

[1435] The device will notify the user of the advice or recipes it has received, and the user can view the notification and incorporate the advice or recipes.

[1436] Step 23:

[1437] The server collects the user's emotional state through the emotion engine. For example, if the user says "I'm feeling stressed," the emotion data is acquired.

[1438] Step 24:

[1439] The server analyzes the emotional data and generates recipes and advice based on the user's emotional state. For example, it generates a relaxation recipe that includes chamomile tea, which is effective in reducing stress.

[1440] Step 25:

[1441] The server sends the generated relaxation recipe to the device, for example, "How to make chamomile tea."

[1442] Step 26:

[1443] The device notifies the user of the relaxation recipe it has received, for example, by displaying "A chamomile tea recipe has been suggested."

[1444] Step 27:

[1445] The server receives the feedback submitted by the user and updates the database, which improves the accuracy of the system and allows for a more personalized service.

[1446] Example 2

[1447] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1448] In dual-income households, it is difficult to maintain an efficient and healthy diet while reducing the burden of food management. There is also a need for optimal dietary recommendations based on the user's emotions and physical condition, but existing systems have difficulty comprehensively considering these factors. Therefore, there is a need for a system that automates the management of food types, quantities, and expiration dates, and also provides personalized recipes and advice based on the user's individual condition.

[1449] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1450] A means of recognizing the type, quantity, and expiration date of food;

[1451] means for transmitting the recognized food data;

[1452] A means for storing the transmitted data and tracking expiration dates or shelf life;

[1453] A means for generating recipes based on the user's preferences, physical condition, season, weather, and economic situation;

[1454] A means for transmitting and notifying the generated recipe;

[1455] A way to check for shortages of ingredients and automatically order them,

[1456] A means to analyze the user's nutritional balance and generate advice to support a healthy diet;

[1457] means for transmitting the generated advice;

[1458] A means of collecting user feedback and updating the data;

[1459] A means to recognize user emotions using an emotion engine and personalize recipes and advice based on this;

[1460] A means to use a generative AI model to generate prompts based on data on food in the refrigerator and user sentiment data, and to suggest recipes and advice based on those prompts;

[1461] A means to transmit the entire food recognition and management process to a server through a secure communication protocol

[1462] This will reduce the burden of food management for dual-income households and enable efficient and healthy eating habits. It will also enable personalized suggestions that take into account the user's emotions and physical condition.

[1463] A "food management device" is a device that manages information such as the type, quantity, and expiration date of food, and provides this information to users.

[1464] The "recognition means" refers to a method or device for detecting and processing information such as the type, quantity, and expiration date of food.

[1465] A "means for transmitting" is a method or device for transmitting the recognized information to another device or system.

[1466] The "storing means" refers to a method or device for storing the transmitted data in a recording medium such as a database.

[1467] A "tracking means" is a method or device for monitoring stored data and updating information or taking action based on certain conditions.

[1468] A "recipe generator" is a method or device for suggesting cooking steps and ingredients based on a user's specific requirements.

[1469] "Notification means" refers to a method or device for notifying the user of the generated information.

[1470] The "automatic ordering means" is a method or device for detecting shortages of ingredients and automatically carrying out the ordering procedure.

[1471] The "means for generating advice" is a method or device for generating suggestions to support a healthy diet based on the user's nutritional balance and health status.

[1472] A "means for collecting feedback" is a method or device for gathering user responses and opinions and updating the system.

[1473] An "emotion engine" is a technology that recognizes a user's emotional state and generates personalized suggestions based on that data.

[1474] A "generative AI model" is an artificial intelligence model that automatically generates text or suggestions based on input data.

[1475] A "prompt" is text containing a question or command that is input to a generative AI model.

[1476] A "secure communication protocol" is a communication protocol that protects data transmission and prevents unauthorized access by third parties.

[1477] MODE FOR CARRYING OUT THE INVENTION

[1478] The present invention provides a system that uses a food management device combined with an emotion engine to reduce the burden of food ingredient management for dual-income households and support efficient and healthy eating habits. Specific embodiments for implementing the present invention are described below.

[1479] System Overview

[1480] This system uses sensors and cameras installed inside the refrigerator to collect information such as food type, quantity, and expiration date, and sends this data to a server. The server analyzes this data to manage expiration dates and analyze the nutritional balance of food. It also generates optimal recipes based on the user's physical condition, preferences, season, weather, economic situation, and emotions, and notifies the user. It also automatically orders ingredients that are running low, preventing food waste and supporting healthy eating habits.

[1481] Food Recognition and Data Collection

[1482] Cameras and RFID readers installed inside the refrigerator recognize the type, quantity, and expiration date of food. This data is sent to a server via a communication device inside the refrigerator, and the server stores it in a database. For example, data such as "Milk (1 bottle, expiration date: October 10, 2023)" and "Eggs (6 eggs, expiration date: October 12, 2023)" may be acquired from the refrigerator. In this case, the camera captures high-resolution images inside the refrigerator, and the RFID reader reads the RFID tags on the food items and obtains the information contained therein.

[1483] Data storage and expiration date tracking

[1484] The data sent to the server is stored in a database, where expiration dates and storage periods are tracked. The server then notifies the user of food products that are approaching their expiration date. For example, if the expiration date of "milk" is approaching in two days, the server will send a notification to the device saying, "Milk's expiration date will expire in two days."

[1485] Recipe Suggestions

[1486] The server generates optimal recipes based on the user's preferences, physical condition, season, weather, economic situation, and emotional data obtained by the emotion engine. Using a generative AI model, the server generates prompts based on the food data in the refrigerator and the user's emotional data, and suggests recipes and advice based on these. For example, it generates a recipe for an "omelette" using the "milk" and "eggs" in the refrigerator and sends it to the device. The user can then refer to the recipe on the device and cook the food. Examples of prompts are as follows:

[1487] "There is one bottle of milk and six eggs in the refrigerator. The user is feeling stressed. Based on this, please suggest the best recipe for the user."

[1488] Automatic ordering of necessary ingredients

[1489] The server analyzes the food data in the refrigerator and lists the ingredients that are running low. For example, if there are fewer than two eggs, the server generates an automatic ordering content based on the user's purchasing history and preferences. This order content is notified to the terminal, and if the user approves, an ordering request is sent to the online shopping site. After the order is completed, the user is notified that "the egg order has been completed."

[1490] Nutritional balance and health management

[1491] The server analyzes the user's eating habits and calculates nutritional balance. To support a healthy diet, it generates necessary health advice and recipes and notifies the device. For example, if the user has not eaten vegetables recently, it will suggest a salad recipe rich in green and yellow vegetables and notify the user. The user sends feedback to the server via their device, which updates the data throughout the system.

[1492] Further suggestions from the emotion engine

[1493] The emotion engine recognizes the user's emotions and further personalizes recipes and health advice based on this. For example, if the user is feeling stressed, it will suggest relaxing recipes that will help reduce stress.

[1494] Specific examples

[1495] Example 1: Emotion-based recipe suggestions

[1496] The server collects the user's emotional state through an emotion engine. For example, if the user is "feeling stressed," emotion data is acquired.

[1497] The server generates relaxation recipes, including chamomile tea, which is effective in reducing stress.

[1498] The server sends this relaxation recipe to the terminal.

[1499] The device will display a notification to the user saying, "A suggestion has been made for making chamomile tea."

[1500] Example 2: Emotion-based health advice

[1501] The server obtains emotional data from the emotion engine, such as "feeling tired" by the user.

[1502] The server generates advice including "vitamin-rich dishes" that are effective in relieving fatigue.

[1503] The server sends this advice to the terminal.

[1504] The device will display a notification to the user saying, "A dish that is effective in relieving fatigue has been suggested."

[1505] In this way, users can utilize a system that supports efficient food management and healthy eating habits based on food data and emotional data in their refrigerator.

[1506] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1507] Step 1:

[1508] A camera installed in the refrigerator takes pictures of the food items, and an RFID reader retrieves the RFID tag information for each food item. The input is the food items in the refrigerator, and the output is the food images and RFID tag data. The camera captures high-resolution images, and the RFID reader reads the food information (e.g., type, quantity, expiration date).

[1509] Step 2:

[1510] A communication device in the refrigerator transmits the acquired data (food images and RFID tag data) to a server. The input is the captured image and the read RFID data, and the output is the data to be sent to the server. The communication device transmits the data using a secure protocol (e.g., HTTPS).

[1511] Step 3:

[1512] The server analyzes the received data and stores the food type, quantity, and expiration date in a database. The input is the transmitted image and RFID data, and the output is the food data stored in the database. The server uses an image recognition algorithm (e.g., convolutional neural network) to identify the food type, match it with the RFID tag information, and update the database.

[1513] Step 4:

[1514] The server periodically scans the stored food data to detect foods approaching their expiration date. The input is the stored database information, and the output is a list of foods approaching their expiration date. The server performs batch processing and filters foods within a certain number of days of their expiration date using JSON or SQL queries.

[1515] Step 5:

[1516] The server generates notification messages about food products that are about to expire and sends them to the device. The input is the filtered expiration warning data, and the output is the notification message. The server generates a specific message such as "Milk will expire in 2 days" and sends it to the device via the push notification API.

[1517] Step 6:

[1518] The server collects data on the user's preferences, physical condition, season, weather, economic situation, and emotions. The input is the user's profile data and data from the emotion engine, and the output is the integrated information of these data. Information is sent periodically from the device, and the emotion engine obtains emotion data in real time.

[1519] Step 7:

[1520] The server compares the food data in the refrigerator with the user's individual data and generates the optimal recipe. The input is the food data, the user's profile information, and emotion data, and the output is a recipe generated by a generative AI model. Using a generative AI model (e.g., GPT-3), input a prompt sentence in the following format:

[1521] "There is one bottle of milk and six eggs in the refrigerator. The user is feeling stressed. Based on this, please suggest the best recipe for the user."

[1522] Step 8:

[1523] The server sends the generated recipe to the terminal and asks for feedback in a prepared form. The input is the generated recipe data, and the output is recipe information in HTML format and a feedback form. The recipe details are sent to the terminal, and the user can check the details on the terminal.

[1524] Step 9:

[1525] The server analyzes the food data in the refrigerator and lists the missing ingredients. The input is the stored food data, and the output is the list of missing ingredients.

[1526] Step 10:

[1527] The server generates automatic ordering information based on the user's purchasing history and preferences and notifies the user. The input is a list of missing ingredients and purchasing history data, and the output is the automatic ordering information and notification message. Notifications are sent to the user using a push notification API.

[1528] Step 11:

[1529] If the user approves, the server sends an order request to the online shopping site. The input is the approved order details, and the output is an order request to the online shopping site. An API is used to add the item to the shopping site's cart and start the order process.

[1530] Step 12:

[1531] The server notifies the terminal that the order has been placed. The input is the confirmation of the order being placed from the online shopping site, and the output is a notification message to the user. The order completion message is sent using the push notification API.

[1532] Step 13:

[1533] The server analyzes the user's dietary habits and calculates the nutritional balance. The input is the user's dietary habits and the output is the calculated nutritional balance. The server performs the nutrient analysis using SQL queries or dedicated algorithms.

[1534] Step 14:

[1535] The server identifies nutrient deficiencies and generates health advice. The input is the calculated nutritional balance, and the output is health advice. For example, it generates a recipe for a salad rich in green and yellow vegetables.

[1536] Step 15:

[1537] The server sends the health advice to the terminal and notifies the user. The input is the generated health advice, and the output is the notification message. The notification is sent to the terminal and can be confirmed by the user.

[1538] Step 16:

[1539] The server obtains the user's emotional data in real time through the emotion engine. The input is the user's emotional state, and the output is emotional data. The user's emotional state is detected through micro-facial expression and voice analysis.

[1540] Step 17:

[1541] The server further personalizes recipes and health suggestions based on the acquired emotional data. The input is emotional data, and the output is personalized suggestions. For example, if you are feeling stressed, it generates a recipe for "chamomile tea," which has a relaxing effect.

[1542] Step 18:

[1543] The server sends sentiment-based suggestions to the device and notifies the user. The input is a personalized suggestion, and the output is a notification message. A notification is sent to the device, allowing the user to confirm the suggestion.

[1544] (Application example 2)

[1545] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1546] In today's dual-income households, managing food ingredients and maintaining a healthy diet is a significant burden. Furthermore, emotional states such as stress and fatigue often affect the entire lifestyle, making appropriate responses necessary. While current food management systems can manage ingredients and provide nutritional balance recommendations to a certain extent, they lack security advice and alert functions tailored to the user's emotional state. Therefore, there is a need for a comprehensive food management and health management system that takes emotional states into account.

[1547] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for recognizing the type, quantity, and expiration date of food products; means for transmitting the recognized food product data; means for saving the transmitted data and tracking the expiration date or shelf life; means for generating recipes based on the user's preferences, physical condition, season, weather, economic situation, and emotional state; means for transmitting and notifying the user of the generated recipes; means for checking for missing ingredients and automatically ordering them; means for analyzing the user's nutritional balance and generating advice to support a healthy diet; means for transmitting the generated advice; means for collecting user feedback and updating data; means for analyzing the user's emotional state and providing security advice and alerts based on the analysis; and means for recording fluctuations in the emotional state and generating a daily report. This enables food ingredient management and maintaining a healthy diet, as well as appropriate responses according to the user's emotional state, thereby improving the overall quality of life.

[1548] A "food management device" is a device that recognizes the type, quantity, and expiration date of food, and manages, transmits, and stores this data.

[1549] "Emotional state" refers to the user's mental and emotional state, and includes categories such as stress, fatigue, and relaxation.

[1550] "User preferences" refers to the user's preferred choices and tendencies regarding ingredients and dishes.

[1551] "Security Advice" is safety advice or suggestions provided based on the user's emotional state.

[1552] "Alerts" refer to warnings or notifications based on the user's emotional state or the state of the system.

[1553] "Recipe generation" is the process of creating cooking instructions based on the user's preferences, physical condition, season, weather, financial situation, and emotional state.

[1554] "Automatic ordering" refers to a system that checks for shortages of ingredients and automatically orders the necessary ingredients.

[1555] "Nutritional balance" refers to the harmony of nutrients that a user should consume in their diet.

[1556] "Feedback" refers to the opinions and thoughts provided by users to the system, and the system's data is updated based on them.

[1557] "Data transmission" refers to the process of transmitting the recognized food data to the server.

[1558] "Expiration date tracking" is the act of managing the expiration date and storage period of food based on the transmitted data and notifying the user appropriately.

[1559] "Supporting a healthy diet" means analyzing the user's nutritional balance and providing advice and suggestions for maintaining a proper diet.

[1560] "Report generation" is the process of recording fluctuations in emotional state and creating a summary report at the end of the day.

[1561] "Data updating" is the act of keeping the data in the system up to date based on user feedback.

[1562] System Overview

[1563] An embodiment of this invention includes a food management device and a security management system incorporating an emotion engine. The food management device uses sensors and cameras installed in the refrigerator to collect information such as food type, quantity, and expiration date, and sends the data to a server. The server analyzes this data to manage expiration dates and analyze the nutritional balance of food. It also recognizes the user's emotional state and provides appropriate recipes and security advice based on that information.

[1564] Hardware and Software

[1565] The system uses the following hardware and software:

[1566] Hardware:

[1567] Cameras and RFID readers inside refrigerators

[1568] Smartphone or tablet

[1569] Security systems (cameras, alarms)

[1570] software:

[1571] OpenCV: Image processing from camera

[1572] EmotionRecognizer (virtual emotion engine): Recognizes user emotions

[1573] Python: General application logic

[1574] Data processing and calculation methods

[1575] Food Recognition and Data Collection

[1576] Cameras and RFID readers identify the type, quantity and expiration date of food and transmit this data through a communication device inside the refrigerator to a server, which stores it in a database and tracks expiration dates and storage times.

[1577] Recognition of emotional states

[1578] It uses the smartphone's camera and microphone to capture the user's facial expressions and voice, which are then analyzed by an emotion engine. If the emotional state is recognized as stress or fatigue, it triggers a security alarm and issues a safety alert.

[1579] Recipe and advice generation

[1580] The server generates appropriate recipes based on the user's preferences, physical condition, season, weather, financial situation, and emotional state. It also analyzes the user's emotional state and provides security advice as needed. For example, if the user is feeling extremely stressed, it will suggest cooking or resting methods that will help reduce stress.

[1581] Daily report generation

[1582] It records fluctuations in emotional state and automatically generates a report at the end of each day summarizing the user's emotional state and corresponding behavior, including security and health management advice for the next day.

[1583] Prompt Sentence Examples

[1584] The detailed realization of this application also utilizes a generative AI model, with the following prompts as examples:

[1585] Please create a program that meets the following requirements as an example of a security service that applies a system that incorporates an emotion engine into a food management device for dual-income households to a smartphone app.

[1586] Recognizes user emotions (stress, fatigue, etc.) using a camera

[1587] Trigger security alerts based on emotions

[1588] Providing security advice

[1589] Generate daily reports based on stress levels

[1590] The application is written in Python and uses the OpenCV library.

[1591] In this way, the present invention uses a food management device incorporating an emotion engine to achieve efficient food management and support for a healthy diet, and also performs security management according to the user's emotional state, thereby improving the overall quality of life.

[1592] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1593] Step 1:

[1594] Recognizing food types, quantities, and expiration dates

[1595] A camera and RFID reader installed inside the refrigerator recognize the type, quantity, and expiration date of food. The recognized data is acquired as the food name, quantity, and expiration date and sent to the server via the communication device inside the refrigerator. This allows the server to obtain food inventory information.

[1596] Step 2:

[1597] Data storage and expiration date tracking

[1598] The server stores the submitted food data in a database, while simultaneously running algorithms that track expiration dates and shelf life to identify foods approaching their expiry date, which are then listed for possible user notification along with updates to the database.

[1599] Step 3:

[1600] Recognizing the user's emotional state

[1601] The user captures their voice and facial expressions using the smartphone's camera and microphone. The emotion engine analyzes this input data and recognizes their emotional state, such as stress, fatigue, or relaxation. The recognized emotional state is then sent to the server.

[1602] Step 4:

[1603] Generating recipes and security advice

[1604] The server generates appropriate recipes based on the user's emotional state, preferences, physical condition, season, weather, and economic situation. It may also provide security advice based on the user's emotional state. The generated recipes and advice are sent to the user's smartphone.

[1605] Step 5:

[1606] Automated ordering

[1607] The server analyzes the food data in the refrigerator and lists any ingredients that are lacking. Based on this information, it generates an automatic order based on the user's purchasing history and preferences, and sends an order request to the online shopping site. Once the order is complete, the user is notified.

[1608] Step 6:

[1609] Analyzing the user's nutritional balance and generating advice

[1610] The server analyzes the user's eating habits and calculates nutritional balance. Based on this, it generates advice and recipes for maintaining a healthy diet and notifies the user, including salad recipes that encourage the appropriate intake of green and yellow vegetables.

[1611] Step 7:

[1612] Recording emotional state fluctuations and generating daily reports

[1613] The server records the user's emotional fluctuations throughout the day and compiles the data throughout the day. Based on this data, a summary report of the day is automatically generated, including security and health management advice for the next day. The report is then sent to the user's device.

[1614] Step 8:

[1615] Gathering feedback and updating the system

[1616] Users can send feedback on the recipes and advice provided to the server from their device, and the server receives this feedback and updates the system's database to improve the accuracy of future suggestions and advice.

[1617] This allows servers, terminals, and users to work together to achieve efficient food management, healthy living, and security management.

[1618] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1619] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1620] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1621] [Fourth embodiment]

[1622] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1623] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1624] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1625] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1626] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1627] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1628] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1629] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1630] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1631] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1632] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1633] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1634] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1635] The present invention provides a system that uses a food management device to reduce the burden of food ingredient management in dual-income households and supports efficient and healthy eating habits. Hereinafter, an embodiment of the present invention will be described in detail.

[1636] System Overview

[1637] This system uses various sensors and cameras installed inside the refrigerator to collect information such as food type, quantity, and expiration date, and sends this data to a server. The server analyzes this data to manage expiration dates and analyze the nutritional balance of food. It also generates optimal recipes based on the user's physical condition, preferences, season, weather, and economic situation, and notifies the user. It also automatically orders ingredients that are running low, preventing food waste and supporting healthy eating habits.

[1638] Food Recognition and Data Collection

[1639] Cameras and RFID readers installed inside the refrigerator recognize the type, quantity, and expiration date of food. This data is sent to a server via a communication device inside the refrigerator, and the server stores it in a database. For example, data such as "Milk (1 bottle, expiration date: October 10, 2023)" and "Eggs (6 eggs, expiration date: October 12, 2023)" may be acquired from the refrigerator.

[1640] Data storage and expiration date tracking

[1641] The data sent to the server is stored in a database, where expiration dates and storage periods are tracked. The server then notifies the user of food products that are approaching their expiration date. For example, if the expiration date of "milk" is approaching in two days, the server will send a notification to the device saying, "Milk's expiration date will expire in two days."

[1642] Recipe Suggestions

[1643] The server generates optimal recipes based on the user's preferences, physical condition, season, weather, and economic situation. For example, it generates a recipe for an "omelet" using milk and eggs based on the user's preferences and the ingredients currently in the refrigerator, and sends it to the device. The user can then refer to the recipe on the device and easily cook the dish.

[1644] Automatic ordering of necessary ingredients

[1645] The server analyzes the food data in the refrigerator and lists any ingredients that are running low. For example, if it detects that there are fewer than two eggs, the server generates an automatic ordering content based on the user's purchasing history and preferences. This order content is notified to the terminal, and if the user approves, an ordering request is sent to the online shopping site. After the order is completed, the user is notified that "the egg order has been completed."

[1646] Nutritional balance and health management

[1647] The server analyzes the user's eating habits and calculates nutritional balance. To support a healthy diet, it generates necessary health advice and recipes and notifies the device. For example, if the user has not eaten vegetables recently, it will suggest a salad recipe rich in green and yellow vegetables and notify the user. The user sends feedback to the server via their device, which updates the data throughout the system.

[1648] Specific examples

[1649] Next, the processing of the system will be described in detail using a specific example.

[1650] Example 1: Milk expiration date notification

[1651] The server retrieves information about "Milk (best before date: October 10, 2023)" from the database and confirms that the expiration date is two days later.

[1652] The server notifies the terminal that "The milk's expiration date will expire in two days."

[1653] The device displays this notification to the user.

[1654] Example 2: Omelette recipe suggestion

[1655] The server generates an omelet recipe based on the user's preferences and the information on milk and eggs in the refrigerator.

[1656] The server sends this recipe to the terminal.

[1657] The device notifies the user of the "omelette recipe."

[1658] Example 3: Automated egg ordering

[1659] The server determines that there are fewer than two eggs in the refrigerator.

[1660] The server generates an order for "eggs (pack of 10)" and sends an order request to the online shopping site.

[1661] The terminal notifies the user that "your egg order has been completed."

[1662] In this way, the present invention realizes efficient food management and support for a healthy diet using a food management device. This system reduces the burden of food ingredient management for dual-income households and provides a healthy diet that takes nutritional balance into consideration.

[1663] The processing flow will be explained below.

[1664] Step 1:

[1665] The refrigerator detects when the door is opened and closed, activating the internal camera and RFID reader. This recognizes data such as the type of food, quantity, and expiration date. For example, it obtains information such as "Milk (1 bottle, expiration date: October 10, 2023)" and "Eggs (6 eggs, expiration date: October 12, 2023)."

[1666] Step 2:

[1667] The refrigerator sends the food data it recognizes to the server, including the type of food, quantity, and expiration date.

[1668] Step 3:

[1669] The server stores the received food data in a database. The data is structured as information such as the type, quantity, expiration date, and storage period of each food item.

[1670] Step 4:

[1671] The server periodically checks the database and lists foods that are close to their expiration date. For example, it detects that the expiration date of "Milk (expiration date: October 10, 2023)" is just two days away.

[1672] Step 5:

[1673] The server prepares notifications for food items that are nearing their expiration date. For items with an expiration date within two days, the server prepares to notify the user.

[1674] Step 6:

[1675] The server sends the notification content to the terminal. For example, it sends notification data saying "The expiration date of the milk will be reached in two days."

[1676] Step 7:

[1677] The device displays the received notification to the user. For example, a notification pops up on the smartphone saying "The expiration date of the milk will expire in two days."

[1678] Step 8:

[1679] The server generates recipes based on the user's preferences, physical condition, season, weather, and economic situation, and analyzes various collected data to select the optimal recipe.

[1680] Step 9:

[1681] The server sends the generated recipe to the terminal. For example, it sends a "recipe for an omelet using milk and eggs."

[1682] Step 10:

[1683] The device notifies the user of the received recipe. For example, the smartphone displays "An omelet recipe has been suggested."

[1684] Step 11:

[1685] The server analyzes the food data in the refrigerator and lists the ingredients that are running low. For example, it might determine that there are fewer than two eggs.

[1686] Step 12:

[1687] The server generates automatic ordering based on the ingredients that are in short supply. Taking into account the user's purchasing history and preferences, it prepares an order request for "Eggs (pack of 10)."

[1688] Step 13:

[1689] The server sends the order to the terminal and asks the user for approval, for example, sending a confirmation message saying "Do you want to automatically order eggs?"

[1690] Step 14:

[1691] The terminal displays the received order confirmation to the user, who can then select "Accept" or "Cancel" in response to the confirmation message.

[1692] Step 15:

[1693] The user responds with "approval" to the order confirmation. The response data is sent from the terminal to the server.

[1694] Step 16:

[1695] The server receives the user's approval and sends an order request to the online shopping site. For example, it sends a request to order "eggs (pack of 10)."

[1696] Step 17:

[1697] The server sends a notification of order completion to the terminal, for example, data such as "Your egg order has been completed."

[1698] Step 18:

[1699] The device will notify the user that their eggs have been ordered. The user can then check the status of their order.

[1700] Step 19:

[1701] The server collects data on the user's eating habits and analyzes their nutritional balance. For example, it obtains information such as, "I've been eating less vegetables recently."

[1702] Step 20:

[1703] The server generates health advice and recipes based on the analysis results, such as a recipe for a salad rich in green and yellow vegetables.

[1704] Step 21:

[1705] The server sends the generated advice or recipe to the terminal, for example, sending a "salad recipe."

[1706] Step 22:

[1707] The device will notify the user of the advice or recipes it has received, and the user can view the notification and incorporate the advice or recipes.

[1708] Step 23:

[1709] If the user actually uses the suggested advice or recipe, feedback is sent via the device.

[1710] Step 24:

[1711] The server receives the feedback submitted by the user and updates the database, which improves the accuracy of the system and allows for a more personalized service.

[1712] Example 1

[1713] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1714] The present invention relates to a system that reduces the burden of food ingredient management for dual-income households and supports efficient and healthy eating habits. Conventional food management systems often require manual management of food expiration dates and analysis of nutritional balance, which can be time-consuming and labor-intensive for dual-income households. Furthermore, automatic food ordering and recipe suggestions may not be performed appropriately, potentially resulting in food waste and nutritional imbalances. A solution to these issues is needed.

[1715] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1716] In this invention, the server includes means for recognizing the type, quantity, and expiration date of food, means for transmitting the recognized food data, means for storing the transmitted data and tracking the expiration date or storage period, means for generating recipes based on the user's preferences, physical condition, season, weather conditions, and economic situation, means for transmitting and notifying the user of the generated recipes, means for checking for missing ingredients and automatically ordering, means for analyzing the user's nutritional balance and generating suggestions to support a healthy diet, means for transmitting the generated suggestions, means including a communication device for collecting food data, and means for collecting user feedback and updating the data. This reduces the burden of food management for dual-income households and makes it possible to support an efficient and healthy diet.

[1717] A "food management device" is a device that recognizes and manages information such as food type, quantity, and expiration date.

[1718] "Means for recognizing the type, quantity, and expiration date of food" refers to means for identifying the type, quantity, and expiration date of food using a camera, RFID reader, etc.

[1719] The "means for transmitting recognized food data" is a communication device for transmitting information about the recognized food to a server or database.

[1720] "Means for storing submitted data and tracking expiration dates or shelf life" refers to means for storing submitted food data in a database and managing and tracking expiration dates or shelf life.

[1721] "Means for generating recipes based on the user's preferences, physical condition, season, weather conditions, and economic situation" refers to a program or system that generates optimal recipes based on the user's personal information and external conditions.

[1722] The "means for transmitting and notifying the generated recipe" is a communication and notification system for transmitting and notifying the generated recipe to the user's terminal.

[1723] The "means of checking for shortages of ingredients and automatically ordering" is a system that analyzes food data in the refrigerator, identifies shortages of food, and automatically orders them.

[1724] The "means for analyzing a user's nutritional balance and generating suggestions to support a healthy diet" is a system that analyzes a user's eating habit data and generates suggestions to support a healthy diet from the perspective of nutritional balance.

[1725] The "means for transmitting the generated suggestions" is a communication system for transmitting the generated health suggestions to the user's terminal.

[1726] The "means including a communication device for collecting food data" is a system including a communication device for collecting food information and transmitting it to a server.

[1727] "Means for collecting user feedback and updating data" refers to means that has the function of receiving feedback from users and updating data throughout the system based on that feedback.

[1728] MODE FOR CARRYING OUT THE INVENTION

[1729] The present invention provides a system that uses a food management device to reduce the burden of food ingredient management in dual-income households and supports efficient and healthy eating habits. Hereinafter, an embodiment of the present invention will be described in detail.

[1730] System Overview

[1731] This system uses various sensors and cameras installed inside the refrigerator to collect information such as food type, quantity, and expiration date, and sends this data to a server. The server analyzes this data to manage expiration dates and analyze the nutritional balance of food. It also generates optimal recipes based on the user's physical condition, preferences, season, weather conditions, and economic situation, and notifies the user. It also automatically orders ingredients that are running low, preventing food waste and supporting healthy eating habits.

[1732] Food Recognition and Data Collection

[1733] Cameras and RFID readers installed inside the refrigerator recognize the type, quantity, and expiration date of food. This data is sent to a server via a communication device inside the refrigerator, and the server stores it in a database. For example, data such as "Milk (1 bottle, expiration date: October 10, 2023)" and "Eggs (6 eggs, expiration date: October 12, 2023)" may be acquired from the refrigerator.

[1734] Data storage and expiration date tracking

[1735] The data sent to the server is stored in a database, where expiration dates and storage periods are tracked. The server then notifies the user of food products that are approaching their expiration date. For example, if the expiration date of "milk" is approaching in two days, the server will send a notification to the device saying, "Milk's expiration date will expire in two days."

[1736] Recipe Suggestions

[1737] The server generates optimal recipes based on the user's preferences, physical condition, season, weather conditions, and economic situation. For example, it generates a recipe for an "omelet" using milk and eggs based on the user's preferences and the ingredients currently in the refrigerator, and sends it to the device. The user can then refer to the recipe on the device and easily cook the dish.

[1738] Automatic ordering of necessary ingredients

[1739] The server analyzes the food data in the refrigerator and lists any ingredients that are running low. For example, if it detects that there are fewer than two eggs, the server generates an automatic ordering content based on the user's purchasing history and preferences. This order content is notified to the terminal, and if the user approves, an ordering request is sent to the e-commerce site. After the order is completed, the server sends a notification to the user that "the egg order has been completed."

[1740] Nutritional balance and health management

[1741] The server analyzes the user's eating habits and calculates nutritional balance. To support a healthy diet, it generates necessary health advice and recipes and notifies the device. For example, if the user has not eaten vegetables recently, it will suggest a salad recipe rich in green and yellow vegetables and notify the user. The user sends feedback to the server via their device, which updates the data throughout the system.

[1742] Specific examples

[1743] Next, the processing of the system will be described in detail using a specific example.

[1744] Example 1: Milk expiration date notification

[1745] The server retrieves information about "Milk (best before date: October 10, 2023)" from the database and confirms that the expiration date is two days later.

[1746] The server notifies the terminal that "The milk's expiration date will expire in two days."

[1747] The device displays this notification to the user.

[1748] Example 2: Omelette recipe suggestion

[1749] The server generates an omelet recipe based on the user's preferences and the information on milk and eggs in the refrigerator.

[1750] The server sends this recipe to the terminal.

[1751] The device notifies the user of the "omelette recipe."

[1752] Example 3: Automated egg ordering

[1753] The server determines that there are fewer than two eggs in the refrigerator.

[1754] The server generates an order for "Eggs (pack of 10)" and sends the order request to the e-commerce site.

[1755] The terminal notifies the user that "your egg order has been completed."

[1756] Prompt Sentence Examples

[1757] Below is an example of a prompt sentence to input to the generative AI model.

[1758] "Please analyze the food in my refrigerator and its expiration date and suggest the best recipes."

[1759] "Analyze the ingredients that are in short supply and use an automated ordering system to generate orders for the ingredients that are needed."

[1760] "Based on the user's eating habits data, create nutritional advice to support healthy eating habits."

[1761] In this way, the present invention realizes efficient food management and support for a healthy diet using a food management device. This system reduces the burden of food ingredient management for dual-income households and provides a healthy diet that takes nutritional balance into consideration.

[1762] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1763] Step 1: Collect food data

[1764] Input: Food in the fridge

[1765] Output: Food type, quantity, expiration date data

[1766] What happens:

[1767] A camera installed inside the refrigerator takes pictures of the food, and an RFID reader reads the data from the RFID tags.

[1768] The communication device inside the refrigerator acquires the image data from the camera and the tag data from the RFID reader.

[1769] The data obtained includes the type of food (e.g., "milk" or "egg"), quantity (e.g., "1 bottle" or "6 pieces"), and expiration date (e.g., "October 10, 2023" or "October 12, 2023").

[1770] The communication device transmits this data to the server.

[1771] Step 2: Data storage and analysis

[1772] Input: Food type, quantity, expiration date data

[1773] Output: Saved food data, analysis results

[1774] What happens:

[1775] The server receives the food data transmitted from the refrigerator.

[1776] The received data includes the type of food, quantity, expiration date, etc.

[1777] The server stores this data in a database (e.g., "Milk (1 bottle, expiration date: October 10, 2023)" and "Eggs (6 eggs, expiration date: October 12, 2023)").

[1778] The server analyzes the stored data to identify current food inventory and foods that are close to their expiration date.

[1779] Step 3: Best before date notification

[1780] Input: Food data in the database

[1781] Output: Best before date notification message

[1782] What happens:

[1783] The server periodically scans the food data in the database to identify foods that are approaching their expiration date.

[1784] For example, find "milk" whose expiration date is just two days away.

[1785] The server generates a notification message saying "Milk expires in 2 days."

[1786] Send this message to the user's terminal.

[1787] The device will display this notification to the user.

[1788] Step 4: Recipe suggestions

[1789] Input: User preferences, physical condition, season, weather conditions, economic situation, food data in refrigerator

[1790] Output: Recipe suggestion message

[1791] What happens:

[1792] The server collects data on the user's preferences, physical condition, season, weather conditions, and economic situation.

[1793] It also retrieves food data from the refrigerator and checks the current ingredients.

[1794] For example, suppose you have milk and eggs in your refrigerator.

[1795] The server uses this information to input prompts into the generative AI model (e.g., "There is currently milk and eggs in the refrigerator. Please suggest some recipes using these.").

[1796] The generative AI model generates a recipe for "omelette" based on prompts.

[1797] The server sends this generated recipe to the user's terminal, and the user is notified of the "omelette recipe."

[1798] Step 5: Automatically order the ingredients you need

[1799] Input: Food data in the refrigerator

[1800] Output: Order completion notification message

[1801] What happens:

[1802] The server analyzes the database and detects that there is a shortage of food in the refrigerator.

[1803] For example, identify a decrease to "less than two eggs."

[1804] The server generates an order (e.g., "pack of 10 eggs") based on the user's past purchasing history and preferences.

[1805] The server sends the order details to the electronic commerce site as an order request.

[1806] After the order is completed, the server sends a notification to the user's terminal saying "Egg order completed."

[1807] Step 6: Nutritional Balance and Health Management

[1808] Input: User's dietary habits data

[1809] Output: Health suggestion message

[1810] What happens:

[1811] The server periodically analyzes the user's eating habits data.

[1812] Based on the analysis results, nutritional balance is calculated and any missing nutrients are identified.

[1813] For example, it detects when a user has not eaten vegetables recently.

[1814] The server generates recipes for salads rich in green vegetables to support a healthy diet.

[1815] This generated recipe is sent to the user's device and notified to the user.

[1816] Users send feedback to the server through their devices, and data is updated throughout the system.

[1817] (Application example 1)

[1818] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1819] In busy households, such as dual-income households, it is difficult to manage food and ensure a balanced diet. Even when shopping at brick-and-mortar stores, it can be time-consuming to know what to buy and whether there are any foods nearing their expiration date. Furthermore, there is a need for a method to efficiently procure ingredients while maintaining a healthy diet. A system that solves these issues and supports food management and healthy eating habits at home is needed.

[1820] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1821] In this invention, the server includes means for recognizing the type, quantity, and expiration date of food, means for transmitting the recognized food data, means for storing the transmitted data and tracking the expiration date or storage period, means for generating recipes based on the user's preferences, physical condition, season, weather, and economic situation, means for transmitting and notifying the user of the generated recipes, means for checking for missing ingredients and automatically ordering, means for analyzing the user's nutritional balance and generating advice to support a healthy diet, means for transmitting the generated advice, means for collecting user feedback and updating data, means for displaying ingredient information in a physical store via smart glasses and presenting a list of ingredients that need to be replenished, and means for presenting recipes and special sale information generated based on the user's profile and external environment information in real time, thereby enabling more efficient food management and supporting a healthy diet.

[1822] A "food management device" is a device for recognizing and managing information such as the type, quantity, and expiration date of food.

[1823] A "sensor" is a device that detects a physical quantity or state and outputs it as an electrical signal.

[1824] A "camera" is a device that can capture video or still images and store and transmit them as digital data.

[1825] A "communication device" is a device for sending and receiving data to and from other devices and servers.

[1826] A "server" is a computer system that stores, analyzes, and processes data and provides information in response to requests from other devices.

[1827] "User preferences" refers to information that indicates the user's taste and cooking preferences.

[1828] "Physical condition" is information that indicates the user's health condition and physical condition.

[1829] "Season" refers to the time of year brought about by changes in the climate.

[1830] "Weather" is information indicating weather conditions in a particular area and time.

[1831] "Financial status" is information that indicates the user's financial status, such as income, expenses, and budget.

[1832] A "recipe" is a detailed list of instructions and ingredients needed to make a particular dish.

[1833] "Notification" means sending messages or alerts to inform users.

[1834] "Auto ordering" is the process of automatically ordering necessary items and ingredients.

[1835] "Nutritional balance" refers to a state in which the user's diet properly incorporates the necessary nutrients.

[1836] "Advice" is information that provides advice or recommendations to the user.

[1837] "Feedback" refers to opinions and evaluations provided by users to the system.

[1838] "Smart glasses" are wearable devices that have the ability to display visual information.

[1839] A "brick and mortar store" is a physical commercial establishment where users can visit and purchase products.

[1840] "Food information" refers to data such as food type, quantity, expiration date, and storage conditions.

[1841] A "profile" is a data set that compiles personal information and characteristics about a user.

[1842] "External environment information" refers to information that is not directly managed by the user, such as weather, season, and surrounding conditions.

[1843] "Special sale information" is information about products being sold at a price lower than the regular price.

[1844] The present invention is a system for supporting home food management and healthy eating habits. The purpose of the present invention is to streamline food management in physical stores through smart glasses and improve the user's shopping experience. Specific embodiments for implementing the present invention are described in detail below.

[1845] System Overview

[1846] This system uses various sensors and cameras installed inside the refrigerator to collect information such as food type, quantity, and expiration date, and sends this data to a server. The server analyzes this data to manage expiration dates and analyze the nutritional balance of food. In addition, when a user wearing smart glasses purchases ingredients in a physical store, the system displays real-time information on necessary ingredients, special sales, recipe suggestions, and more.

[1847] Hardware and Software

[1848] Hardware:

[1849] Cameras and sensors inside the refrigerator: Used to collect food data.

[1850] Smart glasses: Using "Google Glass" and "Vuzix Blade" as examples.

[1851] Server: We use AWS and Azure for data analysis and notifications.

[1852] software:

[1853] Data collection software: Collects data from cameras and sensors inside the refrigerator and sends it to a server.

[1854] Data analysis software: Food data, user information, and external environmental information are analyzed on the server.

[1855] User Interface: An application for display on smart glasses.

[1856] Specific processing

[1857] 1. Food Recognition and Data Collection:

[1858] Cameras and RFID readers installed inside the refrigerator identify the type, quantity and expiration date of food and send this data to a server, which stores it in a database and tracks expiration dates.

[1859] 2. Information display through smart glasses:

[1860] When users shop in a physical store, the smart glasses display real-time information about the ingredients in their refrigerator. For example, if milk or eggs are low in stock, the glasses notify the user. They also display sales and coupon information to support their purchasing behavior.

[1861] 3. Recipe and nutritional suggestions:

[1862] The server uses a generative AI model to generate appropriate recipes based on the user's preferences, physical condition, season, weather, and economic situation. The generated recipes are then sent to the user via smart glasses. The server also analyzes nutritional balance and sends advice to support healthy eating.

[1863] Specific examples

[1864] A specific example of the system's operation is shown below.

[1865] Example 1: Shopping assistance in physical stores

[1866] The user wears smart glasses and goes shopping in a physical store. The smart glasses' display shows real-time information about the food inventory in the refrigerator. A notification such as "You're low on milk and need to buy more" is displayed, along with information about special sales and coupons available in the store.

[1867] Example 2: Recipe suggestions

[1868] When a user is using the smart glasses at home, the server generates an "omelette" recipe based on the user's current health condition, preferences, and information about ingredients in the refrigerator, and notifies the smart glasses. The following prompt sentence is used for the generating AI model:

[1869] Example prompt for a generative AI model:

[1870] "The user has a bottle of milk and two eggs in the refrigerator. It's fall, and the user prefers a balanced diet. Based on this, please suggest some suitable recipes."

[1871] In this way, the present invention can utilize smart glasses to comprehensively support food management in brick-and-mortar stores and improve users' quality of life.

[1872] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1873] Step 1:

[1874] Food data collection

[1875] The server receives input from cameras and RFID readers installed inside the refrigerator. This input includes the type of food, quantity, and expiration date. The server analyzes this data and stores it in a database for each food item. For example, information such as "Milk (Quantity: 1 bottle, Best before date: October 10, 2023)" may be collected.

[1876] Step 2:

[1877] Best before date tracking

[1878] The server periodically analyzes the expiration dates of the food items stored in the database and lists the food items that are approaching their expiration date. For example, it lists the food items whose expiration date is two days away and prepares a notification such as "The expiration date will expire in two days." This notification is sent to the user's device.

[1879] Step 3:

[1880] Detecting missing ingredients

[1881] The server analyzes the food data in the database and detects ingredients that are low in stock. For example, if it determines that there are fewer than two eggs, it adds them to a list of ingredients that are in short supply. The server notifies the user of the list of ingredients that are in short supply, and clearly indicates which ingredients need to be replenished.

[1882] Step 4:

[1883] Preparing for automatic ordering

[1884] The server generates an automatic order for missing ingredients based on the user's purchasing history and preferences. For example, it prepares an order for "eggs (pack of 10)" and prepares to send a request to an online shopping site. This order preparation data is notified to the user and they are asked to approve it.

[1885] Step 5:

[1886] Recipe Generation

[1887] Based on the input data of the user's preferences, physical condition, season, weather, and economic situation, the server uses a generative AI model to generate the optimal recipe. The generative AI model is input with the following prompt:

[1888] "The user has a bottle of milk and two eggs in the refrigerator. It's fall, and the user prefers a balanced diet. Based on this, please suggest some suitable recipes."

[1889] The generated recipe, for example an "omelette" recipe, is completed and sent to the user terminal.

[1890] Step 6:

[1891] Information display on smart glasses

[1892] When a user wears the smart glasses and goes shopping in a physical store, the device displays real-time information about ingredients in the refrigerator and a list of ingredients that are running low. For example, a notification will appear on the smart glasses saying, "You're low on milk and need to buy more." In addition, the device will display information about special sales and coupons in the store, allowing users to shop efficiently.

[1893] Step 7:

[1894] Nutritional balance analysis and advice delivery

[1895] The server analyzes the user's eating habits, calculates nutritional balance, and generates advice to support a healthy diet. For example, it generates advice such as "You haven't been eating enough vegetables recently," and notifies the user's device. This advice collects user feedback and serves as the basis for updating data throughout the system.

[1896] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1897] The present invention provides a system that uses a food management device combined with an emotion engine to reduce the burden of food ingredient management for dual-income households and support efficient and healthy eating habits. The following describes in detail the embodiments of the present invention.

[1898] System Overview

[1899] This system uses sensors and cameras installed inside the refrigerator to collect information such as food type, quantity, and expiration date, and sends this data to a server. The server analyzes this data to manage expiration dates and analyze the nutritional balance of food. It also generates optimal recipes based on the user's physical condition, preferences, season, weather, economic situation, and emotions, and notifies the user. It also automatically orders ingredients that are running low, preventing food waste and supporting healthy eating habits.

[1900] Food Recognition and Data Collection

[1901] Cameras and RFID readers installed inside the refrigerator recognize the type, quantity, and expiration date of food. This data is sent to a server via a communication device inside the refrigerator, and the server stores it in a database. For example, data such as "Milk (1 bottle, expiration date: October 10, 2023)" and "Eggs (6 eggs, expiration date: October 12, 2023)" may be acquired from the refrigerator.

[1902] Data storage and expiration date tracking

[1903] The data sent to the server is stored in a database, where expiration dates and storage periods are tracked. The server then notifies the user of food products that are approaching their expiration date. For example, if the expiration date of "milk" is approaching in two days, the server will send a notification to the device saying, "Milk's expiration date will expire in two days."

[1904] Recipe Suggestions

[1905] The server generates optimal recipes based on the user's preferences, physical condition, season, weather, economic situation, and emotional data obtained by the emotion engine. For example, it generates a recipe for an "omelette" using "milk" and "eggs" based on the user's preferences and the ingredients currently in the refrigerator, and sends it to the device. The user can then refer to the recipe on their device and easily cook the dish.

[1906] Automatic ordering of necessary ingredients

[1907] The server analyzes the food data in the refrigerator and lists any ingredients that are running low. For example, if it detects that there are fewer than two eggs, the server generates an automatic ordering content based on the user's purchasing history and preferences. This order content is notified to the terminal, and if the user approves, an ordering request is sent to the online shopping site. After the order is completed, the user is notified that "the egg order has been completed."

[1908] Nutritional balance and health management

[1909] The server analyzes the user's eating habits and calculates nutritional balance. To support a healthy diet, it generates necessary health advice and recipes and notifies the device. For example, if the user has not eaten vegetables recently, it will suggest a salad recipe rich in green and yellow vegetables and notify the user. The user sends feedback to the server via their device, which updates the data throughout the system.

[1910] Further suggestions from the emotion engine

[1911] The emotion engine recognizes the user's emotions and further personalizes recipes and health advice based on this. For example, if the user is feeling stressed, it will suggest relaxing recipes that will help reduce stress.

[1912] Specific examples

[1913] Example 1: Emotion-based recipe suggestions

[1914] The server collects the user's emotional state through an emotion engine. For example, if the user is "feeling stressed," emotion data is acquired.

[1915] The server generates relaxation recipes, including chamomile tea, which is effective in reducing stress.

[1916] The server sends this relaxation recipe to the terminal.

[1917] The device will display a notification to the user saying, "A suggestion has been made for making chamomile tea."

[1918] Example 2: Emotion-based health advice

[1919] The server obtains emotional data from the emotion engine, such as "feeling tired" by the user.

[1920] The server generates advice including "vitamin-rich dishes" that are effective in relieving fatigue.

[1921] The server sends this advice to the terminal.

[1922] The device will display a notification to the user saying, "A dish that is effective in relieving fatigue has been suggested."

[1923] In this way, the present invention realizes efficient food management and support for healthy eating habits using a food management device with an emotion engine built in. This system reduces the burden of food ingredient management for dual-income households and provides a healthy diet that takes nutritional balance into consideration.

[1924] The processing flow will be explained below.

[1925] Step 1:

[1926] The refrigerator detects when the door is opened and closed, activating the internal camera and RFID reader. This recognizes data such as the type of food, quantity, and expiration date. For example, it obtains information such as "Milk (1 bottle, expiration date: October 10, 2023)" and "Eggs (6 eggs, expiration date: October 12, 2023)."

[1927] Step 2:

[1928] The refrigerator sends the food data it recognizes to the server, including the type of food, quantity, and expiration date.

[1929] Step 3:

[1930] The server stores the received food data in a database. The data is structured as information such as the type, quantity, expiration date, and storage period of each food item.

[1931] Step 4:

[1932] The server periodically checks the database and lists foods that are close to their expiration date. For example, it detects that the expiration date of "Milk (expiration date: October 10, 2023)" is just two days away.

[1933] Step 5:

[1934] The server prepares notifications for food items that are nearing their expiration date. For items with an expiration date within two days, the server prepares to notify the user.

[1935] Step 6:

[1936] The server sends the notification content to the terminal. For example, it sends notification data saying "The expiration date of the milk will be reached in two days."

[1937] Step 7:

[1938] The device displays the received notification to the user. For example, a notification pops up on the smartphone saying "The expiration date of the milk will expire in two days."

[1939] Step 8:

[1940] The server generates optimal recipes based on the user's preferences, physical condition, season, weather, economic situation, and emotional data obtained by the emotion engine. For example, it generates a recipe for "omelette with milk and eggs" based on the user's preferences and the ingredients currently in the refrigerator.

[1941] Step 9:

[1942] The server sends the generated recipe to the terminal. For example, it sends a "recipe for an omelet using milk and eggs."

[1943] Step 10:

[1944] The device notifies the user of the received recipe, for example, "An omelet recipe has been suggested."

[1945] Step 11:

[1946] The server analyzes the food data in the refrigerator and lists the ingredients that are running low. For example, it might determine that there are fewer than two eggs.

[1947] Step 12:

[1948] The server generates automatic ordering based on the ingredients that are in short supply. Taking into account the user's purchasing history and preferences, it prepares an order request for "Eggs (pack of 10)."

[1949] Step 13:

[1950] The server sends the order to the terminal and asks the user for approval, for example, sending a confirmation message saying "Do you want to automatically order eggs?"

[1951] Step 14:

[1952] The terminal displays the received order confirmation to the user, who can then select "Accept" or "Cancel" in response to the confirmation message.

[1953] Step 15:

[1954] The user responds with "approval" to the order confirmation. The response data is sent from the terminal to the server.

[1955] Step 16:

[1956] The server receives the user's approval and sends an order request to the online shopping site. For example, it sends a request to order "eggs (pack of 10)."

[1957] Step 17:

[1958] The server sends a notification of order completion to the terminal, for example, data such as "Your egg order has been com...

Claims

1. In the food management device, A means of recognizing the type, quantity, and expiration date of food; means for transmitting the recognized food data; A means for storing the transmitted data and tracking expiration dates or shelf life; A means for generating recipes based on the user's preferences, physical condition, season, weather, and economic situation; A means for transmitting and notifying the generated recipe; A way to check for shortages of ingredients and automatically order them, A means to analyze the user's nutritional balance and generate advice to support a healthy diet; means for transmitting the generated advice; A system that includes a means to collect user feedback and update data.

2. The system according to claim 1, wherein the transmitted data is analyzed, a list of foods approaching their expiration date is compiled, and a notification is prepared.

3. 2. The system according to claim 1, wherein the system generates automatic ordering content based on the user's purchasing history and preferences, and transmits a request to the online shopping site.

Images