System

Abstract

A system is provided.SOLUTION: A system includes camera means for capturing an image of food in a refrigerator, image analysis means for analyzing the image captured by the camera means to recognize the food, recipe suggestion means for suggesting a currently available recipe based on recognized food information, nutrition management means for calculating nutrients of each food and performing nutritionally balanced dietary management, and food loss prevention means for estimating a use-by date based on purchase date information of the food and generating an alert for the food approaching the use-by date.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] The present invention aims to solve social issues such as preventive medicine in a super-aging society, reducing the housework burden for dual-income households, and food waste. Specifically, the objective is to provide a means to provide balanced meals, improve housework efficiency, and reduce food waste. Conventional refrigerators have made it difficult to manage food in the refrigerator and prevent food waste due to expiration dates. Furthermore, they lacked support functions for easily preparing nutritionally balanced meals. [Means for solving the problem]

[0005] To solve the above-mentioned problems, the present invention provides the following means. Specifically, the refrigerator includes a camera means for photographing food in the refrigerator and an image analysis means for analyzing the images photographed by the camera means to recognize the food. The refrigerator also includes a recipe suggestion means for suggesting currently available recipes based on the recognized food information, thereby supporting nutritionally balanced dietary management. Furthermore, the refrigerator also provides a food waste prevention means that includes a nutrition management means, calculates the nutrients of each food item, estimates the expiration date based on the food's purchase date information, and generates an alert for food that is approaching its expiration date. This realizes a system that efficiently manages food in the refrigerator and can solve social issues such as preventive medicine, reducing the burden of housework, and preventing food waste.

[0006] "Camera means" refers to a camera and its peripheral devices installed to take pictures of food in the refrigerator.

[0007] "Image analysis means" refers to software and algorithms for analyzing images captured by the camera means and recognizing food from the images.

[0008] The "recipe suggestion means" refers to a function and means for searching for recipes that can be made with currently available ingredients based on the recognized food information and suggesting them to the user.

[0009] "Nutrition management tool" refers to the function and implementation means for calculating the nutrients of food and supporting balanced dietary management.

[0010] "Food waste prevention measures" refers to the function and implementation method for estimating the expiration date based on the purchase date of food and generating an alert for food that is approaching its expiration date.

[0011] The "notification means" refers to a function for notifying the user terminal of recipe suggestions and alerts generated by the recipe suggestion means and means for implementing the function.

[0012] "Report generation means" refers to a function for recording a user's dietary history, analyzing nutritional balance, and creating a report, and means for implementing the function. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0014] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0015] First, the terms used in the following description will be explained.

[0016] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0017] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0018] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0019] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0020] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0021] [First embodiment]

[0022] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0023] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0024] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0025] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0026] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0027] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0028] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0029] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0030] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0032] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0033] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0034] The system of the present invention has a built-in camera, image analysis, recipe suggestion, nutrition management, and food waste prevention means. By using this system, food in the refrigerator can be efficiently managed, and social issues such as preventive medicine, reducing the burden of housework, and preventing food waste can be addressed.

[0035] The camera means periodically photographs the food in the refrigerator and sends the image data to the server. The server analyzes the received images using the image analysis means and recognizes the food. The recognized food information is stored in a database, and the recipe suggestion means suggests appropriate recipes to the user based on this information. The nutrition management means also calculates the nutritional value of the food and performs nutritional management to provide a balanced diet.

[0036] Furthermore, the food loss prevention system estimates the expiration date based on the purchase date of the food, generates an alert for food that is approaching its expiration date, and notifies the user's device, thereby preventing food waste.

[0037] The specific processing contents of the program will be explained in natural language below.

[0038] Ingredient recognition processing

[0039] The server receives the image taken by the camera means and analyzes the image using the image analysis means. For example, if there are tomatoes, milk, and eggs on the shelves of a refrigerator, the server identifies these foods and stores them in the database as "3 tomatoes, 1 carton of milk, 6 eggs."

[0040] Recipe suggestion processing

[0041] When a user requests a dinner recipe, the server retrieves information about the food in the refrigerator from the database and searches the recipe database. For example, if there are tomatoes and milk in the refrigerator, it will suggest a recipe for tomato cream pasta. This suggestion is sent to the user's smartphone, and the user can check the cooking instructions and other ingredients needed through the app.

[0042] Nutritional balance management

[0043] To record the meals a user has eaten, the server collects information on suggested recipes and the meals actually cooked, allowing the system to calculate how much protein, carbohydrates, and fat a user has consumed in a week and notify them of any nutrient deficiencies.

[0044] For example, if a user's protein intake is low over the course of a week, the server will generate a report saying, "You're not getting enough protein. Try some high-protein recipes."

[0045] Preventing food waste

[0046] The server manages food expiration dates using information on food purchase dates and an expiration date estimation algorithm. For example, if milk's expiration date is approaching in three days, the server generates an alert saying, "The milk's expiration date is approaching. Please consume it soon," and notifies the user's smartphone.

[0047] As described above, the system of the present invention provides a specific method for efficiently managing food in a refrigerator, supporting health management, and reducing food waste.

[0048] The processing flow will be explained below.

[0049] Ingredient recognition processing

[0050] Step 1:

[0051] The camera takes pictures of the inside of the refrigerator.

[0052] The camera means periodically takes an image of each shelf in the refrigerator, and generates image data.

[0053] Step 2:

[0054] The image data is sent to a server.

[0055] The image data acquired by the camera means is transmitted to a server via the Internet.

[0056] Step 3:

[0057] The server analyzes the image data.

[0058] The server uses image analysis means to recognize foods from the image, for example, the server identifies tomatoes, milk, and eggs from the image.

[0059] Step 4:

[0060] The recognized food information is stored in a database.

[0061] The server stores the recognition result "3 tomatoes, 1 carton of milk, 6 eggs" in a database.

[0062] Recipe suggestion processing

[0063] Step 1:

[0064] A user requests recipe suggestions.

[0065] A user requests dinner recipe suggestions on a smartphone app.

[0066] Step 2:

[0067] The server acquires information about the food in the refrigerator.

[0068] The server retrieves the latest food information from the database.

[0069] Step 3:

[0070] The server searches the recipe database.

[0071] The server searches for available recipes based on the food information in the refrigerator. For example, if there are tomatoes and milk, it will extract the recipe for "Tomato Cream Pasta."

[0072] Step 4:

[0073] The server creates recipe suggestions.

[0074] The server creates a recipe suggestion that includes ingredients, cooking instructions, and nutritional information.

[0075] Step 5:

[0076] Recipe suggestions are notified to the device.

[0077] The server notifies the user of the recipe suggestions on their smartphone, which are then displayed within the app.

[0078] Nutritional balance management

[0079] Step 1:

[0080] Record the user's dietary history.

[0081] The app records the recipes the user has cooked and sends that information to the server.

[0082] Step 2:

[0083] The server calculates the nutrients.

[0084] The server calculates the nutrients contained in each meal and stores them in a database.

[0085] Step 3:

[0086] The server analyzes the nutritional balance.

[0087] The server analyzes nutritional balance based on dietary history over a certain period (e.g., one week) and identifies any nutrient deficiencies or excess intake.

[0088] Step 4:

[0089] Create a report.

[0090] The server generates a detailed report based on the results of the nutritional balance analysis, including, for example, "You are lacking in protein."

[0091] Step 5:

[0092] The report is sent to the device.

[0093] The server sends the created report to the user's smartphone, where the user can check the report contents through the app.

[0094] Preventing food waste

[0095] Step 1:

[0096] The server estimates the expiration date.

[0097] The server estimates the expiration date based on the purchase date of the food product. For example, it calculates the expiration date of milk.

[0098] Step 2:

[0099] Manage expiration date information.

[0100] The server manages the estimated expiration dates in a database and sets an alert flag when the expiration date approaches.

[0101] Step 3:

[0102] Generate an alert.

[0103] The server generates alerts for food products that are approaching their expiration date, for example, "Milk expires tomorrow."

[0104] Step 4:

[0105] Sends alerts to the device.

[0106] The server sends an alert to the user's smartphone, which is then displayed within the app.

[0107] Specific examples

[0108] 1. If there are tomatoes, milk, or eggs in the refrigerator, the camera means takes a picture of them and sends the image to the server.

[0109] 2. The server uses image analysis to recognize three tomatoes, one carton of milk, and six eggs, and stores them in a database.

[0110] 3. When a user requests a recipe for dinner, the server retrieves information about the food in the refrigerator, searches for available recipes, and suggests "Tomato Cream Pasta."

[0111] 4. When the user cooks a recipe and records their meal history in the app, the server calculates the nutrients and analyzes a week's worth of meal history to check nutritional balance.

[0112] 5. When the milk is nearing its expiration date, the server estimates the expiration date and generates an alert to notify the user.

[0113] The above is a specific processing flow using the system of the present invention.

[0114] Example 1

[0115] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0116] In modern society, there is a demand for both labor-saving and health management. However, food management is complicated and time-consuming. In particular, properly managing food in the refrigerator and consuming it while it is fresh is effective in preventing food waste, but manual management is difficult and not often implemented. It is also difficult to provide a balanced diet that takes into account the nutritional value of each food item. Therefore, there is a need for a system that can efficiently manage food in the refrigerator.

[0117] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0118] In this invention, the server includes a camera for capturing images of objects in the refrigerator, an analysis unit for analyzing the images captured by the camera to recognize the objects, a suggestion unit for proposing currently available cooking procedures based on the recognized object information, a management unit for calculating the nutritional content of each object and managing a nutritionally balanced diet, and a deterioration prevention unit for estimating the expiration date of each object based on the object's acquisition date information and generating a notification for objects approaching their expiration date. This allows for efficient and automatic management of food in the refrigerator and encourages consumption at the appropriate time. Furthermore, balanced meal suggestions and nutritional management can support a healthy lifestyle.

[0119] The "photography means" is a device that is installed inside the refrigerator and takes images of food or objects.

[0120] The "analysis means" is a system that analyzes the image captured by the imaging means and recognizes the type of food or object.

[0121] The "suggestion means" is a mechanism that provides currently available cooking procedures to the user based on the information about the food or object recognized by the analysis means.

[0122] The "management means" is a system for calculating the nutritional content of foods and objects and managing a balanced diet.

[0123] The "deterioration prevention means" is a mechanism that estimates the expiration date of a food or object based on the acquisition date information, and generates a notification to the user when the expiration date approaches.

[0124] The system of the present invention efficiently manages food in a refrigerator, enabling health management, reducing the burden of housework, and preventing food waste. This system includes a photographing means, an analyzing means, a suggestion means, a management means, and a deterioration prevention means.

[0125] 1. Photography Method

[0126] The imaging means is a camera installed inside the refrigerator that periodically captures images of food and objects. For example, the camera takes a picture of the shelves inside the refrigerator every hour and sends the image data to a server. The specific hardware used is a high-resolution digital camera.

[0127] 2. Analysis method

[0128] The analysis means is software installed on the server that analyzes the images sent by the image capture means. This analysis uses an object recognition algorithm, such as TensorFlow or OpenCV. The analysis means automatically identifies and identifies objects in the image. For example, if there are tomatoes, milk, and eggs on a shelf in a refrigerator, the analysis means will recognize them and store them in the database as "3 tomatoes, 1 carton of milk, 6 eggs."

[0129] 3. Proposal method

[0130] When a user requests a dinner recipe, the suggestion means retrieves information about the food in the refrigerator from the database and searches the recipe database. For example, if there are tomatoes and milk in the refrigerator, the suggestion means selects a recipe for "tomato cream pasta" and notifies the user's smartphone. The specific software used is a database search algorithm using Python or Java.

[0131] 4. Control measures

[0132] The management means collects information about the meals the user has eaten and calculates the amount of nutrients the user has consumed for the week. This analyzes how much protein, carbohydrates, and fat the user has consumed and notifies the user of any nutrient deficiencies. For example, if the user's protein intake is low for the week, a report such as "You are lacking in protein. Please try a high-protein recipe" is generated and the user is notified.

[0133] 5. Measures to prevent deterioration

[0134] The deterioration prevention method estimates the expiration date based on the food's acquisition date and generates a notification for food that is approaching its expiration date. For example, if the expiration date of milk is approaching in three days, an alert will be generated and sent to the user's smartphone saying, "The milk's expiration date is approaching. Please consume it soon."

[0135] Example of a concrete example and prompt for the generative AI model:

[0136] When an image of the inside of a refrigerator is taken by the photographing means and the analyzing means recognizes that there are "3 tomatoes, 1 carton of milk, and 6 eggs," the specific processing will be as follows:

[0137] Example prompt sentence:

[0138] "Analyze images of the inside of the refrigerator and identify what's inside."

[0139] Input image: 3 tomatoes, 1 carton of milk, 6 eggs

[0140] What happens when a user asks for dinner recipes?

[0141] Example prompt sentence:

[0142] "I have some tomatoes and milk in the fridge. Can you suggest a recipe using them?"

[0143] This system will enable efficient management of food in the refrigerator and encourage consumption at the appropriate time. Furthermore, it will support a healthy lifestyle by suggesting balanced meals and managing nutrition.

[0144] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0145] Step 1:

[0146] Image acquisition

[0147] The server receives an image of the inside of the refrigerator using the photographing means.

[0148] Input: Image data taken inside the refrigerator

[0149] Output: Image data sent to the server

[0150] Specific operation: The camera built into the refrigerator takes pictures every hour and sends them to the server.

[0151] Step 2:

[0152] Image analysis

[0153] The server analyzes the received image using the image analysis means.

[0154] Input: Image data sent to the server

[0155] Output: Analyzed object information

[0156] Data processing and calculation: Using an object recognition algorithm (e.g., TensorFlow, OpenCV), identify the type and number of identified objects.

[0157] Specific operation: The server identifies objects in the image and obtains information such as "3 tomatoes, 1 carton of milk, and 6 eggs."

[0158] Step 3:

[0159] Data storage

[0160] The server stores the analysis results (object information) in a database.

[0161] Input: Parsed object information

[0162] Output: Object information stored in a database

[0163] Specific operation: The server writes information such as "3 tomatoes, 1 carton of milk, 6 eggs" into the database.

[0164] Step 4:

[0165] Recipe request reception

[0166] The server receives a recipe request from a user terminal.

[0167] Input: User request data

[0168] Output: Processing based on the request begins

[0169] Specific Action: A user requests a dinner recipe through a smartphone app.

[0170] Step 5:

[0171] Food information acquisition

[0172] The server obtains information about the food items in the refrigerator from the database.

[0173] Input: A query to the database

[0174] Output: Food information

[0175] Specific operation: The server retrieves food information from the database, such as "3 tomatoes, 1 carton of milk, 6 eggs."

[0176] Step 6:

[0177] Recipe Search

[0178] The server uses the suggestion means to search a recipe database based on the food information.

[0179] Input: Food information

[0180] Output: appropriate recipe information

[0181] Data processing and calculation: Matching food information with the recipe database to select the optimal recipe.

[0182] Specific operation: The server searches the recipe database and selects the recipe for "Tomato Cream Pasta."

[0183] Step 7:

[0184] Recipe Notifications

[0185] The server notifies the user terminal of the selected recipe.

[0186] Input: appropriate recipe information

[0187] Output: Recipe information sent to the user's device

[0188] Specific operation: The server sends the recipe for "Tomato Cream Pasta" to the user's smartphone, allowing the user to view it in the app.

[0189] Step 8:

[0190] Gathering dietary information

[0191] The server collects meal information from the user.

[0192] Input: User's food record data

[0193] Output: Meal data stored on the server

[0194] Specific operation: The user enters a meal record into the app, and the server receives it.

[0195] Step 9:

[0196] Nutrition Calculations

[0197] The server uses the management means to calculate the intake of nutrients from the dietary data for one week.

[0198] Input: Meal data

[0199] Output: Nutritional balance report

[0200] Data processing and calculation: Analyze dietary data and calculate protein and carbohydrate intake.

[0201] Specific operation: The server creates a report based on the user's weekly nutritional intake.

[0202] Step 10:

[0203] Nutrition Report Notification

[0204] The server notifies the user terminal of a report based on the nutrients that are lacking.

[0205] Input: Nutritional Balance Report

[0206] Output: Nutrition report sent to user device

[0207] Specific operation: The server sends a message to the user's smartphone saying, "You are lacking in protein. Please try a high-protein recipe."

[0208] Step 11:

[0209] Expiration date estimation

[0210] The server estimates the expiration date based on the purchase date information of the food product using the deterioration prevention means.

[0211] Input: Food purchase date information

[0212] Output: Estimated expiration date information

[0213] Data processing and calculation: The expiration date is calculated using the purchase date information and an algorithm.

[0214] Specific operation: The server estimates that the milk's expiration date is approaching in three days.

[0215] Step 12:

[0216] Expiration date alert generation

[0217] The server generates alerts for food items approaching their expiration date.

[0218] Input: Estimated expiration date information

[0219] Output: The generated alert message

[0220] Specific behavior: The server creates an alert message saying, "The milk's expiration date is approaching. Please consume it soon."

[0221] Step 13:

[0222] Expiration date alert notification

[0223] The server notifies the user terminal of the generated alert.

[0224] Input: The generated alert message

[0225] Output: The alert message sent to the user's terminal.

[0226] Specific operation: The server sends an alert to the user's smartphone, which the user can check in the app.

[0227] (Application example 1)

[0228] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0229] In modern society, food management, nutritional management, and food waste prevention at home and in brick-and-mortar stores are important issues. There is a need for a system that efficiently manages food not only at home but also in brick-and-mortar stores, and helps consumers purchase appropriate and balanced ingredients. The existence of such a system would greatly contribute to consumer health management and food waste reduction. Another important issue is reducing the financial burden on consumers by providing real-time sales information.

[0230] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0231] In this invention, the server includes a camera means for photographing food in the refrigerator, an image analysis means for analyzing images photographed by the camera means to recognize food, a recipe suggestion means for suggesting currently available recipes based on the recognized food information, a nutrition management means for calculating the nutrients of each food and managing a nutritionally balanced diet, a food waste prevention means for estimating expiration dates based on food purchase date information and generating an alert for food approaching its expiration date, a camera means for photographing food in a physical store, an image analysis means for analyzing images photographed by the camera means in the physical store to recognize food, a shopping list generation means for generating and notifying a shopping list based on the recognized food information in the physical store, a recipe suggestion means for suggesting and notifying recipes based on food available in the physical store, and a sale information notification means for recognizing and notifying sale information in the physical store. This enables efficient food management at home and in physical stores, providing nutritionally balanced meals, preventing food waste, and reducing economic burdens.

[0232] The "camera means" is a device for taking pictures of food in a refrigerator and in a physical store.

[0233] The "image analysis means" is a device for analyzing the image captured by the camera means and recognizing food.

[0234] The "recipe suggestion means" is a device for suggesting currently available recipes based on recognized food information.

[0235] The "nutritional management means" is a device that calculates the nutrients of each food and manages a nutritionally balanced diet.

[0236] The "food waste prevention device" is a device that estimates the expiration date based on the purchase date of food and generates an alert for food that is approaching its expiration date.

[0237] "Camera means in a physical store" refers to a device for photographing food in a physical store.

[0238] The "image analysis means in the physical store" is a device for recognizing food by analyzing images taken by a camera means in the physical store.

[0239] The "shopping list generation means" is a device for generating and notifying a shopping list based on the food information in the recognized physical store.

[0240] The "in-store recipe suggestion means" is a device for suggesting and notifying recipes based on food items that can be purchased in the store.

[0241] The "sale information notification means" is a device for recognizing and notifying sale information within a physical store.

[0242] The system of the present invention efficiently manages food in refrigerators and brick-and-mortar stores, and solves social issues such as preventive medicine, reducing the burden of housework, and preventing food waste. The system includes a camera, image analysis, recipe suggestion, nutritional management, and food waste prevention, as well as brick-and-mortar store-exclusive camera, image analysis, shopping list generation, in-store recipe suggestion, and sale information notification.

[0243] Hardware and software used:

[0244] Camera means: A camera installed in a refrigerator or in a physical store, which is a device for taking pictures of food.

[0245] Image analysis means: A device that analyzes captured images and recognizes the type of food, using common computer vision software such as OpenCV.

[0246] Recipe suggestion unit: A device that searches for possible recipe suggestions based on the recognized food information and notifies the user. It also works in conjunction with a recipe database.

[0247] Nutrition management tool: A device that calculates the nutritional value of food and supports balanced diet management. It uses an AI-based nutrition analysis tool.

[0248] Food waste prevention method: A device that estimates the expiration date of food from the purchase date information and generates an alert to notify the user when food is approaching its expiration date.

[0249] In-store camera means: A device installed in a physical store that takes pictures of food products being sold.

[0250] In-store image analysis means: A device that analyzes images taken by camera means in a physical store and recognizes the food being sold.

[0251] Shopping list generation means: A device that generates a shopping list based on the recognized food information in the physical store, along with the current refrigerator inventory, and notifies the user.

[0252] In-store recipe suggestion means: A device that suggests recipes based on food items available for purchase in a physical store and notifies the user.

[0253] Sale information notification means: A device that analyzes sale information in physical stores and notifies users.

[0254] Example of a system:

[0255] The system constantly monitors food inventory in the refrigerator and automatically generates a shopping list as needed. For example, if there is only one liter of milk, two tomatoes, and three peppers left in the refrigerator, the refrigerator's camera captures this information and analyzes it using image analysis. Based on this information, the recipe suggestion module suggests recipes using these foods to the user and automatically adds any missing ingredients to the shopping list.

[0256] When a user visits a physical store, a camera in the store takes a photo of the food on the shelves, and the image analysis means analyzes the information and notifies the user in real time. Furthermore, sale information is also analyzed, and economical sale items are suggested. In this way, users can comfortably and efficiently purchase the food they need.

[0257] Example prompt sentence:

[0258] "The current inventory in your refrigerator is:

[0259] 1 liter of milk

[0260] 2 tomatoes

[0261] 3 bell peppers

[0262] In-store specials include:

[0263] 2 tomatoes 100 yen

[0264] 20% off 1 pack of milk

[0265] Use this to generate recipe recommendations and shopping lists."

[0266] This invention allows for food management, nutritional balance and economical shopping at home and in physical stores.

[0267] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0268] Step 1:

[0269] The server periodically takes pictures of the food using a camera in the refrigerator. The input is the image obtained from the camera, and the output is an image file of the photographed food. This image file is then handed over to the next analysis process.

[0270] Step 2:

[0271] The server uses image analysis means to analyze the photographed food image and recognize the type and quantity of food. The input is the photographed image file, and the output is a list of the recognized food types and quantities. This list is stored in a database.

[0272] Step 3:

[0273] A user sends a recipe request using a smartphone. The input is the user's request, and the output is a list of suitable recipe candidates produced by the recipe suggestion means. This candidate list is then sent to the user's terminal.

[0274] Step 4:

[0275] The server uses the nutrition management tool to calculate the nutritional value of each food item based on the proposed recipe. The input is the ingredient information of the proposed recipe, and the output is a list of the nutrients contained in each recipe. This list is saved as the user's health management information and can be viewed by the user on their smartphone.

[0276] Step 5:

[0277] The server uses a food waste prevention method to estimate expiration dates based on food purchase date information and identify foods that are approaching their expiration date. The input is the food purchase date information, and the output is a list of foods that are approaching their expiration date. This list is notified to the user's device as an alert.

[0278] Step 6:

[0279] The server takes pictures of food items on shelves using a camera in the physical store and analyzes them using an image analysis means. The input is the image obtained from the camera in the physical store, and the output is a list of the types and quantities of recognized food items. This list is sent to the physical store management system.

[0280] Step 7:

[0281] The server compares the recognized food information in the physical store with the user's refrigerator inventory and generates and notifies the user of a shopping list containing the necessary foods. The input is the food information in the physical store and the refrigerator inventory information, and the output is the generated shopping list. This list is notified to the user's smartphone.

[0282] Step 8:

[0283] The server proposes recipes based on information about food items available in physical stores and notifies the user's device. The input is information about food items in the store and inventory in the refrigerator, and the output is a list of suggested recipes. This list is then sent to the user's smartphone.

[0284] Step 9:

[0285] The server analyzes sales information in physical stores and notifies the user. The input is sales information, and the output is a list of foods on sale. This list is sent to the user's smartphone to assist them in their shopping.

[0286] This series of processing steps allows users to efficiently manage and purchase food, enabling them to maintain a nutritionally balanced lifestyle.

[0287] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0288] The present invention is a system that has a camera means, image analysis means, recipe suggestion means, nutritional management means, food waste prevention means, and emotion engine means built into a refrigerator. This system efficiently manages food in the refrigerator, helping to maintain health, reduce the burden of housework, and reduce food waste, as well as suggesting recipes and managing nutrition based on the user's emotions.

[0289] Ingredient recognition processing

[0290] The camera means periodically takes pictures of the inside of the refrigerator and sends the image data to a server via the Internet. The server uses image analysis means to recognize food from the images and stores information about the food (e.g., 3 tomatoes, 1 carton of milk, 6 eggs) in a database.

[0291] Recipe suggestion processing

[0292] When a user requests a dinner recipe, the server retrieves the latest food information and searches for available recipes. The emotion engine also collects the user's emotion data (e.g., stress, joy). The server selects an appropriate recipe based on this emotion data and notifies the user's smartphone of a recipe such as "tomato cream pasta."

[0293] Nutritional balance management

[0294] The recipes selected by the user and information about the meals they actually made are recorded on the server. Based on this, the server calculates the nutrients contained in each meal and stores the data in a database. It analyzes nutritional balance based on the dietary history over a certain period (for example, one week) and identifies any nutrient deficiencies. At this time, it also takes into account data from the emotion engine and creates a detailed report, such as "On days with high stress, the required intake of vitamin C is low." The report is notified to the user via the app.

[0295] Preventing food waste

[0296] The server estimates the expiration date based on the food's purchase date information and generates an alert for food that is approaching its expiration date. For example, if the expiration date of milk is approaching in three days, the server creates an alert saying, "The milk's expiration date is approaching. Please consume it soon," and sends it to the user's smartphone. The content and timing of the alert can be adjusted to suit the user's emotional state using data from the emotion engine means.

[0297] Specific processing examples of emotion engine means

[0298] 1. While the user is preparing dinner, the emotion engine means analyzes facial expressions and tone of voice through the smartphone's camera and microphone.

[0299] 2. If the emotion engine means determines that the user is feeling stressed, the server will preferentially suggest recipes that use foods that have a relaxing effect.

[0300] 3. When a meal lacks balance, the server will include in the suggestions foods that contain nutrients that help relieve stress.

[0301] 4. When food in the refrigerator is nearing its expiration date, the server will quickly alert the user if they are feeling down, along with an encouraging message.

[0302] The above is a specific processing flow using the system of the present invention. This system optimizes food management in the refrigerator while taking into consideration the user's feelings, making it possible to manage health and reduce the burden of housework, as well as reducing food waste.

[0303] The processing flow will be explained below.

[0304] Ingredient recognition processing

[0305] Step 1:

[0306] The camera takes pictures of the inside of the refrigerator.

[0307] A camera means periodically takes images of the food shelves in the refrigerator.

[0308] Step 2:

[0309] The image data is sent to a server.

[0310] Image data captured by the camera means is transmitted to a server via the Internet.

[0311] Step 3:

[0312] The server analyzes the image data.

[0313] The server uses image analysis to recognize food items from the image, for example, distinguishing between tomatoes, milk, and eggs.

[0314] Step 4:

[0315] The recognized food information is stored in a database.

[0316] The server stores the recognition results (3 tomatoes, 1 carton of milk, and 6 eggs) in a database.

[0317] Recipe suggestion processing

[0318] Step 1:

[0319] A user requests recipe suggestions.

[0320] A user requests a dinner recipe on a smartphone app.

[0321] Step 2:

[0322] The server acquires information about the food in the refrigerator.

[0323] The server retrieves current food information from the database.

[0324] Step 3:

[0325] The emotion engine analyzes the user's emotions.

[0326] The emotion engine means analyzes the user's facial expressions and tone of voice through the smartphone's camera and microphone to determine the user's current emotional state.

[0327] Step 4:

[0328] The server searches the recipe database.

[0329] The server searches for appropriate recipes based on the food information in the refrigerator and the user's emotional data. For example, if the user is feeling stressed, it will select recipes using foods that have a relaxing effect.

[0330] Step 5:

[0331] The server creates recipe suggestions.

[0332] Servers create recipe suggestions that include required ingredients, cooking instructions, nutritional information, and emotionally sensitive comments.

[0333] Step 6:

[0334] Recipe suggestions are notified to the device.

[0335] The server notifies the user's smartphone of the recipe suggestions and displays detailed information within the app.

[0336] Nutritional balance management

[0337] Step 1:

[0338] Record the user's dietary history.

[0339] The app records the recipes the user has cooked and sends that information to the server.

[0340] Step 2:

[0341] The server calculates the nutrients.

[0342] The server calculates the nutrients contained in each meal and stores them in a database.

[0343] Step 3:

[0344] The server analyzes the nutritional balance.

[0345] The server analyzes the user's nutritional balance based on their dietary history over a certain period of time (for example, one week).

[0346] Step 4:

[0347] The server generates the report.

[0348] The server creates a detailed report based on the results of the nutritional balance analysis, taking into account the user's emotional data.

[0349] Step 5:

[0350] The report is sent to the device.

[0351] The server sends a detailed report to the user's smartphone, where the user can check the report using the app.

[0352] Preventing food waste

[0353] Step 1:

[0354] The server estimates the expiration date.

[0355] The server estimates the expiration date of each food item based on the purchase date information of the food item.

[0356] Step 2:

[0357] Save expiration date information in a database.

[0358] The server registers and manages the estimated expiration date in a database.

[0359] Step 3:

[0360] Generate an alert.

[0361] The server generates an alert message for food products that are approaching their expiration date.

[0362] Step 4:

[0363] Sends alerts to the device.

[0364] The server sends an alert to the user's smartphone, informing them that the expiration date is approaching.

[0365] Specific processing of emotion engine means

[0366] Step 1:

[0367] Collect user emotion data.

[0368] The emotion engine means uses the smartphone's camera and microphone to collect the user's facial expressions, tone of voice, and other emotion indicators.

[0369] Step 2:

[0370] Analyze emotion data.

[0371] Based on the data collected by the emotion engine means, the user's current emotional state is analyzed and determined.

[0372] Step 3:

[0373] The determined emotion is sent to the server.

[0374] The emotion engine means transmits the analysis results to the server.

[0375] Step 4:

[0376] Adjust recipes and alerts based on sentiment data.

[0377] The server takes emotional data into consideration and adjusts the content and timing of recipe suggestions and alerts to provide users with the most appropriate information.

[0378] The above is a specific processing flow using the system of the present invention. By combining it with an emotion engine means, recipe suggestions and food waste prevention that take into account the user's emotional state can be realized, further improving the user experience.

[0379] Example 2

[0380] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0381] Managing food in the refrigerator poses the problem of making it difficult to understand expiration dates and manage nutritional balance, making household diet management, which relies on this, cumbersome. Another issue is the inability to provide recipe suggestions or alert notifications that take into account the user's emotional state. The objective of this invention is to solve these issues and achieve health maintenance, reduce the burden of housework, and reduce food waste.

[0382] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0383] In this invention, the server includes an image acquisition means for capturing images of food in the refrigerator, an image analysis means for analyzing images captured by the image acquisition means to recognize the food, a recipe suggestion means for suggesting currently available recipes based on the recognized food information, a nutrition management means for calculating the nutrients contained in each meal and managing a nutritionally balanced diet, a food waste prevention means for estimating expiration dates based on food purchase date information and generating alerts for food approaching its expiration date, and an emotion analysis means for analyzing user emotion data and adjusting the content and timing of recipe suggestions and alerts based on the user's emotional state. This allows for optimized refrigerator food management while taking user emotions into consideration, thereby reducing the burden of health management and housework and reducing food waste.

[0384] "Image acquisition means" refers to a camera and associated equipment for photographing food in the refrigerator.

[0385] "Image analysis means" refers to algorithms or software for analyzing the images obtained by the image acquisition means and recognizing food.

[0386] "Recipe suggestion means" refers to a system or method for suggesting available recipes based on recognized food information.

[0387] "Nutrition management measures" refers to systems and methods for calculating the nutrients contained in each meal and managing nutritional balance.

[0388] "Food waste prevention measures" refers to systems and methods for estimating expiration dates based on food purchase date information and generating alerts for food that is approaching its expiration date.

[0389] "Emotion analysis means" refers to a system or method for analyzing a user's emotional data and adjusting the content and timing of recipe suggestions and alerts based on the user's emotional state.

[0390] "Notification means" refers to a system or method for sending recipe suggestions and alert notifications to a user terminal.

[0391] "Report generation means" refers to a system or method for creating a report based on the results of the nutritional balance analysis and presenting it to the user.

[0392] This system efficiently manages food in the refrigerator, helping maintain health, reduce the burden of housework, and reduce food waste. Its unique feature is its ability to suggest recipes and manage nutrition based on the user's emotions.

[0393] System configuration

[0394] The system includes the following main components:

[0395] 1. Image acquisition method:

[0396] A camera installed inside the refrigerator automatically captures images when the refrigerator is opened or closed.

[0397] 2. Image analysis methods:

[0398] The captured images are analyzed and food is recognized, specifically using Google Vision API and other image analysis algorithms.

[0399] 3. Recipe suggestion methods:

[0400] Based on the recognized food information, the system suggests currently available recipes. The recipe information is stored in a database, and the system searches for recipes that match the user's ingredients.

[0401] 4. Nutritional Management Measures:

[0402] Calculates the nutrients contained in each meal and manages a nutritionally balanced diet. Calculates the content of various nutrients based on a food composition database.

[0403] 5. Food waste prevention measures:

[0404] The system estimates the expiration date based on the purchase date of the food item and generates an alert for food items approaching their expiration date. The alert is sent to a smartphone.

[0405] 6. Emotion analysis means:

[0406] Analyzes user emotional data and adjusts the content and timing of recipe suggestions and alerts based on the user's emotional state. Sentiment analysis is performed using machine learning models such as TensorFlow.

[0407] Example of operation

[0408] Ingredient recognition processing

[0409] The camera unit takes an image of the inside of the refrigerator when it is opened, and the terminal transmits the image data to a server via the Internet.

[0410] The server uses image analysis to recognize the food items and stores their information in a database, for example, three tomatoes, one carton of milk, and six eggs.

[0411] Recipe suggestion processing

[0412] A user requests a dinner recipe through a smartphone app, and the device sends the request to a server.

[0413] The server selects appropriate recipes based on food information and the user's emotional data. Using an emotional engine, it suggests dishes such as "tomato cream pasta" that have a relaxing effect to users who are under stress.

[0414] Nutritional balance management

[0415] The user enters the recipe they selected and the information about the meal they actually made into the app, and the device sends that information to the server.

[0416] The server calculates the nutrients contained in each meal and stores them in a database. It analyzes nutritional balance based on a week's diet history and identifies nutrient deficiencies. A report containing the results is sent to the user via the app.

[0417] Preventing food waste

[0418] The server estimates expiration dates based on the purchase date information of food items registered in the database, and generates an alert for food items approaching their expiration date and notifies the user's smartphone.

[0419] The alerts are tailored in content and timing based on emotional data, and if the user is feeling down, they'll be accompanied by an encouraging message.

[0420] Specific examples

[0421] I took a picture of the food in my refrigerator. I can see three tomatoes, one carton of milk, and six eggs. Can you suggest a dinner recipe? The user is stressed.

[0422] This system optimizes food management in the refrigerator while taking into consideration the user's emotions, making it possible to manage health and reduce the burden of housework, as well as reduce food waste.

[0423] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0424] Ingredient recognition processing

[0425] Step 1:

[0426] The camera means periodically takes images of the inside of the refrigerator.

[0427] Input: Interior image of a refrigerator when it is opened.

[0428] Output: Image data of the inside of the refrigerator.

[0429] What it does: The camera takes a picture as soon as the refrigerator door opens, capturing a photo of the food inside.

[0430] Step 2:

[0431] The terminal transmits the captured image data to a server via the Internet.

[0432] Input: Image data of the inside of a refrigerator.

[0433] Output: Image data sent to the server.

[0434] Specific operation: The device uploads image data to the server via Wi-Fi. Once the upload is complete, the server confirms receipt of the image.

[0435] Step 3:

[0436] The server uses image analysis means to recognize the food from the image.

[0437] Input: Image data of the inside of the refrigerator sent to the server.

[0438] Output: Food information in the refrigerator (e.g., 3 tomatoes, 1 carton of milk, 6 eggs).

[0439] Specific operation: The server uses the Google Vision API to analyze objects in the image, recognize food, and compile information about it.

[0440] Step 4:

[0441] The server stores the recognized food information in a database.

[0442] Input: Food information in the refrigerator.

[0443] Output: Food information stored in a database.

[0444] Specific operation: The server connects to a MySQL database and accurately stores the recognized food information.

[0445] Recipe suggestion processing

[0446] Step 1:

[0447] A user requests a dinner recipe via a smartphone app.

[0448] Input: A recipe request from a user.

[0449] Output: Request information from the smartphone app.

[0450] What it does: A user uses the app and taps or voice commands to say, "Suggest recipes for dinner."

[0451] Step 2:

[0452] The terminal sends a recipe request to the server.

[0453] Input: Request information from the smartphone app.

[0454] Output: The recipe request sent to the server.

[0455] Specific operation: The device sends the request information to the server via the Internet. It notifies the user that the transmission was successful.

[0456] Step 3:

[0457] The server retrieves the latest food information from the database.

[0458] Input: The recipe request sent to the server.

[0459] Output: Food information retrieved from the database.

[0460] What happens: The server runs an SQL query to pull food information from the database.

[0461] Step 4:

[0462] The server uses emotion analysis means to collect emotion data of the user.

[0463] Input: User emotion data when requesting a recipe.

[0464] Output: Analyzed user emotion data (e.g. stress level).

[0465] Specific operation: The emotion analysis means analyzes facial expressions and tone of voice through the smartphone's camera and microphone, and determines emotions using a TensorFlow model.

[0466] Step 5:

[0467] The server selects the optimal recipe based on food information and emotional data.

[0468] Input: Retrieved food information and parsed emotion data.

[0469] Output: The best recipe information to be suggested to the user.

[0470] Specific operation: The server uses an algorithm to select the recipe that best suits the food inventory and the user's emotions, and assembles the recipe information.

[0471] Step 6:

[0472] The server notifies the user of the selected recipe on their smartphone.

[0473] Input: Selected recipe information.

[0474] Output: Notification to the user's smartphone.

[0475] Specific operation: The server uses push notifications to send recipe information to the user's smartphone app.

[0476] Nutritional balance management

[0477] Step 1:

[0478] The user enters the recipe they selected and information about the meal they actually made into the app.

[0479] Input: Meal information entered by the user.

[0480] Output: Meal information sent to the server.

[0481] What it does: The app is designed to make it easy to record your meal information by scanning barcodes or entering your voice.

[0482] Step 2:

[0483] The terminal transmits the information to the server.

[0484] Input: Meal information entered by the user into the app.

[0485] Output: Meal information sent to the server.

[0486] Specific operation: The device sends the input information to the server via the Internet and notifies the user when the transmission is complete.

[0487] Step 3:

[0488] The server calculates the nutrients contained in each meal based on the input dietary information.

[0489] Input: Meal information sent to the server.

[0490] Output: Calculated nutrient information for each meal.

[0491] Specific operation: The server uses Python's pandas library to calculate the nutrients for each food.

[0492] Step 4:

[0493] The server stores the calculated nutrient information in a database.

[0494] Input: Calculated nutrient information.

[0495] Output: Nutrient information stored in a database.

[0496] Specific operation: The server records the nutrient information in a database and manages it for each user.

[0497] Step 5:

[0498] The server analyzes nutritional balance based on a week's dietary history and identifies any nutrient deficiencies.

[0499] Input: A week's worth of dietary history recorded in a database.

[0500] Output: Nutritional balance analysis results.

[0501] Specific operation: Analyzes dietary history using Python's scikit-learn library and evaluates nutritional balance.

[0502] Step 6:

[0503] The server creates a nutritional balance report and notifies the user through the app.

[0504] Input: Nutritional balance analysis results.

[0505] Output: Report notification to user.

[0506] Specific operation: The server creates a report based on the analysis results and sends a push notification to the app.

[0507] Preventing food waste

[0508] Step 1:

[0509] The server estimates the expiration date based on the purchase date information of the food product.

[0510] Input: Food purchase date information recorded in the database.

[0511] Output: Estimated expiration date.

[0512] What it does: Calculates expiration dates using Python's datetime library.

[0513] Step 2:

[0514] The server generates alerts for food items approaching their expiration date.

[0515] Input: Estimated expiration date information.

[0516] Output: Alert information.

[0517] Specific behavior: The server generates an alert message based on the expiration date and adds it to the alert list.

[0518] Step 3:

[0519] The server notifies the user of the generated alert on their smartphone.

[0520] Input: Alert information.

[0521] Output: Notification to the user's smartphone.

[0522] What it does: Sends an alert message to the user's smartphone app using push notification.

[0523] Examples of application of emotion analysis methods

[0524] Step 1:

[0525] While the user prepares dinner, the emotion analysis means analyzes facial expressions and tone of voice through the smartphone's camera and microphone.

[0526] Input: User's facial expression and voice data.

[0527] Output: Parsed emotion data.

[0528] How it works: The emotion analyzer analyzes the user's facial expressions and tone of voice in real time to assess their stress and happiness levels.

[0529] Step 2:

[0530] If the emotion analysis means determines that the user is feeling stressed, the server preferentially suggests recipes that use foods that have a relaxing effect.

[0531] Input: Parsed emotion data.

[0532] Output: Relaxing recipe suggestions.

[0533] What it does: The server searches the database for relaxing recipes and updates the list of suggestions.

[0534] Step 3:

[0535] When your diet is lacking in balance, the server will suggest foods that contain nutrients that help relieve stress.

[0536] Input: Nutritional balance analysis results and emotional data.

[0537] Output: Nutrition-conscious recipe suggestions.

[0538] What it does: Based on the missing nutritional information, the server recommends recipes that include foods that help relieve stress.

[0539] Step 4:

[0540] When food in the refrigerator is nearing its expiration date, the server will quickly alert the user if they are feeling down, along with an encouraging message.

[0541] Input: Emotion data and expiration date information.

[0542] Output: Emotion-sensitive alert notification.

[0543] What it does: The server adjusts the content and timing of alert notifications based on emotion data and adds encouraging messages.

[0544] (Application example 2)

[0545] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0546] Conventional refrigerator systems require complicated management of food items in the refrigerator and expiration dates, and do not adequately achieve health management for users, reduce the burden of housework, or reduce food waste. Furthermore, they do not suggest or manage food based on the user's emotional state, and do not provide detailed support to users. This results in insufficient management of users' health and efficiency in housework, and often leads to food waste.

[0547] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0548] In this invention, the server includes a camera for taking pictures of food in the refrigerator, an image analysis unit for analyzing the images taken by the camera and recognizing the food, a suggestion unit for suggesting currently available recipes based on the recognized food information and the user's emotional information, a nutrition calculation unit for calculating the nutrients of each food and managing a nutritionally balanced diet, a food consumption management unit for estimating expiration dates based on food purchase dates and generating alerts for food items approaching their expiration date, and an emotional response unit for providing messages and suggestions based on the user's emotional state. This improves the efficiency of refrigerator food management, reduces the user's health management, eases housework burden, and reduces food waste. Furthermore, by suggesting recipes and generating alerts based on the user's emotional state, the server provides detailed support and improves user satisfaction.

[0549] 1. "Photographing means" refers to a photographing device placed to photograph food items in a refrigerator.

[0550] 2. "Image analysis means" refers to a means that has the function of analyzing the image acquired by the photographing means and recognizing food.

[0551] 3. "Suggestion means" refers to a means that has the function of suggesting currently available recipes based on recognized food information and user emotion information.

[0552] 4. "Nutrition calculation tool" refers to a tool that has the function of calculating the nutrients of each food and managing a nutritionally balanced diet.

[0553] 5. "Food expiration date management means" refers to a means that has the function of estimating the expiration date based on the purchase date information of food and generating an alert for food that is approaching its expiration date.

[0554] 6. "Emotion response means" refers to means that have the function of sending messages or suggestions according to the user's emotional state.

[0555] DETAILED DESCRIPTION OF THE INVENTION

[0556] The following describes an embodiment of the present invention: The system mainly operates using a camera means built into the refrigerator, a server, and a user's smartphone or other user terminal.

[0557] First, a camera installed inside the refrigerator periodically takes images of the inside of the refrigerator. This image data is sent to a server via the Internet. The server then uses image analysis means (such as image analysis software) to recognize food from the images and store the food information in a database. Specific image analysis uses the widely used OpenCV library, etc.

[0558] Next, when the user requests a dinner recipe, the server retrieves the latest food information and generates the optimal recipe using a suggestion method (recipe suggestion algorithm). This also takes the user's emotional information into consideration. Emotion recognition software such as EmotionEngine is used as an emotional response method to analyze the user's emotions. Based on the ingredient data and the user's emotional state, a recipe such as "tomato cream pasta" is sent to the user's smartphone.

[0559] The nutritional calculation tool calculates the nutritional value of each recipe based on the recipe selected by the user and the information on the meal they have actually made. This information is recorded on a server, and a report is created that analyzes the nutritional balance based on a week's worth of meal history and identifies any nutrient deficiencies. The report is sent to the user via the app.

[0560] Furthermore, the food consumption period management means estimates the expiration date based on the purchase date information of the food, and generates an alert for food that is approaching its expiration date. This alert is also notified to the user at an appropriate time using the emotion response means.

[0561] For example, for milk that is nearing its expiration date, a specific alert could be generated such as "Your milk is nearing its expiration date. Please consume it soon," accompanied by an uplifting message that reflects the user's emotional state.

[0562] Examples:

[0563] 1. Images taken by the refrigerator camera are periodically sent to the server.

[0564] 2. Using image analysis software (e.g., OpenCV), the food items in the refrigerator are recognized. For example, data such as "3 tomatoes, 1 carton of milk, and 6 eggs" is obtained.

[0565] 3. When a user requests a dinner recipe on their smartphone, the server uses the Emotion Engine to analyze the user's emotional information (e.g., stress level).

[0566] 4. Based on the user’s emotions and the food information in the refrigerator, the suggestion tool will suggest a recipe for “Tomato Cream Pasta” and notify the smartphone.

[0567] 5. The nutrition calculation means calculates the nutrients in the tomato cream pasta and creates a weekly nutritional balance report based on the user's dietary history.

[0568] 6. An alert will be generated for food products (such as milk) that are nearing their expiration date, and a notification will be sent to your smartphone saying, "Your milk is nearing its expiration date. Please consume it as soon as possible."

[0569] Example prompt sentence:

[0570] "Analyze images of the inside of the refrigerator and tell me the current food inventory."

[0571] "Please suggest a relaxing recipe based on my mood."

[0572] "Get alerts for foods that are nearing their expiration date."

[0573] This system will improve the efficiency of food management in the refrigerator, enabling users to manage their health, reduce household chores, and reduce food waste. It will also provide detailed support by suggesting recipes and generating alerts based on the user's emotional state, improving user satisfaction.

[0574] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0575] Specific operation steps:

[0576] Step 1:

[0577] The refrigerator's camera periodically captures images of the inside of the refrigerator, and the server receives this image data via the Internet.

[0578] Input: Image data of the inside of the refrigerator

[0579] Data processing: Image data is sent to a server via the Internet

[0580] Output: Image data received by the server

[0581] Step 2:

[0582] The server uses image analysis means (image analysis software) to analyze the received image data and identify the food items in the refrigerator. The information about the identified food items is stored in a database.

[0583] Input: Image data received by the server

[0584] Data analysis: Recognize food using image analysis software (e.g., OpenCV)

[0585] Output: Recognized food information (e.g., 3 tomatoes, 1 carton of milk, 6 eggs)

[0586] Step 3:

[0587] A user requests a dinner recipe via a device such as a smartphone. This request is sent to the server, which then retrieves the latest information about the food in the refrigerator based on the recognized food information.

[0588] Input: User recipe request

[0589] Data acquisition: Acquire the latest refrigerator food information from the database

[0590] Output: Latest information on food in the refrigerator

[0591] Step 4:

[0592] The server uses emotion response means (emotion recognition software) to acquire the user's emotion data from the camera and microphone of the smartphone or terminal in order to analyze the user's emotion information.

[0593] Input: User's camera video and audio data

[0594] Data analysis: Analyze emotions using emotion recognition software (e.g., EmotionEngine)

[0595] Output: User's emotional state (e.g., stress state)

[0596] Step 5:

[0597] The server uses a suggestion means (recipe suggestion algorithm) to select the optimal recipe based on the recognized food information and the user's emotional information, and notifies the user of the recipe to their smartphone.

[0598] Input: Recognized food information, user emotion information

[0599] Data processing: Applying recipe suggestion algorithm

[0600] Output: Notification of a suggested recipe (e.g., "Tomato Cream Pasta")

[0601] Step 6:

[0602] The server records the recipe information that the user selects and actually makes, and the server then calculates the nutritional content of each meal based on this information.

[0603] Input: User-generated meal information

[0604] Data Analysis: Nutrient Calculations

[0605] Output: Nutrition information for each meal

[0606] Step 7:

[0607] The server uses a nutritional calculation tool to analyze nutritional balance based on a week's worth of dietary history and creates a report identifying nutrient deficiencies, which is then sent to the user via the app.

[0608] Input: One week's worth of food history

[0609] Data analysis: Nutritional balance analysis

[0610] Output: Nutritional balance report creation and notification

[0611] Step 8:

[0612] The server uses the food consumption period management means to estimate the expiration date based on the purchase date information of the food, and generates an alert for food that is approaching its expiration date. The alert is sent to the smartphone with appropriate timing and content according to the user's emotional state using the emotional response means.

[0613] Input: Food purchase date information

[0614] Data analysis: Estimating expiration dates and considering emotional data

[0615] Output: Alert notification for food products that are nearing their expiration date (e.g. "The milk is nearing its expiration date. Please consume it as soon as possible.")

[0616] Example prompt sentence:

[0617] "Analyze images of the inside of the refrigerator and tell me the current food inventory."

[0618] "Please suggest a relaxing recipe based on my mood."

[0619] "Get alerts for foods that are nearing their expiration date."

[0620] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0621] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0622] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0623] [Second embodiment]

[0624] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0625] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0626] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0627] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0628] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0629] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0630] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0631] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0632] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0633] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0634] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0635] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0636] The system of the present invention has a built-in camera, image analysis, recipe suggestion, nutrition management, and food waste prevention means. By using this system, food in the refrigerator can be efficiently managed, and social issues such as preventive medicine, reducing the burden of housework, and preventing food waste can be addressed.

[0637] The camera means periodically photographs the food in the refrigerator and sends the image data to the server. The server analyzes the received images using the image analysis means and recognizes the food. The recognized food information is stored in a database, and the recipe suggestion means suggests appropriate recipes to the user based on this information. The nutrition management means also calculates the nutritional value of the food and performs nutritional management to provide a balanced diet.

[0638] Furthermore, the food loss prevention system estimates the expiration date based on the purchase date of the food, generates an alert for food that is approaching its expiration date, and notifies the user's device, thereby preventing food waste.

[0639] The specific processing contents of the program will be explained in natural language below.

[0640] Ingredient recognition processing

[0641] The server receives the image taken by the camera means and analyzes the image using the image analysis means. For example, if there are tomatoes, milk, and eggs on the shelves of a refrigerator, the server identifies these foods and stores them in the database as "3 tomatoes, 1 carton of milk, 6 eggs."

[0642] Recipe suggestion processing

[0643] When a user requests a dinner recipe, the server retrieves information about the food in the refrigerator from the database and searches the recipe database. For example, if there are tomatoes and milk in the refrigerator, it will suggest a recipe for tomato cream pasta. This suggestion is sent to the user's smartphone, and the user can check the cooking instructions and other ingredients needed through the app.

[0644] Nutritional balance management

[0645] To record the meals a user has eaten, the server collects information on suggested recipes and the meals actually cooked, allowing the system to calculate how much protein, carbohydrates, and fat a user has consumed in a week and notify them of any nutrient deficiencies.

[0646] For example, if a user's protein intake is low over the course of a week, the server will generate a report saying, "You're not getting enough protein. Try some high-protein recipes."

[0647] Preventing food waste

[0648] The server manages food expiration dates using information on food purchase dates and an expiration date estimation algorithm. For example, if milk's expiration date is approaching in three days, the server generates an alert saying, "The milk's expiration date is approaching. Please consume it soon," and notifies the user's smartphone.

[0649] As described above, the system of the present invention provides a specific method for efficiently managing food in a refrigerator, supporting health management, and reducing food waste.

[0650] The processing flow will be explained below.

[0651] Ingredient recognition processing

[0652] Step 1:

[0653] The camera takes pictures of the inside of the refrigerator.

[0654] The camera means periodically takes an image of each shelf in the refrigerator, and generates image data.

[0655] Step 2:

[0656] The image data is sent to a server.

[0657] The image data acquired by the camera means is transmitted to a server via the Internet.

[0658] Step 3:

[0659] The server analyzes the image data.

[0660] The server uses image analysis means to recognize foods from the image, for example, the server identifies tomatoes, milk, and eggs from the image.

[0661] Step 4:

[0662] The recognized food information is stored in a database.

[0663] The server stores the recognition result "3 tomatoes, 1 carton of milk, 6 eggs" in a database.

[0664] Recipe suggestion processing

[0665] Step 1:

[0666] A user requests recipe suggestions.

[0667] A user requests dinner recipe suggestions on a smartphone app.

[0668] Step 2:

[0669] The server acquires information about the food in the refrigerator.

[0670] The server retrieves the latest food information from the database.

[0671] Step 3:

[0672] The server searches the recipe database.

[0673] The server searches for available recipes based on the food information in the refrigerator. For example, if there are tomatoes and milk, it will extract the recipe for "Tomato Cream Pasta."

[0674] Step 4:

[0675] The server creates recipe suggestions.

[0676] The server creates a recipe suggestion that includes ingredients, cooking instructions, and nutritional information.

[0677] Step 5:

[0678] Recipe suggestions are notified to the device.

[0679] The server notifies the user of the recipe suggestions on their smartphone, which are then displayed within the app.

[0680] Nutritional balance management

[0681] Step 1:

[0682] Record the user's dietary history.

[0683] The app records the recipes the user has cooked and sends that information to the server.

[0684] Step 2:

[0685] The server calculates the nutrients.

[0686] The server calculates the nutrients contained in each meal and stores them in a database.

[0687] Step 3:

[0688] The server analyzes the nutritional balance.

[0689] The server analyzes nutritional balance based on dietary history over a certain period (e.g., one week) and identifies any nutrient deficiencies or excess intake.

[0690] Step 4:

[0691] Create a report.

[0692] The server generates a detailed report based on the results of the nutritional balance analysis, including, for example, "You are lacking in protein."

[0693] Step 5:

[0694] The report is sent to the device.

[0695] The server sends the created report to the user's smartphone, where the user can check the report contents through the app.

[0696] Preventing food waste

[0697] Step 1:

[0698] The server estimates the expiration date.

[0699] The server estimates the expiration date based on the purchase date of the food product. For example, it calculates the expiration date of milk.

[0700] Step 2:

[0701] Manage expiration date information.

[0702] The server manages the estimated expiration dates in a database and sets an alert flag when the expiration date approaches.

[0703] Step 3:

[0704] Generate an alert.

[0705] The server generates alerts for food products that are approaching their expiration date, for example, "Milk expires tomorrow."

[0706] Step 4:

[0707] Sends alerts to the device.

[0708] The server sends an alert to the user's smartphone, which is then displayed within the app.

[0709] Specific examples

[0710] 1. If there are tomatoes, milk, or eggs in the refrigerator, the camera means takes a picture of them and sends the image to the server.

[0711] 2. The server uses image analysis to recognize three tomatoes, one carton of milk, and six eggs, and stores them in a database.

[0712] 3. When a user requests a recipe for dinner, the server retrieves information about the food in the refrigerator, searches for available recipes, and suggests "Tomato Cream Pasta."

[0713] 4. When the user cooks a recipe and records their meal history in the app, the server calculates the nutrients and analyzes a week's worth of meal history to check nutritional balance.

[0714] 5. When the milk is nearing its expiration date, the server estimates the expiration date and generates an alert to notify the user.

[0715] The above is a specific processing flow using the system of the present invention.

[0716] Example 1

[0717] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0718] In modern society, there is a demand for both labor-saving and health management. However, food management is complicated and time-consuming. In particular, properly managing food in the refrigerator and consuming it while it is fresh is effective in preventing food waste, but manual management is difficult and not often implemented. It is also difficult to provide a balanced diet that takes into account the nutritional value of each food item. Therefore, there is a need for a system that can efficiently manage food in the refrigerator.

[0719] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0720] In this invention, the server includes a camera for capturing images of objects in the refrigerator, an analysis unit for analyzing the images captured by the camera to recognize the objects, a suggestion unit for proposing currently available cooking procedures based on the recognized object information, a management unit for calculating the nutritional content of each object and managing a nutritionally balanced diet, and a deterioration prevention unit for estimating the expiration date of each object based on the object's acquisition date information and generating a notification for objects approaching their expiration date. This allows for efficient and automatic management of food in the refrigerator and encourages consumption at the appropriate time. Furthermore, balanced meal suggestions and nutritional management can support a healthy lifestyle.

[0721] The "photography means" is a device that is installed inside the refrigerator and takes images of food or objects.

[0722] The "analysis means" is a system that analyzes the image captured by the imaging means and recognizes the type of food or object.

[0723] The "suggestion means" is a mechanism that provides currently available cooking procedures to the user based on the information about the food or object recognized by the analysis means.

[0724] The "management means" is a system for calculating the nutritional content of foods and objects and managing a balanced diet.

[0725] The "deterioration prevention means" is a mechanism that estimates the expiration date of a food or object based on the acquisition date information, and generates a notification to the user when the expiration date approaches.

[0726] The system of the present invention efficiently manages food in a refrigerator, enabling health management, reducing the burden of housework, and preventing food waste. This system includes a photographing means, an analyzing means, a suggestion means, a management means, and a deterioration prevention means.

[0727] 1. Photography Method

[0728] The imaging means is a camera installed inside the refrigerator that periodically captures images of food and objects. For example, the camera takes a picture of the shelves inside the refrigerator every hour and sends the image data to a server. The specific hardware used is a high-resolution digital camera.

[0729] 2. Analysis method

[0730] The analysis means is software installed on the server that analyzes the images sent by the image capture means. This analysis uses an object recognition algorithm, such as TensorFlow or OpenCV. The analysis means automatically identifies and identifies objects in the image. For example, if there are tomatoes, milk, and eggs on a shelf in a refrigerator, the analysis means will recognize them and store them in the database as "3 tomatoes, 1 carton of milk, 6 eggs."

[0731] 3. Proposal method

[0732] When a user requests a dinner recipe, the suggestion means retrieves information about the food in the refrigerator from the database and searches the recipe database. For example, if there are tomatoes and milk in the refrigerator, the suggestion means selects a recipe for "tomato cream pasta" and notifies the user's smartphone. The specific software used is a database search algorithm using Python or Java.

[0733] 4. Control measures

[0734] The management means collects information about the meals the user has eaten and calculates the amount of nutrients the user has consumed for the week. This analyzes how much protein, carbohydrates, and fat the user has consumed and notifies the user of any nutrient deficiencies. For example, if the user's protein intake is low for the week, a report such as "You are lacking in protein. Please try a high-protein recipe" is generated and the user is notified.

[0735] 5. Measures to prevent deterioration

[0736] The deterioration prevention method estimates the expiration date based on the food's acquisition date and generates a notification for food that is approaching its expiration date. For example, if the expiration date of milk is approaching in three days, an alert will be generated and sent to the user's smartphone saying, "The milk's expiration date is approaching. Please consume it soon."

[0737] Example of a concrete example and prompt for the generative AI model:

[0738] When an image of the inside of a refrigerator is taken by the photographing means and the analyzing means recognizes that there are "3 tomatoes, 1 carton of milk, and 6 eggs," the specific processing will be as follows:

[0739] Example prompt sentence:

[0740] "Analyze images of the inside of the refrigerator and identify what's inside."

[0741] Input image: 3 tomatoes, 1 carton of milk, 6 eggs

[0742] What happens when a user asks for dinner recipes?

[0743] Example prompt sentence:

[0744] "I have some tomatoes and milk in the fridge. Can you suggest a recipe using them?"

[0745] This system will enable efficient management of food in the refrigerator and encourage consumption at the appropriate time. Furthermore, it will support a healthy lifestyle by suggesting balanced meals and managing nutrition.

[0746] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0747] Step 1:

[0748] Image acquisition

[0749] The server receives an image of the inside of the refrigerator using the photographing means.

[0750] Input: Image data taken inside the refrigerator

[0751] Output: Image data sent to the server

[0752] Specific operation: The camera built into the refrigerator takes pictures every hour and sends them to the server.

[0753] Step 2:

[0754] Image analysis

[0755] The server analyzes the received image using the image analysis means.

[0756] Input: Image data sent to the server

[0757] Output: Analyzed object information

[0758] Data processing and calculation: Using an object recognition algorithm (e.g., TensorFlow, OpenCV), identify the type and number of identified objects.

[0759] Specific operation: The server identifies objects in the image and obtains information such as "3 tomatoes, 1 carton of milk, and 6 eggs."

[0760] Step 3:

[0761] Data storage

[0762] The server stores the analysis results (object information) in a database.

[0763] Input: Parsed object information

[0764] Output: Object information stored in a database

[0765] Specific operation: The server writes information such as "3 tomatoes, 1 carton of milk, 6 eggs" into the database.

[0766] Step 4:

[0767] Recipe request reception

[0768] The server receives a recipe request from a user terminal.

[0769] Input: User request data

[0770] Output: Processing based on the request begins

[0771] Specific Action: A user requests a dinner recipe through a smartphone app.

[0772] Step 5:

[0773] Food information acquisition

[0774] The server obtains information about the food items in the refrigerator from the database.

[0775] Input: A query to the database

[0776] Output: Food information

[0777] Specific operation: The server retrieves food information from the database, such as "3 tomatoes, 1 carton of milk, 6 eggs."

[0778] Step 6:

[0779] Recipe Search

[0780] The server uses the suggestion means to search a recipe database based on the food information.

[0781] Input: Food information

[0782] Output: appropriate recipe information

[0783] Data processing and calculation: Matching food information with the recipe database to select the optimal recipe.

[0784] Specific operation: The server searches the recipe database and selects the recipe for "Tomato Cream Pasta."

[0785] Step 7:

[0786] Recipe Notifications

[0787] The server notifies the user terminal of the selected recipe.

[0788] Input: appropriate recipe information

[0789] Output: Recipe information sent to the user's device

[0790] Specific operation: The server sends the recipe for "Tomato Cream Pasta" to the user's smartphone, allowing the user to view it in the app.

[0791] Step 8:

[0792] Gathering dietary information

[0793] The server collects meal information from the user.

[0794] Input: User's food record data

[0795] Output: Meal data stored on the server

[0796] Specific operation: The user enters a meal record into the app, and the server receives it.

[0797] Step 9:

[0798] Nutrition Calculations

[0799] The server uses the management means to calculate the intake of nutrients from the dietary data for one week.

[0800] Input: Meal data

[0801] Output: Nutritional balance report

[0802] Data processing and calculation: Analyze dietary data and calculate protein and carbohydrate intake.

[0803] Specific operation: The server creates a report based on the user's weekly nutritional intake.

[0804] Step 10:

[0805] Nutrition Report Notification

[0806] The server notifies the user terminal of a report based on the nutrients that are lacking.

[0807] Input: Nutritional Balance Report

[0808] Output: Nutrition report sent to user device

[0809] Specific operation: The server sends a message to the user's smartphone saying, "You are lacking in protein. Please try a high-protein recipe."

[0810] Step 11:

[0811] Expiration date estimation

[0812] The server estimates the expiration date based on the purchase date information of the food product using the deterioration prevention means.

[0813] Input: Food purchase date information

[0814] Output: Estimated expiration date information

[0815] Data processing and calculation: The expiration date is calculated using the purchase date information and an algorithm.

[0816] Specific operation: The server estimates that the milk's expiration date is approaching in three days.

[0817] Step 12:

[0818] Expiration date alert generation

[0819] The server generates alerts for food items approaching their expiration date.

[0820] Input: Estimated expiration date information

[0821] Output: The generated alert message

[0822] Specific behavior: The server creates an alert message saying, "The milk's expiration date is approaching. Please consume it soon."

[0823] Step 13:

[0824] Expiration date alert notification

[0825] The server notifies the user terminal of the generated alert.

[0826] Input: The generated alert message

[0827] Output: The alert message sent to the user's terminal.

[0828] Specific operation: The server sends an alert to the user's smartphone, which the user can check in the app.

[0829] (Application example 1)

[0830] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0831] In modern society, food management, nutritional management, and food waste prevention at home and in brick-and-mortar stores are important issues. There is a need for a system that efficiently manages food not only at home but also in brick-and-mortar stores, and helps consumers purchase appropriate and balanced ingredients. The existence of such a system would greatly contribute to consumer health management and food waste reduction. Another important issue is reducing the financial burden on consumers by providing real-time sales information.

[0832] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0833] In this invention, the server includes a camera means for photographing food in the refrigerator, an image analysis means for analyzing images photographed by the camera means to recognize food, a recipe suggestion means for suggesting currently available recipes based on the recognized food information, a nutrition management means for calculating the nutrients of each food and managing a nutritionally balanced diet, a food waste prevention means for estimating expiration dates based on food purchase date information and generating an alert for food approaching its expiration date, a camera means for photographing food in a physical store, an image analysis means for analyzing images photographed by the camera means in the physical store to recognize food, a shopping list generation means for generating and notifying a shopping list based on the recognized food information in the physical store, a recipe suggestion means for suggesting and notifying recipes based on food available in the physical store, and a sale information notification means for recognizing and notifying sale information in the physical store. This enables efficient food management at home and in physical stores, providing nutritionally balanced meals, preventing food waste, and reducing economic burdens.

[0834] The "camera means" is a device for taking pictures of food in a refrigerator and in a physical store.

[0835] The "image analysis means" is a device for analyzing the image captured by the camera means and recognizing food.

[0836] The "recipe suggestion means" is a device for suggesting currently available recipes based on recognized food information.

[0837] The "nutritional management means" is a device that calculates the nutrients of each food and manages a nutritionally balanced diet.

[0838] The "food waste prevention device" is a device that estimates the expiration date based on the purchase date of food and generates an alert for food that is approaching its expiration date.

[0839] "Camera means in a physical store" refers to a device for photographing food in a physical store.

[0840] The "image analysis means in the physical store" is a device for recognizing food by analyzing images taken by a camera means in the physical store.

[0841] The "shopping list generation means" is a device for generating and notifying a shopping list based on the food information in the recognized physical store.

[0842] The "in-store recipe suggestion means" is a device for suggesting and notifying recipes based on food items that can be purchased in the store.

[0843] The "sale information notification means" is a device for recognizing and notifying sale information within a physical store.

[0844] The system of the present invention efficiently manages food in refrigerators and brick-and-mortar stores, and solves social issues such as preventive medicine, reducing the burden of housework, and preventing food waste. The system includes a camera, image analysis, recipe suggestion, nutritional management, and food waste prevention, as well as brick-and-mortar store-exclusive camera, image analysis, shopping list generation, in-store recipe suggestion, and sale information notification.

[0845] Hardware and software used:

[0846] Camera means: A camera installed in a refrigerator or in a physical store, which is a device for taking pictures of food.

[0847] Image analysis means: A device that analyzes captured images and recognizes the type of food, using common computer vision software such as OpenCV.

[0848] Recipe suggestion unit: A device that searches for possible recipe suggestions based on the recognized food information and notifies the user. It also works in conjunction with a recipe database.

[0849] Nutrition management tool: A device that calculates the nutritional value of food and supports balanced diet management. It uses an AI-based nutrition analysis tool.

[0850] Food waste prevention method: A device that estimates the expiration date of food from the purchase date information and generates an alert to notify the user when food is approaching its expiration date.

[0851] In-store camera means: A device installed in a physical store that takes pictures of food products being sold.

[0852] In-store image analysis means: A device that analyzes images taken by camera means in a physical store and recognizes the food being sold.

[0853] Shopping list generation means: A device that generates a shopping list based on the recognized food information in the physical store, along with the current refrigerator inventory, and notifies the user.

[0854] In-store recipe suggestion means: A device that suggests recipes based on food items available for purchase in a physical store and notifies the user.

[0855] Sale information notification means: A device that analyzes sale information in physical stores and notifies users.

[0856] Example of a system:

[0857] The system constantly monitors food inventory in the refrigerator and automatically generates a shopping list as needed. For example, if there is only one liter of milk, two tomatoes, and three peppers left in the refrigerator, the refrigerator's camera captures this information and analyzes it using image analysis. Based on this information, the recipe suggestion module suggests recipes using these foods to the user and automatically adds any missing ingredients to the shopping list.

[0858] When a user visits a physical store, a camera in the store takes a photo of the food on the shelves, and the image analysis means analyzes the information and notifies the user in real time. Furthermore, sale information is also analyzed, and economical sale items are suggested. In this way, users can comfortably and efficiently purchase the food they need.

[0859] Example prompt sentence:

[0860] "The current inventory in your refrigerator is:

[0861] 1 liter of milk

[0862] 2 tomatoes

[0863] 3 bell peppers

[0864] In-store specials include:

[0865] 2 tomatoes 100 yen

[0866] 20% off 1 pack of milk

[0867] Use this to generate recipe recommendations and shopping lists."

[0868] This invention allows for food management, nutritional balance and economical shopping at home and in physical stores.

[0869] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0870] Step 1:

[0871] The server periodically takes pictures of the food using a camera in the refrigerator. The input is the image obtained from the camera, and the output is an image file of the photographed food. This image file is then handed over to the next analysis process.

[0872] Step 2:

[0873] The server uses image analysis means to analyze the photographed food image and recognize the type and quantity of food. The input is the photographed image file, and the output is a list of the recognized food types and quantities. This list is stored in a database.

[0874] Step 3:

[0875] A user sends a recipe request using a smartphone. The input is the user's request, and the output is a list of suitable recipe candidates produced by the recipe suggestion means. This candidate list is then sent to the user's terminal.

[0876] Step 4:

[0877] The server uses the nutrition management tool to calculate the nutritional value of each food item based on the proposed recipe. The input is the ingredient information of the proposed recipe, and the output is a list of the nutrients contained in each recipe. This list is saved as the user's health management information and can be viewed by the user on their smartphone.

[0878] Step 5:

[0879] The server uses a food waste prevention method to estimate expiration dates based on food purchase date information and identify foods that are approaching their expiration date. The input is the food purchase date information, and the output is a list of foods that are approaching their expiration date. This list is notified to the user's device as an alert.

[0880] Step 6:

[0881] The server takes pictures of food items on shelves using a camera in the physical store and analyzes them using an image analysis means. The input is the image obtained from the camera in the physical store, and the output is a list of the types and quantities of recognized food items. This list is sent to the physical store management system.

[0882] Step 7:

[0883] The server compares the recognized food information in the physical store with the user's refrigerator inventory and generates and notifies the user of a shopping list containing the necessary foods. The input is the food information in the physical store and the refrigerator inventory information, and the output is the generated shopping list. This list is notified to the user's smartphone.

[0884] Step 8:

[0885] The server proposes recipes based on information about food items available in physical stores and notifies the user's device. The input is information about food items in the store and inventory in the refrigerator, and the output is a list of suggested recipes. This list is then sent to the user's smartphone.

[0886] Step 9:

[0887] The server analyzes sales information in physical stores and notifies the user. The input is sales information, and the output is a list of foods on sale. This list is sent to the user's smartphone to assist them in their shopping.

[0888] This series of processing steps allows users to efficiently manage and purchase food, enabling them to maintain a nutritionally balanced lifestyle.

[0889] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0890] The present invention is a system that has a camera means, image analysis means, recipe suggestion means, nutritional management means, food waste prevention means, and emotion engine means built into a refrigerator. This system efficiently manages food in the refrigerator, helping to maintain health, reduce the burden of housework, and reduce food waste, as well as suggesting recipes and managing nutrition based on the user's emotions.

[0891] Ingredient recognition processing

[0892] The camera means periodically takes pictures of the inside of the refrigerator and sends the image data to a server via the Internet. The server uses image analysis means to recognize food from the images and stores information about the food (e.g., 3 tomatoes, 1 carton of milk, 6 eggs) in a database.

[0893] Recipe suggestion processing

[0894] When a user requests a dinner recipe, the server retrieves the latest food information and searches for available recipes. The emotion engine also collects the user's emotion data (e.g., stress, joy). The server selects an appropriate recipe based on this emotion data and notifies the user's smartphone of a recipe such as "tomato cream pasta."

[0895] Nutritional balance management

[0896] The recipes selected by the user and information about the meals they actually made are recorded on the server. Based on this, the server calculates the nutrients contained in each meal and stores the data in a database. It analyzes nutritional balance based on the dietary history over a certain period (for example, one week) and identifies any nutrient deficiencies. At this time, it also takes into account data from the emotion engine and creates a detailed report, such as "On days with high stress, the required intake of vitamin C is low." The report is notified to the user via the app.

[0897] Preventing food waste

[0898] The server estimates the expiration date based on the food's purchase date information and generates an alert for food that is approaching its expiration date. For example, if the expiration date of milk is approaching in three days, the server creates an alert saying, "The milk's expiration date is approaching. Please consume it soon," and sends it to the user's smartphone. The content and timing of the alert can be adjusted to suit the user's emotional state using data from the emotion engine means.

[0899] Specific processing examples of emotion engine means

[0900] 1. While the user is preparing dinner, the emotion engine means analyzes facial expressions and tone of voice through the smartphone's camera and microphone.

[0901] 2. If the emotion engine means determines that the user is feeling stressed, the server will preferentially suggest recipes that use foods that have a relaxing effect.

[0902] 3. When a meal lacks balance, the server will include in the suggestions foods that contain nutrients that help relieve stress.

[0903] 4. When food in the refrigerator is nearing its expiration date, the server will quickly alert the user if they are feeling down, along with an encouraging message.

[0904] The above is a specific processing flow using the system of the present invention. This system optimizes food management in the refrigerator while taking into consideration the user's feelings, making it possible to manage health and reduce the burden of housework, as well as reducing food waste.

[0905] The processing flow will be explained below.

[0906] Ingredient recognition processing

[0907] Step 1:

[0908] The camera takes pictures of the inside of the refrigerator.

[0909] A camera means periodically takes images of the food shelves in the refrigerator.

[0910] Step 2:

[0911] The image data is sent to a server.

[0912] Image data captured by the camera means is transmitted to a server via the Internet.

[0913] Step 3:

[0914] The server analyzes the image data.

[0915] The server uses image analysis to recognize food items from the image, for example, distinguishing between tomatoes, milk, and eggs.

[0916] Step 4:

[0917] The recognized food information is stored in a database.

[0918] The server stores the recognition results (3 tomatoes, 1 carton of milk, and 6 eggs) in a database.

[0919] Recipe suggestion processing

[0920] Step 1:

[0921] A user requests recipe suggestions.

[0922] A user requests a dinner recipe on a smartphone app.

[0923] Step 2:

[0924] The server acquires information about the food in the refrigerator.

[0925] The server retrieves current food information from the database.

[0926] Step 3:

[0927] The emotion engine analyzes the user's emotions.

[0928] The emotion engine means analyzes the user's facial expressions and tone of voice through the smartphone's camera and microphone to determine the user's current emotional state.

[0929] Step 4:

[0930] The server searches the recipe database.

[0931] The server searches for appropriate recipes based on the food information in the refrigerator and the user's emotional data. For example, if the user is feeling stressed, it will select recipes using foods that have a relaxing effect.

[0932] Step 5:

[0933] The server creates recipe suggestions.

[0934] Servers create recipe suggestions that include required ingredients, cooking instructions, nutritional information, and emotionally sensitive comments.

[0935] Step 6:

[0936] Recipe suggestions are notified to the device.

[0937] The server notifies the user's smartphone of the recipe suggestions and displays detailed information within the app.

[0938] Nutritional balance management

[0939] Step 1:

[0940] Record the user's dietary history.

[0941] The app records the recipes the user has cooked and sends that information to the server.

[0942] Step 2:

[0943] The server calculates the nutrients.

[0944] The server calculates the nutrients contained in each meal and stores them in a database.

[0945] Step 3:

[0946] The server analyzes the nutritional balance.

[0947] The server analyzes the user's nutritional balance based on their dietary history over a certain period of time (for example, one week).

[0948] Step 4:

[0949] The server generates the report.

[0950] The server creates a detailed report based on the results of the nutritional balance analysis, taking into account the user's emotional data.

[0951] Step 5:

[0952] The report is sent to the device.

[0953] The server sends a detailed report to the user's smartphone, where the user can check the report using the app.

[0954] Preventing food waste

[0955] Step 1:

[0956] The server estimates the expiration date.

[0957] The server estimates the expiration date of each food item based on the purchase date information of the food item.

[0958] Step 2:

[0959] Save expiration date information in a database.

[0960] The server registers and manages the estimated expiration date in a database.

[0961] Step 3:

[0962] Generate an alert.

[0963] The server generates an alert message for food products that are approaching their expiration date.

[0964] Step 4:

[0965] Sends alerts to the device.

[0966] The server sends an alert to the user's smartphone, informing them that the expiration date is approaching.

[0967] Specific processing of emotion engine means

[0968] Step 1:

[0969] Collect user emotion data.

[0970] The emotion engine means uses the smartphone's camera and microphone to collect the user's facial expressions, tone of voice, and other emotion indicators.

[0971] Step 2:

[0972] Analyze emotion data.

[0973] Based on the data collected by the emotion engine means, the user's current emotional state is analyzed and determined.

[0974] Step 3:

[0975] The determined emotion is sent to the server.

[0976] The emotion engine means transmits the analysis results to the server.

[0977] Step 4:

[0978] Adjust recipes and alerts based on sentiment data.

[0979] The server takes emotional data into consideration and adjusts the content and timing of recipe suggestions and alerts to provide users with the most appropriate information.

[0980] The above is a specific processing flow using the system of the present invention. By combining it with an emotion engine means, recipe suggestions and food waste prevention that take into account the user's emotional state can be realized, further improving the user experience.

[0981] Example 2

[0982] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0983] Managing food in the refrigerator poses the problem of making it difficult to understand expiration dates and manage nutritional balance, making household diet management, which relies on this, cumbersome. Another issue is the inability to provide recipe suggestions or alert notifications that take into account the user's emotional state. The objective of this invention is to solve these issues and achieve health maintenance, reduce the burden of housework, and reduce food waste.

[0984] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0985] In this invention, the server includes an image acquisition means for capturing images of food in the refrigerator, an image analysis means for analyzing images captured by the image acquisition means to recognize the food, a recipe suggestion means for suggesting currently available recipes based on the recognized food information, a nutrition management means for calculating the nutrients contained in each meal and managing a nutritionally balanced diet, a food waste prevention means for estimating expiration dates based on food purchase date information and generating alerts for food approaching its expiration date, and an emotion analysis means for analyzing user emotion data and adjusting the content and timing of recipe suggestions and alerts based on the user's emotional state. This allows for optimized refrigerator food management while taking user emotions into consideration, thereby reducing the burden of health management and housework and reducing food waste.

[0986] "Image acquisition means" refers to a camera and associated equipment for photographing food in the refrigerator.

[0987] "Image analysis means" refers to algorithms or software for analyzing the images obtained by the image acquisition means and recognizing food.

[0988] "Recipe suggestion means" refers to a system or method for suggesting available recipes based on recognized food information.

[0989] "Nutrition management measures" refers to systems and methods for calculating the nutrients contained in each meal and managing nutritional balance.

[0990] "Food waste prevention measures" refers to systems and methods for estimating expiration dates based on food purchase date information and generating alerts for food that is approaching its expiration date.

[0991] "Emotion analysis means" refers to a system or method for analyzing a user's emotional data and adjusting the content and timing of recipe suggestions and alerts based on the user's emotional state.

[0992] "Notification means" refers to a system or method for sending recipe suggestions and alert notifications to a user terminal.

[0993] "Report generation means" refers to a system or method for creating a report based on the results of the nutritional balance analysis and presenting it to the user.

[0994] This system efficiently manages food in the refrigerator, helping maintain health, reduce the burden of housework, and reduce food waste. Its unique feature is its ability to suggest recipes and manage nutrition based on the user's emotions.

[0995] System configuration

[0996] The system includes the following main components:

[0997] 1. Image acquisition method:

[0998] A camera installed inside the refrigerator automatically captures images when the refrigerator is opened or closed.

[0999] 2. Image analysis methods:

[1000] The captured images are analyzed and food is recognized, specifically using Google Vision API and other image analysis algorithms.

[1001] 3. Recipe suggestion methods:

[1002] Based on the recognized food information, the system suggests currently available recipes. The recipe information is stored in a database, and the system searches for recipes that match the user's ingredients.

[1003] 4. Nutritional Management Measures:

[1004] Calculates the nutrients contained in each meal and manages a nutritionally balanced diet. Calculates the content of various nutrients based on a food composition database.

[1005] 5. Food waste prevention measures:

[1006] The system estimates the expiration date based on the purchase date of the food item and generates an alert for food items approaching their expiration date. The alert is sent to a smartphone.

[1007] 6. Emotion analysis means:

[1008] Analyzes user emotional data and adjusts the content and timing of recipe suggestions and alerts based on the user's emotional state. Sentiment analysis is performed using machine learning models such as TensorFlow.

[1009] Example of operation

[1010] Ingredient recognition processing

[1011] The camera unit takes an image of the inside of the refrigerator when it is opened, and the terminal transmits the image data to a server via the Internet.

[1012] The server uses image analysis to recognize the food items and stores their information in a database, for example, three tomatoes, one carton of milk, and six eggs.

[1013] Recipe suggestion processing

[1014] A user requests a dinner recipe through a smartphone app, and the device sends the request to a server.

[1015] The server selects appropriate recipes based on food information and the user's emotional data. Using an emotional engine, it suggests dishes such as "tomato cream pasta" that have a relaxing effect to users who are under stress.

[1016] Nutritional balance management

[1017] The user enters the recipe they selected and the information about the meal they actually made into the app, and the device sends that information to the server.

[1018] The server calculates the nutrients contained in each meal and stores them in a database. It analyzes nutritional balance based on a week's diet history and identifies nutrient deficiencies. A report containing the results is sent to the user via the app.

[1019] Preventing food waste

[1020] The server estimates expiration dates based on the purchase date information of food items registered in the database, and generates an alert for food items approaching their expiration date and notifies the user's smartphone.

[1021] The alerts are tailored in content and timing based on emotional data, and if the user is feeling down, they'll be accompanied by an encouraging message.

[1022] Specific examples

[1023] I took a picture of the food in my refrigerator. I can see three tomatoes, one carton of milk, and six eggs. Can you suggest a dinner recipe? The user is stressed.

[1024] This system optimizes food management in the refrigerator while taking into consideration the user's emotions, making it possible to manage health and reduce the burden of housework, as well as reduce food waste.

[1025] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1026] Ingredient recognition processing

[1027] Step 1:

[1028] The camera means periodically takes images of the inside of the refrigerator.

[1029] Input: Interior image of a refrigerator when it is opened.

[1030] Output: Image data of the inside of the refrigerator.

[1031] What it does: The camera takes a picture as soon as the refrigerator door opens, capturing a photo of the food inside.

[1032] Step 2:

[1033] The terminal transmits the captured image data to a server via the Internet.

[1034] Input: Image data of the inside of a refrigerator.

[1035] Output: Image data sent to the server.

[1036] Specific operation: The device uploads image data to the server via Wi-Fi. Once the upload is complete, the server confirms receipt of the image.

[1037] Step 3:

[1038] The server uses image analysis means to recognize the food from the image.

[1039] Input: Image data of the inside of the refrigerator sent to the server.

[1040] Output: Food information in the refrigerator (e.g., 3 tomatoes, 1 carton of milk, 6 eggs).

[1041] Specific operation: The server uses the Google Vision API to analyze objects in the image, recognize food, and compile information about it.

[1042] Step 4:

[1043] The server stores the recognized food information in a database.

[1044] Input: Food information in the refrigerator.

[1045] Output: Food information stored in a database.

[1046] Specific operation: The server connects to a MySQL database and accurately stores the recognized food information.

[1047] Recipe suggestion processing

[1048] Step 1:

[1049] A user requests a dinner recipe via a smartphone app.

[1050] Input: A recipe request from a user.

[1051] Output: Request information from the smartphone app.

[1052] What it does: A user uses the app and taps or voice commands to say, "Suggest recipes for dinner."

[1053] Step 2:

[1054] The terminal sends a recipe request to the server.

[1055] Input: Request information from the smartphone app.

[1056] Output: The recipe request sent to the server.

[1057] Specific operation: The device sends the request information to the server via the Internet. It notifies the user that the transmission was successful.

[1058] Step 3:

[1059] The server retrieves the latest food information from the database.

[1060] Input: The recipe request sent to the server.

[1061] Output: Food information retrieved from the database.

[1062] What happens: The server runs an SQL query to pull food information from the database.

[1063] Step 4:

[1064] The server uses emotion analysis means to collect emotion data of the user.

[1065] Input: User emotion data when requesting a recipe.

[1066] Output: Analyzed user emotion data (e.g. stress level).

[1067] Specific operation: The emotion analysis means analyzes facial expressions and tone of voice through the smartphone's camera and microphone, and determines emotions using a TensorFlow model.

[1068] Step 5:

[1069] The server selects the optimal recipe based on food information and emotional data.

[1070] Input: Retrieved food information and parsed emotion data.

[1071] Output: The best recipe information to be suggested to the user.

[1072] Specific operation: The server uses an algorithm to select the recipe that best suits the food inventory and the user's emotions, and assembles the recipe information.

[1073] Step 6:

[1074] The server notifies the user of the selected recipe on their smartphone.

[1075] Input: Selected recipe information.

[1076] Output: Notification to the user's smartphone.

[1077] Specific operation: The server uses push notifications to send recipe information to the user's smartphone app.

[1078] Nutritional balance management

[1079] Step 1:

[1080] The user enters the recipe they selected and information about the meal they actually made into the app.

[1081] Input: Meal information entered by the user.

[1082] Output: Meal information sent to the server.

[1083] What it does: The app is designed to make it easy to record your meal information by scanning barcodes or entering your voice.

[1084] Step 2:

[1085] The terminal transmits the information to the server.

[1086] Input: Meal information entered by the user into the app.

[1087] Output: Meal information sent to the server.

[1088] Specific operation: The device sends the input information to the server via the Internet and notifies the user when the transmission is complete.

[1089] Step 3:

[1090] The server calculates the nutrients contained in each meal based on the input dietary information.

[1091] Input: Meal information sent to the server.

[1092] Output: Calculated nutrient information for each meal.

[1093] Specific operation: The server uses Python's pandas library to calculate the nutrients for each food.

[1094] Step 4:

[1095] The server stores the calculated nutrient information in a database.

[1096] Input: Calculated nutrient information.

[1097] Output: Nutrient information stored in a database.

[1098] Specific operation: The server records the nutrient information in a database and manages it for each user.

[1099] Step 5:

[1100] The server analyzes nutritional balance based on a week's dietary history and identifies any nutrient deficiencies.

[1101] Input: A week's worth of dietary history recorded in a database.

[1102] Output: Nutritional balance analysis results.

[1103] Specific operation: Analyzes dietary history using Python's scikit-learn library and evaluates nutritional balance.

[1104] Step 6:

[1105] The server creates a nutritional balance report and notifies the user through the app.

[1106] Input: Nutritional balance analysis results.

[1107] Output: Report notification to user.

[1108] Specific operation: The server creates a report based on the analysis results and sends a push notification to the app.

[1109] Preventing food waste

[1110] Step 1:

[1111] The server estimates the expiration date based on the purchase date information of the food product.

[1112] Input: Food purchase date information recorded in the database.

[1113] Output: Estimated expiration date.

[1114] What it does: Calculates expiration dates using Python's datetime library.

[1115] Step 2:

[1116] The server generates alerts for food items approaching their expiration date.

[1117] Input: Estimated expiration date information.

[1118] Output: Alert information.

[1119] Specific behavior: The server generates an alert message based on the expiration date and adds it to the alert list.

[1120] Step 3:

[1121] The server notifies the user of the generated alert on their smartphone.

[1122] Input: Alert information.

[1123] Output: Notification to the user's smartphone.

[1124] What it does: Sends an alert message to the user's smartphone app using push notification.

[1125] Examples of application of emotion analysis methods

[1126] Step 1:

[1127] While the user prepares dinner, the emotion analysis means analyzes facial expressions and tone of voice through the smartphone's camera and microphone.

[1128] Input: User's facial expression and voice data.

[1129] Output: Parsed emotion data.

[1130] How it works: The emotion analyzer analyzes the user's facial expressions and tone of voice in real time to assess their stress and happiness levels.

[1131] Step 2:

[1132] If the emotion analysis means determines that the user is feeling stressed, the server preferentially suggests recipes that use foods that have a relaxing effect.

[1133] Input: Parsed emotion data.

[1134] Output: Relaxing recipe suggestions.

[1135] What it does: The server searches the database for relaxing recipes and updates the list of suggestions.

[1136] Step 3:

[1137] When your diet is lacking in balance, the server will suggest foods that contain nutrients that help relieve stress.

[1138] Input: Nutritional balance analysis results and emotional data.

[1139] Output: Nutrition-conscious recipe suggestions.

[1140] What it does: Based on the missing nutritional information, the server recommends recipes that include foods that help relieve stress.

[1141] Step 4:

[1142] When food in the refrigerator is nearing its expiration date, the server will quickly alert the user if they are feeling down, along with an encouraging message.

[1143] Input: Emotion data and expiration date information.

[1144] Output: Emotion-sensitive alert notification.

[1145] What it does: The server adjusts the content and timing of alert notifications based on emotion data and adds encouraging messages.

[1146] (Application example 2)

[1147] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1148] Conventional refrigerator systems require complicated management of food items in the refrigerator and expiration dates, and do not adequately achieve health management for users, reduce the burden of housework, or reduce food waste. Furthermore, they do not suggest or manage food based on the user's emotional state, and do not provide detailed support to users. This results in insufficient management of users' health and efficiency in housework, and often leads to food waste.

[1149] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1150] In this invention, the server includes a camera for taking pictures of food in the refrigerator, an image analysis unit for analyzing the images taken by the camera and recognizing the food, a suggestion unit for suggesting currently available recipes based on the recognized food information and the user's emotional information, a nutrition calculation unit for calculating the nutrients of each food and managing a nutritionally balanced diet, a food consumption management unit for estimating expiration dates based on food purchase dates and generating alerts for food items approaching their expiration date, and an emotional response unit for providing messages and suggestions based on the user's emotional state. This improves the efficiency of refrigerator food management, reduces the user's health management, eases housework burden, and reduces food waste. Furthermore, by suggesting recipes and generating alerts based on the user's emotional state, the server provides detailed support and improves user satisfaction.

[1151] 1. "Photographing means" refers to a photographing device placed to photograph food items in a refrigerator.

[1152] 2. "Image analysis means" refers to a means that has the function of analyzing the image acquired by the photographing means and recognizing food.

[1153] 3. "Suggestion means" refers to a means that has the function of suggesting currently available recipes based on recognized food information and user emotion information.

[1154] 4. "Nutrition calculation tool" refers to a tool that has the function of calculating the nutrients of each food and managing a nutritionally balanced diet.

[1155] 5. "Food expiration date management means" refers to a means that has the function of estimating the expiration date based on the purchase date information of food and generating an alert for food that is approaching its expiration date.

[1156] 6. "Emotion response means" refers to means that have the function of sending messages or suggestions according to the user's emotional state.

[1157] DETAILED DESCRIPTION OF THE INVENTION

[1158] The following describes an embodiment of the present invention: The system mainly operates using a camera means built into the refrigerator, a server, and a user's smartphone or other user terminal.

[1159] First, a camera installed inside the refrigerator periodically takes images of the inside of the refrigerator. This image data is sent to a server via the Internet. The server then uses image analysis means (such as image analysis software) to recognize food from the images and store the food information in a database. Specific image analysis uses the widely used OpenCV library, etc.

[1160] Next, when the user requests a dinner recipe, the server retrieves the latest food information and generates the optimal recipe using a suggestion method (recipe suggestion algorithm). This also takes the user's emotional information into consideration. Emotion recognition software such as EmotionEngine is used as an emotional response method to analyze the user's emotions. Based on the ingredient data and the user's emotional state, a recipe such as "tomato cream pasta" is sent to the user's smartphone.

[1161] The nutritional calculation tool calculates the nutritional value of each recipe based on the recipe selected by the user and the information on the meal they have actually made. This information is recorded on a server, and a report is created that analyzes the nutritional balance based on a week's worth of meal history and identifies any nutrient deficiencies. The report is sent to the user via the app.

[1162] Furthermore, the food consumption period management means estimates the expiration date based on the purchase date information of the food, and generates an alert for food that is approaching its expiration date. This alert is also notified to the user at an appropriate time using the emotion response means.

[1163] For example, for milk that is nearing its expiration date, a specific alert could be generated such as "Your milk is nearing its expiration date. Please consume it soon," accompanied by an uplifting message that reflects the user's emotional state.

[1164] Examples:

[1165] 1. Images taken by the refrigerator camera are periodically sent to the server.

[1166] 2. Using image analysis software (e.g., OpenCV), the food items in the refrigerator are recognized. For example, data such as "3 tomatoes, 1 carton of milk, and 6 eggs" is obtained.

[1167] 3. When a user requests a dinner recipe on their smartphone, the server uses the Emotion Engine to analyze the user's emotional information (e.g., stress level).

[1168] 4. Based on the user’s emotions and the food information in the refrigerator, the suggestion tool will suggest a recipe for “Tomato Cream Pasta” and notify the smartphone.

[1169] 5. The nutrition calculation means calculates the nutrients in the tomato cream pasta and creates a weekly nutritional balance report based on the user's dietary history.

[1170] 6. An alert will be generated for food products (such as milk) that are nearing their expiration date, and a notification will be sent to your smartphone saying, "Your milk is nearing its expiration date. Please consume it as soon as possible."

[1171] Example prompt sentence:

[1172] "Analyze images of the inside of the refrigerator and tell me the current food inventory."

[1173] "Please suggest a relaxing recipe based on my mood."

[1174] "Get alerts for foods that are nearing their expiration date."

[1175] This system will improve the efficiency of food management in the refrigerator, enabling users to manage their health, reduce household chores, and reduce food waste. It will also provide detailed support by suggesting recipes and generating alerts based on the user's emotional state, improving user satisfaction.

[1176] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1177] Specific operation steps:

[1178] Step 1:

[1179] The refrigerator's camera periodically captures images of the inside of the refrigerator, and the server receives this image data via the Internet.

[1180] Input: Image data of the inside of the refrigerator

[1181] Data processing: Image data is sent to a server via the Internet

[1182] Output: Image data received by the server

[1183] Step 2:

[1184] The server uses image analysis means (image analysis software) to analyze the received image data and identify the food items in the refrigerator. The information about the identified food items is stored in a database.

[1185] Input: Image data received by the server

[1186] Data analysis: Recognize food using image analysis software (e.g., OpenCV)

[1187] Output: Recognized food information (e.g., 3 tomatoes, 1 carton of milk, 6 eggs)

[1188] Step 3:

[1189] A user requests a dinner recipe via a device such as a smartphone. This request is sent to the server, which then retrieves the latest information about the food in the refrigerator based on the recognized food information.

[1190] Input: User recipe request

[1191] Data acquisition: Acquire the latest refrigerator food information from the database

[1192] Output: Latest information on food in the refrigerator

[1193] Step 4:

[1194] The server uses emotion response means (emotion recognition software) to acquire the user's emotion data from the camera and microphone of the smartphone or terminal in order to analyze the user's emotion information.

[1195] Input: User's camera video and audio data

[1196] Data analysis: Analyze emotions using emotion recognition software (e.g., EmotionEngine)

[1197] Output: User's emotional state (e.g., stress state)

[1198] Step 5:

[1199] The server uses a suggestion means (recipe suggestion algorithm) to select the optimal recipe based on the recognized food information and the user's emotional information, and notifies the user of the recipe to their smartphone.

[1200] Input: Recognized food information, user emotion information

[1201] Data processing: Applying recipe suggestion algorithm

[1202] Output: Notification of a suggested recipe (e.g., "Tomato Cream Pasta")

[1203] Step 6:

[1204] The server records the recipe information that the user selects and actually makes, and the server then calculates the nutritional content of each meal based on this information.

[1205] Input: User-generated meal information

[1206] Data Analysis: Nutrient Calculations

[1207] Output: Nutrition information for each meal

[1208] Step 7:

[1209] The server uses a nutritional calculation tool to analyze nutritional balance based on a week's worth of dietary history and creates a report identifying nutrient deficiencies, which is then sent to the user via the app.

[1210] Input: One week's worth of food history

[1211] Data analysis: Nutritional balance analysis

[1212] Output: Nutritional balance report creation and notification

[1213] Step 8:

[1214] The server uses the food consumption period management means to estimate the expiration date based on the purchase date information of the food, and generates an alert for food that is approaching its expiration date. The alert is sent to the smartphone with appropriate timing and content according to the user's emotional state using the emotional response means.

[1215] Input: Food purchase date information

[1216] Data analysis: Estimating expiration dates and considering emotional data

[1217] Output: Alert notification for food products that are nearing their expiration date (e.g. "The milk is nearing its expiration date. Please consume it as soon as possible.")

[1218] Example prompt sentence:

[1219] "Analyze images of the inside of the refrigerator and tell me the current food inventory."

[1220] "Please suggest a relaxing recipe based on my mood."

[1221] "Get alerts for foods that are nearing their expiration date."

[1222] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1223] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1224] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1225] [Third embodiment]

[1226] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1227] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1228] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1229] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1230] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1231] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1232] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1233] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1234] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1235] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1236] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1237] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1238] The system of the present invention has a built-in camera, image analysis, recipe suggestion, nutrition management, and food waste prevention means. By using this system, food in the refrigerator can be efficiently managed, and social issues such as preventive medicine, reducing the burden of housework, and preventing food waste can be addressed.

[1239] The camera means periodically photographs the food in the refrigerator and sends the image data to the server. The server analyzes the received images using the image analysis means and recognizes the food. The recognized food information is stored in a database, and the recipe suggestion means suggests appropriate recipes to the user based on this information. The nutrition management means also calculates the nutritional value of the food and performs nutritional management to provide a balanced diet.

[1240] Furthermore, the food loss prevention system estimates the expiration date based on the purchase date of the food, generates an alert for food that is approaching its expiration date, and notifies the user's device, thereby preventing food waste.

[1241] The specific processing contents of the program will be explained in natural language below.

[1242] Ingredient recognition processing

[1243] The server receives the image taken by the camera means and analyzes the image using the image analysis means. For example, if there are tomatoes, milk, and eggs on the shelves of a refrigerator, the server identifies these foods and stores them in the database as "3 tomatoes, 1 carton of milk, 6 eggs."

[1244] Recipe suggestion processing

[1245] When a user requests a dinner recipe, the server retrieves information about the food in the refrigerator from the database and searches the recipe database. For example, if there are tomatoes and milk in the refrigerator, it will suggest a recipe for tomato cream pasta. This suggestion is sent to the user's smartphone, and the user can check the cooking instructions and other ingredients needed through the app.

[1246] Nutritional balance management

[1247] To record the meals a user has eaten, the server collects information on suggested recipes and the meals actually cooked, allowing the system to calculate how much protein, carbohydrates, and fat a user has consumed in a week and notify them of any nutrient deficiencies.

[1248] For example, if a user's protein intake is low over the course of a week, the server will generate a report saying, "You're not getting enough protein. Try some high-protein recipes."

[1249] Preventing food waste

[1250] The server manages food expiration dates using information on food purchase dates and an expiration date estimation algorithm. For example, if milk's expiration date is approaching in three days, the server generates an alert saying, "The milk's expiration date is approaching. Please consume it soon," and notifies the user's smartphone.

[1251] As described above, the system of the present invention provides a specific method for efficiently managing food in a refrigerator, supporting health management, and reducing food waste.

[1252] The processing flow will be explained below.

[1253] Ingredient recognition processing

[1254] Step 1:

[1255] The camera takes pictures of the inside of the refrigerator.

[1256] The camera means periodically takes an image of each shelf in the refrigerator, and generates image data.

[1257] Step 2:

[1258] The image data is sent to a server.

[1259] The image data acquired by the camera means is transmitted to a server via the Internet.

[1260] Step 3:

[1261] The server analyzes the image data.

[1262] The server uses image analysis means to recognize foods from the image, for example, the server identifies tomatoes, milk, and eggs from the image.

[1263] Step 4:

[1264] The recognized food information is stored in a database.

[1265] The server stores the recognition result "3 tomatoes, 1 carton of milk, 6 eggs" in a database.

[1266] Recipe suggestion processing

[1267] Step 1:

[1268] A user requests recipe suggestions.

[1269] A user requests dinner recipe suggestions on a smartphone app.

[1270] Step 2:

[1271] The server acquires information about the food in the refrigerator.

[1272] The server retrieves the latest food information from the database.

[1273] Step 3:

[1274] The server searches the recipe database.

[1275] The server searches for available recipes based on the food information in the refrigerator. For example, if there are tomatoes and milk, it will extract the recipe for "Tomato Cream Pasta."

[1276] Step 4:

[1277] The server creates recipe suggestions.

[1278] The server creates a recipe suggestion that includes ingredients, cooking instructions, and nutritional information.

[1279] Step 5:

[1280] Recipe suggestions are notified to the device.

[1281] The server notifies the user of the recipe suggestions on their smartphone, which are then displayed within the app.

[1282] Nutritional balance management

[1283] Step 1:

[1284] Record the user's dietary history.

[1285] The app records the recipes the user has cooked and sends that information to the server.

[1286] Step 2:

[1287] The server calculates the nutrients.

[1288] The server calculates the nutrients contained in each meal and stores them in a database.

[1289] Step 3:

[1290] The server analyzes the nutritional balance.

[1291] The server analyzes nutritional balance based on dietary history over a certain period (e.g., one week) and identifies any nutrient deficiencies or excess intake.

[1292] Step 4:

[1293] Create a report.

[1294] The server generates a detailed report based on the results of the nutritional balance analysis, including, for example, "You are lacking in protein."

[1295] Step 5:

[1296] The report is sent to the device.

[1297] The server sends the created report to the user's smartphone, where the user can check the report contents through the app.

[1298] Preventing food waste

[1299] Step 1:

[1300] The server estimates the expiration date.

[1301] The server estimates the expiration date based on the purchase date of the food product. For example, it calculates the expiration date of milk.

[1302] Step 2:

[1303] Manage expiration date information.

[1304] The server manages the estimated expiration dates in a database and sets an alert flag when the expiration date approaches.

[1305] Step 3:

[1306] Generate an alert.

[1307] The server generates alerts for food products that are approaching their expiration date, for example, "Milk expires tomorrow."

[1308] Step 4:

[1309] Sends alerts to the device.

[1310] The server sends an alert to the user's smartphone, which is then displayed within the app.

[1311] Specific examples

[1312] 1. If there are tomatoes, milk, or eggs in the refrigerator, the camera means takes a picture of them and sends the image to the server.

[1313] 2. The server uses image analysis to recognize three tomatoes, one carton of milk, and six eggs, and stores them in a database.

[1314] 3. When a user requests a recipe for dinner, the server retrieves information about the food in the refrigerator, searches for available recipes, and suggests "Tomato Cream Pasta."

[1315] 4. When the user cooks a recipe and records their meal history in the app, the server calculates the nutrients and analyzes a week's worth of meal history to check nutritional balance.

[1316] 5. When the milk is nearing its expiration date, the server estimates the expiration date and generates an alert to notify the user.

[1317] The above is a specific processing flow using the system of the present invention.

[1318] Example 1

[1319] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1320] In modern society, there is a demand for both labor-saving and health management. However, food management is complicated and time-consuming. In particular, properly managing food in the refrigerator and consuming it while it is fresh is effective in preventing food waste, but manual management is difficult and not often implemented. It is also difficult to provide a balanced diet that takes into account the nutritional value of each food item. Therefore, there is a need for a system that can efficiently manage food in the refrigerator.

[1321] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1322] In this invention, the server includes a camera for capturing images of objects in the refrigerator, an analysis unit for analyzing the images captured by the camera to recognize the objects, a suggestion unit for proposing currently available cooking procedures based on the recognized object information, a management unit for calculating the nutritional content of each object and managing a nutritionally balanced diet, and a deterioration prevention unit for estimating the expiration date of each object based on the object's acquisition date information and generating a notification for objects approaching their expiration date. This allows for efficient and automatic management of food in the refrigerator and encourages consumption at the appropriate time. Furthermore, balanced meal suggestions and nutritional management can support a healthy lifestyle.

[1323] The "photography means" is a device that is installed inside the refrigerator and takes images of food or objects.

[1324] The "analysis means" is a system that analyzes the image captured by the imaging means and recognizes the type of food or object.

[1325] The "suggestion means" is a mechanism that provides currently available cooking procedures to the user based on the information about the food or object recognized by the analysis means.

[1326] The "management means" is a system for calculating the nutritional content of foods and objects and managing a balanced diet.

[1327] The "deterioration prevention means" is a mechanism that estimates the expiration date of a food or object based on the acquisition date information, and generates a notification to the user when the expiration date approaches.

[1328] The system of the present invention efficiently manages food in a refrigerator, enabling health management, reducing the burden of housework, and preventing food waste. This system includes a photographing means, an analyzing means, a suggestion means, a management means, and a deterioration prevention means.

[1329] 1. Photography Method

[1330] The imaging means is a camera installed inside the refrigerator that periodically captures images of food and objects. For example, the camera takes a picture of the shelves inside the refrigerator every hour and sends the image data to a server. The specific hardware used is a high-resolution digital camera.

[1331] 2. Analysis method

[1332] The analysis means is software installed on the server that analyzes the images sent by the image capture means. This analysis uses an object recognition algorithm, such as TensorFlow or OpenCV. The analysis means automatically identifies and identifies objects in the image. For example, if there are tomatoes, milk, and eggs on a shelf in a refrigerator, the analysis means will recognize them and store them in the database as "3 tomatoes, 1 carton of milk, 6 eggs."

[1333] 3. Proposal method

[1334] When a user requests a dinner recipe, the suggestion means retrieves information about the food in the refrigerator from the database and searches the recipe database. For example, if there are tomatoes and milk in the refrigerator, the suggestion means selects a recipe for "tomato cream pasta" and notifies the user's smartphone. The specific software used is a database search algorithm using Python or Java.

[1335] 4. Control measures

[1336] The management means collects information about the meals the user has eaten and calculates the amount of nutrients the user has consumed for the week. This analyzes how much protein, carbohydrates, and fat the user has consumed and notifies the user of any nutrient deficiencies. For example, if the user's protein intake is low for the week, a report such as "You are lacking in protein. Please try a high-protein recipe" is generated and the user is notified.

[1337] 5. Measures to prevent deterioration

[1338] The deterioration prevention method estimates the expiration date based on the food's acquisition date and generates a notification for food that is approaching its expiration date. For example, if the expiration date of milk is approaching in three days, an alert will be generated and sent to the user's smartphone saying, "The milk's expiration date is approaching. Please consume it soon."

[1339] Example of a concrete example and prompt for the generative AI model:

[1340] When an image of the inside of a refrigerator is taken by the photographing means and the analyzing means recognizes that there are "3 tomatoes, 1 carton of milk, and 6 eggs," the specific processing will be as follows:

[1341] Example prompt sentence:

[1342] "Analyze images of the inside of the refrigerator and identify what's inside."

[1343] Input image: 3 tomatoes, 1 carton of milk, 6 eggs

[1344] What happens when a user asks for dinner recipes?

[1345] Example prompt sentence:

[1346] "I have some tomatoes and milk in the fridge. Can you suggest a recipe using them?"

[1347] This system will enable efficient management of food in the refrigerator and encourage consumption at the appropriate time. Furthermore, it will support a healthy lifestyle by suggesting balanced meals and managing nutrition.

[1348] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1349] Step 1:

[1350] Image acquisition

[1351] The server receives an image of the inside of the refrigerator using the photographing means.

[1352] Input: Image data taken inside the refrigerator

[1353] Output: Image data sent to the server

[1354] Specific operation: The camera built into the refrigerator takes pictures every hour and sends them to the server.

[1355] Step 2:

[1356] Image analysis

[1357] The server analyzes the received image using the image analysis means.

[1358] Input: Image data sent to the server

[1359] Output: Analyzed object information

[1360] Data processing and calculation: Using an object recognition algorithm (e.g., TensorFlow, OpenCV), identify the type and number of identified objects.

[1361] Specific operation: The server identifies objects in the image and obtains information such as "3 tomatoes, 1 carton of milk, and 6 eggs."

[1362] Step 3:

[1363] Data storage

[1364] The server stores the analysis results (object information) in a database.

[1365] Input: Parsed object information

[1366] Output: Object information stored in a database

[1367] Specific operation: The server writes information such as "3 tomatoes, 1 carton of milk, 6 eggs" into the database.

[1368] Step 4:

[1369] Recipe request reception

[1370] The server receives a recipe request from a user terminal.

[1371] Input: User request data

[1372] Output: Processing based on the request begins

[1373] Specific Action: A user requests a dinner recipe through a smartphone app.

[1374] Step 5:

[1375] Food information acquisition

[1376] The server obtains information about the food items in the refrigerator from the database.

[1377] Input: A query to the database

[1378] Output: Food information

[1379] Specific operation: The server retrieves food information from the database, such as "3 tomatoes, 1 carton of milk, 6 eggs."

[1380] Step 6:

[1381] Recipe Search

[1382] The server uses the suggestion means to search a recipe database based on the food information.

[1383] Input: Food information

[1384] Output: appropriate recipe information

[1385] Data processing and calculation: Matching food information with the recipe database to select the optimal recipe.

[1386] Specific operation: The server searches the recipe database and selects the recipe for "Tomato Cream Pasta."

[1387] Step 7:

[1388] Recipe Notifications

[1389] The server notifies the user terminal of the selected recipe.

[1390] Input: appropriate recipe information

[1391] Output: Recipe information sent to the user's device

[1392] Specific operation: The server sends the recipe for "Tomato Cream Pasta" to the user's smartphone, allowing the user to view it in the app.

[1393] Step 8:

[1394] Gathering dietary information

[1395] The server collects meal information from the user.

[1396] Input: User's food record data

[1397] Output: Meal data stored on the server

[1398] Specific operation: The user enters a meal record into the app, and the server receives it.

[1399] Step 9:

[1400] Nutrition Calculations

[1401] The server uses the management means to calculate the intake of nutrients from the dietary data for one week.

[1402] Input: Meal data

[1403] Output: Nutritional balance report

[1404] Data processing and calculation: Analyze dietary data and calculate protein and carbohydrate intake.

[1405] Specific operation: The server creates a report based on the user's weekly nutritional intake.

[1406] Step 10:

[1407] Nutrition Report Notification

[1408] The server notifies the user terminal of a report based on the nutrients that are lacking.

[1409] Input: Nutritional Balance Report

[1410] Output: Nutrition report sent to user device

[1411] Specific operation: The server sends a message to the user's smartphone saying, "You are lacking in protein. Please try a high-protein recipe."

[1412] Step 11:

[1413] Expiration date estimation

[1414] The server estimates the expiration date based on the purchase date information of the food product using the deterioration prevention means.

[1415] Input: Food purchase date information

[1416] Output: Estimated expiration date information

[1417] Data processing and calculation: The expiration date is calculated using the purchase date information and an algorithm.

[1418] Specific operation: The server estimates that the milk's expiration date is approaching in three days.

[1419] Step 12:

[1420] Expiration date alert generation

[1421] The server generates alerts for food items approaching their expiration date.

[1422] Input: Estimated expiration date information

[1423] Output: The generated alert message

[1424] Specific behavior: The server creates an alert message saying, "The milk's expiration date is approaching. Please consume it soon."

[1425] Step 13:

[1426] Expiration date alert notification

[1427] The server notifies the user terminal of the generated alert.

[1428] Input: The generated alert message

[1429] Output: The alert message sent to the user's terminal.

[1430] Specific operation: The server sends an alert to the user's smartphone, which the user can check in the app.

[1431] (Application example 1)

[1432] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1433] In modern society, food management, nutritional management, and food waste prevention at home and in brick-and-mortar stores are important issues. There is a need for a system that efficiently manages food not only at home but also in brick-and-mortar stores, and helps consumers purchase appropriate and balanced ingredients. The existence of such a system would greatly contribute to consumer health management and food waste reduction. Another important issue is reducing the financial burden on consumers by providing real-time sales information.

[1434] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1435] In this invention, the server includes a camera means for photographing food in the refrigerator, an image analysis means for analyzing images photographed by the camera means to recognize food, a recipe suggestion means for suggesting currently available recipes based on the recognized food information, a nutrition management means for calculating the nutrients of each food and managing a nutritionally balanced diet, a food waste prevention means for estimating expiration dates based on food purchase date information and generating an alert for food approaching its expiration date, a camera means for photographing food in a physical store, an image analysis means for analyzing images photographed by the camera means in the physical store to recognize food, a shopping list generation means for generating and notifying a shopping list based on the recognized food information in the physical store, a recipe suggestion means for suggesting and notifying recipes based on food available in the physical store, and a sale information notification means for recognizing and notifying sale information in the physical store. This enables efficient food management at home and in physical stores, providing nutritionally balanced meals, preventing food waste, and reducing economic burdens.

[1436] The "camera means" is a device for taking pictures of food in a refrigerator and in a physical store.

[1437] The "image analysis means" is a device for analyzing the image captured by the camera means and recognizing food.

[1438] The "recipe suggestion means" is a device for suggesting currently available recipes based on recognized food information.

[1439] The "nutritional management means" is a device that calculates the nutrients of each food and manages a nutritionally balanced diet.

[1440] The "food waste prevention device" is a device that estimates the expiration date based on the purchase date of food and generates an alert for food that is approaching its expiration date.

[1441] "Camera means in a physical store" refers to a device for photographing food in a physical store.

[1442] The "image analysis means in the physical store" is a device for recognizing food by analyzing images taken by a camera means in the physical store.

[1443] The "shopping list generation means" is a device for generating and notifying a shopping list based on the food information in the recognized physical store.

[1444] The "in-store recipe suggestion means" is a device for suggesting and notifying recipes based on food items that can be purchased in the store.

[1445] The "sale information notification means" is a device for recognizing and notifying sale information within a physical store.

[1446] The system of the present invention efficiently manages food in refrigerators and brick-and-mortar stores, and solves social issues such as preventive medicine, reducing the burden of housework, and preventing food waste. The system includes a camera, image analysis, recipe suggestion, nutritional management, and food waste prevention, as well as brick-and-mortar store-exclusive camera, image analysis, shopping list generation, in-store recipe suggestion, and sale information notification.

[1447] Hardware and software used:

[1448] Camera means: A camera installed in a refrigerator or in a physical store, which is a device for taking pictures of food.

[1449] Image analysis means: A device that analyzes captured images and recognizes the type of food, using common computer vision software such as OpenCV.

[1450] Recipe suggestion unit: A device that searches for possible recipe suggestions based on the recognized food information and notifies the user. It also works in conjunction with a recipe database.

[1451] Nutrition management tool: A device that calculates the nutritional value of food and supports balanced diet management. It uses an AI-based nutrition analysis tool.

[1452] Food waste prevention method: A device that estimates the expiration date of food from the purchase date information and generates an alert to notify the user when food is approaching its expiration date.

[1453] In-store camera means: A device installed in a physical store that takes pictures of food products being sold.

[1454] In-store image analysis means: A device that analyzes images taken by camera means in a physical store and recognizes the food being sold.

[1455] Shopping list generation means: A device that generates a shopping list based on the recognized food information in the physical store, along with the current refrigerator inventory, and notifies the user.

[1456] In-store recipe suggestion means: A device that suggests recipes based on food items available for purchase in a physical store and notifies the user.

[1457] Sale information notification means: A device that analyzes sale information in physical stores and notifies users.

[1458] Example of a system:

[1459] The system constantly monitors food inventory in the refrigerator and automatically generates a shopping list as needed. For example, if there is only one liter of milk, two tomatoes, and three peppers left in the refrigerator, the refrigerator's camera captures this information and analyzes it using image analysis. Based on this information, the recipe suggestion module suggests recipes using these foods to the user and automatically adds any missing ingredients to the shopping list.

[1460] When a user visits a physical store, a camera in the store takes a photo of the food on the shelves, and the image analysis means analyzes the information and notifies the user in real time. Furthermore, sale information is also analyzed, and economical sale items are suggested. In this way, users can comfortably and efficiently purchase the food they need.

[1461] Example prompt sentence:

[1462] "The current inventory in your refrigerator is:

[1463] 1 liter of milk

[1464] 2 tomatoes

[1465] 3 bell peppers

[1466] In-store specials include:

[1467] 2 tomatoes 100 yen

[1468] 20% off 1 pack of milk

[1469] Use this to generate recipe recommendations and shopping lists."

[1470] This invention allows for food management, nutritional balance and economical shopping at home and in physical stores.

[1471] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1472] Step 1:

[1473] The server periodically takes pictures of the food using a camera in the refrigerator. The input is the image obtained from the camera, and the output is an image file of the photographed food. This image file is then handed over to the next analysis process.

[1474] Step 2:

[1475] The server uses image analysis means to analyze the photographed food image and recognize the type and quantity of food. The input is the photographed image file, and the output is a list of the recognized food types and quantities. This list is stored in a database.

[1476] Step 3:

[1477] A user sends a recipe request using a smartphone. The input is the user's request, and the output is a list of suitable recipe candidates produced by the recipe suggestion means. This candidate list is then sent to the user's terminal.

[1478] Step 4:

[1479] The server uses the nutrition management tool to calculate the nutritional value of each food item based on the proposed recipe. The input is the ingredient information of the proposed recipe, and the output is a list of the nutrients contained in each recipe. This list is saved as the user's health management information and can be viewed by the user on their smartphone.

[1480] Step 5:

[1481] The server uses a food waste prevention method to estimate expiration dates based on food purchase date information and identify foods that are approaching their expiration date. The input is the food purchase date information, and the output is a list of foods that are approaching their expiration date. This list is notified to the user's device as an alert.

[1482] Step 6:

[1483] The server takes pictures of food items on shelves using a camera in the physical store and analyzes them using an image analysis means. The input is the image obtained from the camera in the physical store, and the output is a list of the types and quantities of recognized food items. This list is sent to the physical store management system.

[1484] Step 7:

[1485] The server compares the recognized food information in the physical store with the user's refrigerator inventory and generates and notifies the user of a shopping list containing the necessary foods. The input is the food information in the physical store and the refrigerator inventory information, and the output is the generated shopping list. This list is notified to the user's smartphone.

[1486] Step 8:

[1487] The server proposes recipes based on information about food items available in physical stores and notifies the user's device. The input is information about food items in the store and inventory in the refrigerator, and the output is a list of suggested recipes. This list is then sent to the user's smartphone.

[1488] Step 9:

[1489] The server analyzes sales information in physical stores and notifies the user. The input is sales information, and the output is a list of foods on sale. This list is sent to the user's smartphone to assist them in their shopping.

[1490] This series of processing steps allows users to efficiently manage and purchase food, enabling them to maintain a nutritionally balanced lifestyle.

[1491] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1492] The present invention is a system that has a camera means, image analysis means, recipe suggestion means, nutritional management means, food waste prevention means, and emotion engine means built into a refrigerator. This system efficiently manages food in the refrigerator, helping to maintain health, reduce the burden of housework, and reduce food waste, as well as suggesting recipes and managing nutrition based on the user's emotions.

[1493] Ingredient recognition processing

[1494] The camera means periodically takes pictures of the inside of the refrigerator and sends the image data to a server via the Internet. The server uses image analysis means to recognize food from the images and stores information about the food (e.g., 3 tomatoes, 1 carton of milk, 6 eggs) in a database.

[1495] Recipe suggestion processing

[1496] When a user requests a dinner recipe, the server retrieves the latest food information and searches for available recipes. The emotion engine also collects the user's emotion data (e.g., stress, joy). The server selects an appropriate recipe based on this emotion data and notifies the user's smartphone of a recipe such as "tomato cream pasta."

[1497] Nutritional balance management

[1498] The recipes selected by the user and information about the meals they actually made are recorded on the server. Based on this, the server calculates the nutrients contained in each meal and stores the data in a database. It analyzes nutritional balance based on the dietary history over a certain period (for example, one week) and identifies any nutrient deficiencies. At this time, it also takes into account data from the emotion engine and creates a detailed report, such as "On days with high stress, the required intake of vitamin C is low." The report is notified to the user via the app.

[1499] Preventing food waste

[1500] The server estimates the expiration date based on the food's purchase date information and generates an alert for food that is approaching its expiration date. For example, if the expiration date of milk is approaching in three days, the server creates an alert saying, "The milk's expiration date is approaching. Please consume it soon," and sends it to the user's smartphone. The content and timing of the alert can be adjusted to suit the user's emotional state using data from the emotion engine means.

[1501] Specific processing examples of emotion engine means

[1502] 1. While the user is preparing dinner, the emotion engine means analyzes facial expressions and tone of voice through the smartphone's camera and microphone.

[1503] 2. If the emotion engine means determines that the user is feeling stressed, the server will preferentially suggest recipes that use foods that have a relaxing effect.

[1504] 3. When a meal lacks balance, the server will include in the suggestions foods that contain nutrients that help relieve stress.

[1505] 4. When food in the refrigerator is nearing its expiration date, the server will quickly alert the user if they are feeling down, along with an encouraging message.

[1506] The above is a specific processing flow using the system of the present invention. This system optimizes food management in the refrigerator while taking into consideration the user's feelings, making it possible to manage health and reduce the burden of housework, as well as reducing food waste.

[1507] The processing flow will be explained below.

[1508] Ingredient recognition processing

[1509] Step 1:

[1510] The camera takes pictures of the inside of the refrigerator.

[1511] A camera means periodically takes images of the food shelves in the refrigerator.

[1512] Step 2:

[1513] The image data is sent to a server.

[1514] Image data captured by the camera means is transmitted to a server via the Internet.

[1515] Step 3:

[1516] The server analyzes the image data.

[1517] The server uses image analysis to recognize food items from the image, for example, distinguishing between tomatoes, milk, and eggs.

[1518] Step 4:

[1519] The recognized food information is stored in a database.

[1520] The server stores the recognition results (3 tomatoes, 1 carton of milk, and 6 eggs) in a database.

[1521] Recipe suggestion processing

[1522] Step 1:

[1523] A user requests recipe suggestions.

[1524] A user requests a dinner recipe on a smartphone app.

[1525] Step 2:

[1526] The server acquires information about the food in the refrigerator.

[1527] The server retrieves current food information from the database.

[1528] Step 3:

[1529] The emotion engine analyzes the user's emotions.

[1530] The emotion engine means analyzes the user's facial expressions and tone of voice through the smartphone's camera and microphone to determine the user's current emotional state.

[1531] Step 4:

[1532] The server searches the recipe database.

[1533] The server searches for appropriate recipes based on the food information in the refrigerator and the user's emotional data. For example, if the user is feeling stressed, it will select recipes using foods that have a relaxing effect.

[1534] Step 5:

[1535] The server creates recipe suggestions.

[1536] Servers create recipe suggestions that include required ingredients, cooking instructions, nutritional information, and emotionally sensitive comments.

[1537] Step 6:

[1538] Recipe suggestions are notified to the device.

[1539] The server notifies the user's smartphone of the recipe suggestions and displays detailed information within the app.

[1540] Nutritional balance management

[1541] Step 1:

[1542] Record the user's dietary history.

[1543] The app records the recipes the user has cooked and sends that information to the server.

[1544] Step 2:

[1545] The server calculates the nutrients.

[1546] The server calculates the nutrients contained in each meal and stores them in a database.

[1547] Step 3:

[1548] The server analyzes the nutritional balance.

[1549] The server analyzes the user's nutritional balance based on their dietary history over a certain period of time (for example, one week).

[1550] Step 4:

[1551] The server generates the report.

[1552] The server creates a detailed report based on the results of the nutritional balance analysis, taking into account the user's emotional data.

[1553] Step 5:

[1554] The report is sent to the device.

[1555] The server sends a detailed report to the user's smartphone, where the user can check the report using the app.

[1556] Preventing food waste

[1557] Step 1:

[1558] The server estimates the expiration date.

[1559] The server estimates the expiration date of each food item based on the purchase date information of the food item.

[1560] Step 2:

[1561] Save expiration date information in a database.

[1562] The server registers and manages the estimated expiration date in a database.

[1563] Step 3:

[1564] Generate an alert.

[1565] The server generates an alert message for food products that are approaching their expiration date.

[1566] Step 4:

[1567] Sends alerts to the device.

[1568] The server sends an alert to the user's smartphone, informing them that the expiration date is approaching.

[1569] Specific processing of emotion engine means

[1570] Step 1:

[1571] Collect user emotion data.

[1572] The emotion engine means uses the smartphone's camera and microphone to collect the user's facial expressions, tone of voice, and other emotion indicators.

[1573] Step 2:

[1574] Analyze emotion data.

[1575] Based on the data collected by the emotion engine means, the user's current emotional state is analyzed and determined.

[1576] Step 3:

[1577] The determined emotion is sent to the server.

[1578] The emotion engine means transmits the analysis results to the server.

[1579] Step 4:

[1580] Adjust recipes and alerts based on sentiment data.

[1581] The server takes emotional data into consideration and adjusts the content and timing of recipe suggestions and alerts to provide users with the most appropriate information.

[1582] The above is a specific processing flow using the system of the present invention. By combining it with an emotion engine means, recipe suggestions and food waste prevention that take into account the user's emotional state can be realized, further improving the user experience.

[1583] Example 2

[1584] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1585] Managing food in the refrigerator poses the problem of making it difficult to understand expiration dates and manage nutritional balance, making household diet management, which relies on this, cumbersome. Another issue is the inability to provide recipe suggestions or alert notifications that take into account the user's emotional state. The objective of this invention is to solve these issues and achieve health maintenance, reduce the burden of housework, and reduce food waste.

[1586] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1587] In this invention, the server includes an image acquisition means for capturing images of food in the refrigerator, an image analysis means for analyzing images captured by the image acquisition means to recognize the food, a recipe suggestion means for suggesting currently available recipes based on the recognized food information, a nutrition management means for calculating the nutrients contained in each meal and managing a nutritionally balanced diet, a food waste prevention means for estimating expiration dates based on food purchase date information and generating alerts for food approaching its expiration date, and an emotion analysis means for analyzing user emotion data and adjusting the content and timing of recipe suggestions and alerts based on the user's emotional state. This allows for optimized refrigerator food management while taking user emotions into consideration, thereby reducing the burden of health management and housework and reducing food waste.

[1588] "Image acquisition means" refers to a camera and associated equipment for photographing food in the refrigerator.

[1589] "Image analysis means" refers to algorithms or software for analyzing the images obtained by the image acquisition means and recognizing food.

[1590] "Recipe suggestion means" refers to a system or method for suggesting available recipes based on recognized food information.

[1591] "Nutrition management measures" refers to systems and methods for calculating the nutrients contained in each meal and managing nutritional balance.

[1592] "Food waste prevention measures" refers to systems and methods for estimating expiration dates based on food purchase date information and generating alerts for food that is approaching its expiration date.

[1593] "Emotion analysis means" refers to a system or method for analyzing a user's emotional data and adjusting the content and timing of recipe suggestions and alerts based on the user's emotional state.

[1594] "Notification means" refers to a system or method for sending recipe suggestions and alert notifications to a user terminal.

[1595] "Report generation means" refers to a system or method for creating a report based on the results of the nutritional balance analysis and presenting it to the user.

[1596] This system efficiently manages food in the refrigerator, helping maintain health, reduce the burden of housework, and reduce food waste. Its unique feature is its ability to suggest recipes and manage nutrition based on the user's emotions.

[1597] System configuration

[1598] The system includes the following main components:

[1599] 1. Image acquisition method:

[1600] A camera installed inside the refrigerator automatically captures images when the refrigerator is opened or closed.

[1601] 2. Image analysis methods:

[1602] The captured images are analyzed and food is recognized, specifically using Google Vision API and other image analysis algorithms.

[1603] 3. Recipe suggestion methods:

[1604] Based on the recognized food information, the system suggests currently available recipes. The recipe information is stored in a database, and the system searches for recipes that match the user's ingredients.

[1605] 4. Nutritional Management Measures:

[1606] Calculates the nutrients contained in each meal and manages a nutritionally balanced diet. Calculates the content of various nutrients based on a food composition database.

[1607] 5. Food waste prevention measures:

[1608] The system estimates the expiration date based on the purchase date of the food item and generates an alert for food items approaching their expiration date. The alert is sent to a smartphone.

[1609] 6. Emotion analysis means:

[1610] Analyzes user emotional data and adjusts the content and timing of recipe suggestions and alerts based on the user's emotional state. Sentiment analysis is performed using machine learning models such as TensorFlow.

[1611] Example of operation

[1612] Ingredient recognition processing

[1613] The camera unit takes an image of the inside of the refrigerator when it is opened, and the terminal transmits the image data to a server via the Internet.

[1614] The server uses image analysis to recognize the food items and stores their information in a database, for example, three tomatoes, one carton of milk, and six eggs.

[1615] Recipe suggestion processing

[1616] A user requests a dinner recipe through a smartphone app, and the device sends the request to a server.

[1617] The server selects appropriate recipes based on food information and the user's emotional data. Using an emotional engine, it suggests dishes such as "tomato cream pasta" that have a relaxing effect to users who are under stress.

[1618] Nutritional balance management

[1619] The user enters the recipe they selected and the information about the meal they actually made into the app, and the device sends that information to the server.

[1620] The server calculates the nutrients contained in each meal and stores them in a database. It analyzes nutritional balance based on a week's diet history and identifies nutrient deficiencies. A report containing the results is sent to the user via the app.

[1621] Preventing food waste

[1622] The server estimates expiration dates based on the purchase date information of food items registered in the database, and generates an alert for food items approaching their expiration date and notifies the user's smartphone.

[1623] The alerts are tailored in content and timing based on emotional data, and if the user is feeling down, they'll be accompanied by an encouraging message.

[1624] Specific examples

[1625] I took a picture of the food in my refrigerator. I can see three tomatoes, one carton of milk, and six eggs. Can you suggest a dinner recipe? The user is stressed.

[1626] This system optimizes food management in the refrigerator while taking into consideration the user's emotions, making it possible to manage health and reduce the burden of housework, as well as reduce food waste.

[1627] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1628] Ingredient recognition processing

[1629] Step 1:

[1630] The camera means periodically takes images of the inside of the refrigerator.

[1631] Input: Interior image of a refrigerator when it is opened.

[1632] Output: Image data of the inside of the refrigerator.

[1633] What it does: The camera takes a picture as soon as the refrigerator door opens, capturing a photo of the food inside.

[1634] Step 2:

[1635] The terminal transmits the captured image data to a server via the Internet.

[1636] Input: Image data of the inside of a refrigerator.

[1637] Output: Image data sent to the server.

[1638] Specific operation: The device uploads image data to the server via Wi-Fi. Once the upload is complete, the server confirms receipt of the image.

[1639] Step 3:

[1640] The server uses image analysis means to recognize the food from the image.

[1641] Input: Image data of the inside of the refrigerator sent to the server.

[1642] Output: Food information in the refrigerator (e.g., 3 tomatoes, 1 carton of milk, 6 eggs).

[1643] Specific operation: The server uses the Google Vision API to analyze objects in the image, recognize food, and compile information about it.

[1644] Step 4:

[1645] The server stores the recognized food information in a database.

[1646] Input: Food information in the refrigerator.

[1647] Output: Food information stored in a database.

[1648] Specific operation: The server connects to a MySQL database and accurately stores the recognized food information.

[1649] Recipe suggestion processing

[1650] Step 1:

[1651] A user requests a dinner recipe via a smartphone app.

[1652] Input: A recipe request from a user.

[1653] Output: Request information from the smartphone app.

[1654] What it does: A user uses the app and taps or voice commands to say, "Suggest recipes for dinner."

[1655] Step 2:

[1656] The terminal sends a recipe request to the server.

[1657] Input: Request information from the smartphone app.

[1658] Output: The recipe request sent to the server.

[1659] Specific operation: The device sends the request information to the server via the Internet. It notifies the user that the transmission was successful.

[1660] Step 3:

[1661] The server retrieves the latest food information from the database.

[1662] Input: The recipe request sent to the server.

[1663] Output: Food information retrieved from the database.

[1664] What happens: The server runs an SQL query to pull food information from the database.

[1665] Step 4:

[1666] The server uses emotion analysis means to collect emotion data of the user.

[1667] Input: User emotion data when requesting a recipe.

[1668] Output: Analyzed user emotion data (e.g. stress level).

[1669] Specific operation: The emotion analysis means analyzes facial expressions and tone of voice through the smartphone's camera and microphone, and determines emotions using a TensorFlow model.

[1670] Step 5:

[1671] The server selects the optimal recipe based on food information and emotional data.

[1672] Input: Retrieved food information and parsed emotion data.

[1673] Output: The best recipe information to be suggested to the user.

[1674] Specific operation: The server uses an algorithm to select the recipe that best suits the food inventory and the user's emotions, and assembles the recipe information.

[1675] Step 6:

[1676] The server notifies the user of the selected recipe on their smartphone.

[1677] Input: Selected recipe information.

[1678] Output: Notification to the user's smartphone.

[1679] Specific operation: The server uses push notifications to send recipe information to the user's smartphone app.

[1680] Nutritional balance management

[1681] Step 1:

[1682] The user enters the recipe they selected and information about the meal they actually made into the app.

[1683] Input: Meal information entered by the user.

[1684] Output: Meal information sent to the server.

[1685] What it does: The app is designed to make it easy to record your meal information by scanning barcodes or entering your voice.

[1686] Step 2:

[1687] The terminal transmits the information to the server.

[1688] Input: Meal information entered by the user into the app.

[1689] Output: Meal information sent to the server.

[1690] Specific operation: The device sends the input information to the server via the Internet and notifies the user when the transmission is complete.

[1691] Step 3:

[1692] The server calculates the nutrients contained in each meal based on the input dietary information.

[1693] Input: Meal information sent to the server.

[1694] Output: Calculated nutrient information for each meal.

[1695] Specific operation: The server uses Python's pandas library to calculate the nutrients for each food.

[1696] Step 4:

[1697] The server stores the calculated nutrient information in a database.

[1698] Input: Calculated nutrient information.

[1699] Output: Nutrient information stored in a database.

[1700] Specific operation: The server records the nutrient information in a database and manages it for each user.

[1701] Step 5:

[1702] The server analyzes nutritional balance based on a week's dietary history and identifies any nutrient deficiencies.

[1703] Input: A week's worth of dietary history recorded in a database.

[1704] Output: Nutritional balance analysis results.

[1705] Specific operation: Analyzes dietary history using Python's scikit-learn library and evaluates nutritional balance.

[1706] Step 6:

[1707] The server creates a nutritional balance report and notifies the user through the app.

[1708] Input: Nutritional balance analysis results.

[1709] Output: Report notification to user.

[1710] Specific operation: The server creates a report based on the analysis results and sends a push notification to the app.

[1711] Preventing food waste

[1712] Step 1:

[1713] The server estimates the expiration date based on the purchase date information of the food product.

[1714] Input: Food purchase date information recorded in the database.

[1715] Output: Estimated expiration date.

[1716] What it does: Calculates expiration dates using Python's datetime library.

[1717] Step 2:

[1718] The server generates alerts for food items approaching their expiration date.

[1719] Input: Estimated expiration date information.

[1720] Output: Alert information.

[1721] Specific behavior: The server generates an alert message based on the expiration date and adds it to the alert list.

[1722] Step 3:

[1723] The server notifies the user of the generated alert on their smartphone.

[1724] Input: Alert information.

[1725] Output: Notification to the user's smartphone.

[1726] What it does: Sends an alert message to the user's smartphone app using push notification.

[1727] Examples of application of emotion analysis methods

[1728] Step 1:

[1729] While the user prepares dinner, the emotion analysis means analyzes facial expressions and tone of voice through the smartphone's camera and microphone.

[1730] Input: User's facial expression and voice data.

[1731] Output: Parsed emotion data.

[1732] How it works: The emotion analyzer analyzes the user's facial expressions and tone of voice in real time to assess their stress and happiness levels.

[1733] Step 2:

[1734] If the emotion analysis means determines that the user is feeling stressed, the server preferentially suggests recipes that use foods that have a relaxing effect.

[1735] Input: Parsed emotion data.

[1736] Output: Relaxing recipe suggestions.

[1737] What it does: The server searches the database for relaxing recipes and updates the list of suggestions.

[1738] Step 3:

[1739] When your diet is lacking in balance, the server will suggest foods that contain nutrients that help relieve stress.

[1740] Input: Nutritional balance analysis results and emotional data.

[1741] Output: Nutrition-conscious recipe suggestions.

[1742] What it does: Based on the missing nutritional information, the server recommends recipes that include foods that help relieve stress.

[1743] Step 4:

[1744] When food in the refrigerator is nearing its expiration date, the server will quickly alert the user if they are feeling down, along with an encouraging message.

[1745] Input: Emotion data and expiration date information.

[1746] Output: Emotion-sensitive alert notification.

[1747] What it does: The server adjusts the content and timing of alert notifications based on emotion data and adds encouraging messages.

[1748] (Application example 2)

[1749] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1750] Conventional refrigerator systems require complicated management of food items in the refrigerator and expiration dates, and do not adequately achieve health management for users, reduce the burden of housework, or reduce food waste. Furthermore, they do not suggest or manage food based on the user's emotional state, and do not provide detailed support to users. This results in insufficient management of users' health and efficiency in housework, and often leads to food waste.

[1751] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1752] In this invention, the server includes a camera for taking pictures of food in the refrigerator, an image analysis unit for analyzing the images taken by the camera and recognizing the food, a suggestion unit for suggesting currently available recipes based on the recognized food information and the user's emotional information, a nutrition calculation unit for calculating the nutrients of each food and managing a nutritionally balanced diet, a food consumption management unit for estimating expiration dates based on food purchase dates and generating alerts for food items approaching their expiration date, and an emotional response unit for providing messages and suggestions based on the user's emotional state. This improves the efficiency of refrigerator food management, reduces the user's health management, eases housework burden, and reduces food waste. Furthermore, by suggesting recipes and generating alerts based on the user's emotional state, the server provides detailed support and improves user satisfaction.

[1753] 1. "Photographing means" refers to a photographing device placed to photograph food items in a refrigerator.

[1754] 2. "Image analysis means" refers to a means that has the function of analyzing the image acquired by the photographing means and recognizing food.

[1755] 3. "Suggestion means" refers to a means that has the function of suggesting currently available recipes based on recognized food information and user emotion information.

[1756] 4. "Nutrition calculation tool" refers to a tool that has the function of calculating the nutrients of each food and managing a nutritionally balanced diet.

[1757] 5. "Food expiration date management means" refers to a means that has the function of estimating the expiration date based on the purchase date information of food and generating an alert for food that is approaching its expiration date.

[1758] 6. "Emotion response means" refers to means that have the function of sending messages or suggestions according to the user's emotional state.

[1759] DETAILED DESCRIPTION OF THE INVENTION

[1760] The following describes an embodiment of the present invention: The system mainly operates using a camera means built into the refrigerator, a server, and a user's smartphone or other user terminal.

[1761] First, a camera installed inside the refrigerator periodically takes images of the inside of the refrigerator. This image data is sent to a server via the Internet. The server then uses image analysis means (such as image analysis software) to recognize food from the images and store the food information in a database. Specific image analysis uses the widely used OpenCV library, etc.

[1762] Next, when the user requests a dinner recipe, the server retrieves the latest food information and generates the optimal recipe using a suggestion method (recipe suggestion algorithm). This also takes the user's emotional information into consideration. Emotion recognition software such as EmotionEngine is used as an emotional response method to analyze the user's emotions. Based on the ingredient data and the user's emotional state, a recipe such as "tomato cream pasta" is sent to the user's smartphone.

[1763] The nutritional calculation tool calculates the nutritional value of each recipe based on the recipe selected by the user and the information on the meal they have actually made. This information is recorded on a server, and a report is created that analyzes the nutritional balance based on a week's worth of meal history and identifies any nutrient deficiencies. The report is sent to the user via the app.

[1764] Furthermore, the food consumption period management means estimates the expiration date based on the purchase date information of the food, and generates an alert for food that is approaching its expiration date. This alert is also notified to the user at an appropriate time using the emotion response means.

[1765] For example, for milk that is nearing its expiration date, a specific alert could be generated such as "Your milk is nearing its expiration date. Please consume it soon," accompanied by an uplifting message that reflects the user's emotional state.

[1766] Examples:

[1767] 1. Images taken by the refrigerator camera are periodically sent to the server.

[1768] 2. Using image analysis software (e.g., OpenCV), the food items in the refrigerator are recognized. For example, data such as "3 tomatoes, 1 carton of milk, and 6 eggs" is obtained.

[1769] 3. When a user requests a dinner recipe on their smartphone, the server uses the Emotion Engine to analyze the user's emotional information (e.g., stress level).

[1770] 4. Based on the user’s emotions and the food information in the refrigerator, the suggestion tool will suggest a recipe for “Tomato Cream Pasta” and notify the smartphone.

[1771] 5. The nutrition calculation means calculates the nutrients in the tomato cream pasta and creates a weekly nutritional balance report based on the user's dietary history.

[1772] 6. An alert will be generated for food products (such as milk) that are nearing their expiration date, and a notification will be sent to your smartphone saying, "Your milk is nearing its expiration date. Please consume it as soon as possible."

[1773] Example prompt sentence:

[1774] "Analyze images of the inside of the refrigerator and tell me the current food inventory."

[1775] "Please suggest a relaxing recipe based on my mood."

[1776] "Get alerts for foods that are nearing their expiration date."

[1777] This system will improve the efficiency of food management in the refrigerator, enabling users to manage their health, reduce household chores, and reduce food waste. It will also provide detailed support by suggesting recipes and generating alerts based on the user's emotional state, improving user satisfaction.

[1778] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1779] Specific operation steps:

[1780] Step 1:

[1781] The refrigerator's camera periodically captures images of the inside of the refrigerator, and the server receives this image data via the Internet.

[1782] Input: Image data of the inside of the refrigerator

[1783] Data processing: Image data is sent to a server via the Internet

[1784] Output: Image data received by the server

[1785] Step 2:

[1786] The server uses image analysis means (image analysis software) to analyze the received image data and identify the food items in the refrigerator. The information about the identified food items is stored in a database.

[1787] Input: Image data received by the server

[1788] Data analysis: Recognize food using image analysis software (e.g., OpenCV)

[1789] Output: Recognized food information (e.g., 3 tomatoes, 1 carton of milk, 6 eggs)

[1790] Step 3:

[1791] A user requests a dinner recipe via a device such as a smartphone. This request is sent to the server, which then retrieves the latest information about the food in the refrigerator based on the recognized food information.

[1792] Input: User recipe request

[1793] Data acquisition: Acquire the latest refrigerator food information from the database

[1794] Output: Latest information on food in the refrigerator

[1795] Step 4:

[1796] The server uses emotion response means (emotion recognition software) to acquire the user's emotion data from the camera and microphone of the smartphone or terminal in order to analyze the user's emotion information.

[1797] Input: User's camera video and audio data

[1798] Data analysis: Analyze emotions using emotion recognition software (e.g., EmotionEngine)

[1799] Output: User's emotional state (e.g., stress state)

[1800] Step 5:

[1801] The server uses a suggestion means (recipe suggestion algorithm) to select the optimal recipe based on the recognized food information and the user's emotional information, and notifies the user of the recipe to their smartphone.

[1802] Input: Recognized food information, user emotion information

[1803] Data processing: Applying recipe suggestion algorithm

[1804] Output: Notification of a suggested recipe (e.g., "Tomato Cream Pasta")

[1805] Step 6:

[1806] The server records the recipe information that the user selects and actually makes, and the server then calculates the nutritional content of each meal based on this information.

[1807] Input: User-generated meal information

[1808] Data Analysis: Nutrient Calculations

[1809] Output: Nutrition information for each meal

[1810] Step 7:

[1811] The server uses a nutritional calculation tool to analyze nutritional balance based on a week's worth of dietary history and creates a report identifying nutrient deficiencies, which is then sent to the user via the app.

[1812] Input: One week's worth of food history

[1813] Data analysis: Nutritional balance analysis

[1814] Output: Nutritional balance report creation and notification

[1815] Step 8:

[1816] The server uses the food consumption period management means to estimate the expiration date based on the purchase date information of the food, and generates an alert for food that is approaching its expiration date. The alert is sent to the smartphone with appropriate timing and content according to the user's emotional state using the emotional response means.

[1817] Input: Food purchase date information

[1818] Data analysis: Estimating expiration dates and considering emotional data

[1819] Output: Alert notification for food products that are nearing their expiration date (e.g. "The milk is nearing its expiration date. Please consume it as soon as possible.")

[1820] Example prompt sentence:

[1821] "Analyze images of the inside of the refrigerator and tell me the current food inventory."

[1822] "Please suggest a relaxing recipe based on my mood."

[1823] "Get alerts for foods that are nearing their expiration date."

[1824] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1825] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1826] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1827] [Fourth embodiment]

[1828] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1829] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1830] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1831] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1832] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1833] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1834] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1835] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1836] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1837] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1838] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1839] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1840] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1841] The system of the present invention has a built-in camera, image analysis, recipe suggestion, nutrition management, and food waste prevention means. By using this system, food in the refrigerator can be efficiently managed, and social issues such as preventive medicine, reducing the burden of housework, and preventing food waste can be addressed.

[1842] The camera means periodically photographs the food in the refrigerator and sends the image data to the server. The server analyzes the received images using the image analysis means and recognizes the food. The recognized food information is stored in a database, and the recipe suggestion means suggests appropriate recipes to the user based on this information. The nutrition management means also calculates the nutritional value of the food and performs nutritional management to provide a balanced diet.

[1843] Furthermore, the food loss prevention system estimates the expiration date based on the purchase date of the food, generates an alert for food that is approaching its expiration date, and notifies the user's device, thereby preventing food waste.

[1844] The specific processing contents of the program will be explained in natural language below.

[1845] Ingredient recognition processing

[1846] The server receives the image taken by the camera means and analyzes the image using the image analysis means. For example, if there are tomatoes, milk, and eggs on the shelves of a refrigerator, the server identifies these foods and stores them in the database as "3 tomatoes, 1 carton of milk, 6 eggs."

[1847] Recipe suggestion processing

[1848] When a user requests a dinner recipe, the server retrieves information about the food in the refrigerator from the database and searches the recipe database. For example, if there are tomatoes and milk in the refrigerator, it will suggest a recipe for tomato cream pasta. This suggestion is sent to the user's smartphone, and the user can check the cooking instructions and other ingredients needed through the app.

[1849] Nutritional balance management

[1850] To record the meals a user has eaten, the server collects information on suggested recipes and the meals actually cooked, allowing the system to calculate how much protein, carbohydrates, and fat a user has consumed in a week and notify them of any nutrient deficiencies.

[1851] For example, if a user's protein intake is low over the course of a week, the server will generate a report saying, "You're not getting enough protein. Try some high-protein recipes."

[1852] Preventing food waste

[1853] The server manages food expiration dates using information on food purchase dates and an expiration date estimation algorithm. For example, if milk's expiration date is approaching in three days, the server generates an alert saying, "The milk's expiration date is approaching. Please consume it soon," and notifies the user's smartphone.

[1854] As described above, the system of the present invention provides a specific method for efficiently managing food in a refrigerator, supporting health management, and reducing food waste.

[1855] The processing flow will be explained below.

[1856] Ingredient recognition processing

[1857] Step 1:

[1858] The camera takes pictures of the inside of the refrigerator.

[1859] The camera means periodically takes an image of each shelf in the refrigerator, and generates image data.

[1860] Step 2:

[1861] The image data is sent to a server.

[1862] The image data acquired by the camera means is transmitted to a server via the Internet.

[1863] Step 3:

[1864] The server analyzes the image data.

[1865] The server uses image analysis means to recognize foods from the image, for example, the server identifies tomatoes, milk, and eggs from the image.

[1866] Step 4:

[1867] The recognized food information is stored in a database.

[1868] The server stores the recognition result "3 tomatoes, 1 carton of milk, 6 eggs" in a database.

[1869] Recipe suggestion processing

[1870] Step 1:

[1871] A user requests recipe suggestions.

[1872] A user requests dinner recipe suggestions on a smartphone app.

[1873] Step 2:

[1874] The server acquires information about the food in the refrigerator.

[1875] The server retrieves the latest food information from the database.

[1876] Step 3:

[1877] The server searches the recipe database.

[1878] The server searches for available recipes based on the food information in the refrigerator. For example, if there are tomatoes and milk, it will extract the recipe for "Tomato Cream Pasta."

[1879] Step 4:

[1880] The server creates recipe suggestions.

[1881] The server creates a recipe suggestion that includes ingredients, cooking instructions, and nutritional information.

[1882] Step 5:

[1883] Recipe suggestions are notified to the device.

[1884] The server notifies the user of the recipe suggestions on their smartphone, which are then displayed within the app.

[1885] Nutritional balance management

[1886] Step 1:

[1887] Record the user's dietary history.

[1888] The app records the recipes the user has cooked and sends that information to the server.

[1889] Step 2:

[1890] The server calculates the nutrients.

[1891] The server calculates the nutrients contained in each meal and stores them in a database.

[1892] Step 3:

[1893] The server analyzes the nutritional balance.

[1894] The server analyzes nutritional balance based on dietary history over a certain period (e.g., one week) and identifies any nutrient deficiencies or excess intake.

[1895] Step 4:

[1896] Create a report.

[1897] The server generates a detailed report based on the results of the nutritional balance analysis, including, for example, "You are lacking in protein."

[1898] Step 5:

[1899] The report is sent to the device.

[1900] The server sends the created report to the user's smartphone, where the user can check the report contents through the app.

[1901] Preventing food waste

[1902] Step 1:

[1903] The server estimates the expiration date.

[1904] The server estimates the expiration date based on the purchase date of the food product. For example, it calculates the expiration date of milk.

[1905] Step 2:

[1906] Manage expiration date information.

[1907] The server manages the estimated expiration dates in a database and sets an alert flag when the expiration date approaches.

[1908] Step 3:

[1909] Generate an alert.

[1910] The server generates alerts for food products that are approaching their expiration date, for example, "Milk expires tomorrow."

[1911] Step 4:

[1912] Sends alerts to the device.

[1913] The server sends an alert to the user's smartphone, which is then displayed within the app.

[1914] Specific examples

[1915] 1. If there are tomatoes, milk, or eggs in the refrigerator, the camera means takes a picture of them and sends the image to the server.

[1916] 2. The server uses image analysis to recognize three tomatoes, one carton of milk, and six eggs, and stores them in a database.

[1917] 3. When a user requests a recipe for dinner, the server retrieves information about the food in the refrigerator, searches for available recipes, and suggests "Tomato Cream Pasta."

[1918] 4. When the user cooks a recipe and records their meal history in the app, the server calculates the nutrients and analyzes a week's worth of meal history to check nutritional balance.

[1919] 5. When the milk is nearing its expiration date, the server estimates the expiration date and generates an alert to notify the user.

[1920] The above is a specific processing flow using the system of the present invention.

[1921] Example 1

[1922] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1923] In modern society, there is a demand for both labor-saving and health management. However, food management is complicated and time-consuming. In particular, properly managing food in the refrigerator and consuming it while it is fresh is effective in preventing food waste, but manual management is difficult and not often implemented. It is also difficult to provide a balanced diet that takes into account the nutritional value of each food item. Therefore, there is a need for a system that can efficiently manage food in the refrigerator.

[1924] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1925] In this invention, the server includes a camera for capturing images of objects in the refrigerator, an analysis unit for analyzing the images captured by the camera to recognize the objects, a suggestion unit for proposing currently available cooking procedures based on the recognized object information, a management unit for calculating the nutritional content of each object and managing a nutritionally balanced diet, and a deterioration prevention unit for estimating the expiration date of each object based on the object's acquisition date information and generating a notification for objects approaching their expiration date. This allows for efficient and automatic management of food in the refrigerator and encourages consumption at the appropriate time. Furthermore, balanced meal suggestions and nutritional management can support a healthy lifestyle.

[1926] The "photography means" is a device that is installed inside the refrigerator and takes images of food or objects.

[1927] The "analysis means" is a system that analyzes the image captured by the imaging means and recognizes the type of food or object.

[1928] The "suggestion means" is a mechanism that provides currently available cooking procedures to the user based on the information about the food or object recognized by the analysis means.

[1929] The "management means" is a system for calculating the nutritional content of foods and objects and managing a balanced diet.

[1930] The "deterioration prevention means" is a mechanism that estimates the expiration date of a food or object based on the acquisition date information, and generates a notification to the user when the expiration date approaches.

[1931] The system of the present invention efficiently manages food in a refrigerator, enabling health management, reducing the burden of housework, and preventing food waste. This system includes a photographing means, an analyzing means, a suggestion means, a management means, and a deterioration prevention means.

[1932] 1. Photography Method

[1933] The imaging means is a camera installed inside the refrigerator that periodically captures images of food and objects. For example, the camera takes a picture of the shelves inside the refrigerator every hour and sends the image data to a server. The specific hardware used is a high-resolution digital camera.

[1934] 2. Analysis method

[1935] The analysis means is software installed on the server that analyzes the images sent by the image capture means. This analysis uses an object recognition algorithm, such as TensorFlow or OpenCV. The analysis means automatically identifies and identifies objects in the image. For example, if there are tomatoes, milk, and eggs on a shelf in a refrigerator, the analysis means will recognize them and store them in the database as "3 tomatoes, 1 carton of milk, 6 eggs."

[1936] 3. Proposal method

[1937] When a user requests a dinner recipe, the suggestion means retrieves information about the food in the refrigerator from the database and searches the recipe database. For example, if there are tomatoes and milk in the refrigerator, the suggestion means selects a recipe for "tomato cream pasta" and notifies the user's smartphone. The specific software used is a database search algorithm using Python or Java.

[1938] 4. Control measures

[1939] The management means collects information about the meals the user has eaten and calculates the amount of nutrients the user has consumed for the week. This analyzes how much protein, carbohydrates, and fat the user has consumed and notifies the user of any nutrient deficiencies. For example, if the user's protein intake is low for the week, a report such as "You are lacking in protein. Please try a high-protein recipe" is generated and the user is notified.

[1940] 5. Measures to prevent deterioration

[1941] The deterioration prevention method estimates the expiration date based on the food's acquisition date and generates a notification for food that is approaching its expiration date. For example, if the expiration date of milk is approaching in three days, an alert will be generated and sent to the user's smartphone saying, "The milk's expiration date is approaching. Please consume it soon."

[1942] Example of a concrete example and prompt for the generative AI model:

[1943] When an image of the inside of a refrigerator is taken by the photographing means and the analyzing means recognizes that there are "3 tomatoes, 1 carton of milk, and 6 eggs," the specific processing will be as follows:

[1944] Example prompt sentence:

[1945] "Analyze images of the inside of the refrigerator and identify what's inside."

[1946] Input image: 3 tomatoes, 1 carton of milk, 6 eggs

[1947] What happens when a user asks for dinner recipes?

[1948] Example prompt sentence:

[1949] "I have some tomatoes and milk in the fridge. Can you suggest a recipe using them?"

[1950] This system will enable efficient management of food in the refrigerator and encourage consumption at the appropriate time. Furthermore, it will support a healthy lifestyle by suggesting balanced meals and managing nutrition.

[1951] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1952] Step 1:

[1953] Image acquisition

[1954] The server receives an image of the inside of the refrigerator using the photographing means.

[1955] Input: Image data taken inside the refrigerator

[1956] Output: Image data sent to the server

[1957] Specific operation: The camera built into the refrigerator takes pictures every hour and sends them to the server.

[1958] Step 2:

[1959] Image analysis

[1960] The server analyzes the received image using the image analysis means.

[1961] Input: Image data sent to the server

[1962] Output: Analyzed object information

[1963] Data processing and calculation: Using an object recognition algorithm (e.g., TensorFlow, OpenCV), identify the type and number of identified objects.

[1964] Specific operation: The server identifies objects in the image and obtains information such as "3 tomatoes, 1 carton of milk, and 6 eggs."

[1965] Step 3:

[1966] Data storage

[1967] The server stores the analysis results (object information) in a database.

[1968] Input: Parsed object information

[1969] Output: Object information stored in a database

[1970] Specific operation: The server writes information such as "3 tomatoes, 1 carton of milk, 6 eggs" into the database.

[1971] Step 4:

[1972] Recipe request reception

[1973] The server receives a recipe request from a user terminal.

[1974] Input: User request data

[1975] Output: Processing based on the request begins

[1976] Specific Action: A user requests a dinner recipe through a smartphone app.

[1977] Step 5:

[1978] Food information acquisition

[1979] The server obtains information about the food items in the refrigerator from the database.

[1980] Input: A query to the database

[1981] Output: Food information

[1982] Specific operation: The server retrieves food information from the database, such as "3 tomatoes, 1 carton of milk, 6 eggs."

[1983] Step 6:

[1984] Recipe Search

[1985] The server uses the suggestion means to search a recipe database based on the food information.

[1986] Input: Food information

[1987] Output: appropriate recipe information

[1988] Data processing and calculation: Matching food information with the recipe database to select the optimal recipe.

[1989] Specific operation: The server searches the recipe database and selects the recipe for "Tomato Cream Pasta."

[1990] Step 7:

[1991] Recipe Notifications

[1992] The server notifies the user terminal of the selected recipe.

[1993] Input: appropriate recipe information

[1994] Output: Recipe information sent to the user's device

[1995] Specific operation: The server sends the recipe for "Tomato Cream Pasta" to the user's smartphone, allowing the user to view it in the app.

[1996] Step 8:

[1997] Gathering dietary information

[1998] The server collects meal information from the user.

[1999] Input: User's food record data

[2000] Output: Meal data stored on the server

[2001] Specific operation: The user enters a meal record into the app, and the server receives it.

[2002] Step 9:

[2003] Nutrition Calculations

[2004] The server uses the management means to calculate the intake of nutrients from the dietary data for one week.

[2005] Input: Meal data

[2006] Output: Nutritional balance report

[2007] Data processing and calculation: Analyze dietary data and calculate protein and carbohydrate intake.

[2008] Specific operation: The server creates a report based on the user's weekly nutritional intake.

[2009] Step 10:

[2010] Nutrition Report Notification

[2011] The server notifies the user terminal of a report based on the nutrients that are lacking.

[2012] Input: Nutritional Balance Report

[2013] Output: Nutrition report sent to user device

[2014] Specific operation: The server sends a message to the user's smartphone saying, "You are lacking in protein. Please try a high-protein recipe."

[2015] Step 11:

[2016] Expiration date estimation

[2017] The server estimates the expiration date based on the purchase date information of the food product using the deterioration prevention means.

[2018] Input: Food purchase date information

[2019] Output: Estimated expiration date information

[2020] Data processing and calculation: The expiration date is calculated using the purchase date information and an algorithm.

[2021] Specific operation: The server estimates that the milk's expiration date is approaching in three days.

[2022] Step 12:

[2023] Expiration date alert generation

[2024] The server generates alerts for food items approaching their expiration date.

[2025] Input: Estimated expiration date information

[2026] Output: The generated alert message

[2027] Specific behavior: The server creates an alert message saying, "The milk's expiration date is approaching. Please consume it soon."

[2028] Step 13:

[2029] Expiration date alert notification

[2030] The server notifies the user terminal of the generated alert.

[2031] Input: The generated alert message

[2032] Output: The alert message sent to the user's terminal.

[2033] Specific operation: The server sends an alert to the user's smartphone, which the user can check in the app.

[2034] (Application example 1)

[2035] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2036] In modern society, food management, nutritional management, and food waste prevention at home and in brick-and-mortar stores are important issues. There is a need for a system that efficiently manages food not only at home but also in brick-and-mortar stores, and helps consumers purchase appropriate and balanced ingredients. The existence of such a system would greatly contribute to consumer health management and food waste reduction. Another important issue is reducing the financial burden on consumers by providing real-time sales information.

[2037] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[2038] In this invention, the server includes a camera means for photographing food in the refrigerator, an image analysis means for analyzing images photographed by the camera means to recognize food, a recipe suggestion means for suggesting currently available recipes based on the recognized food information, a nutrition management means for calculating the nutrients of each food and managing a nutritionally balanced diet, a food waste prevention means for estimating expiration dates based on food purchase date information and generating an alert for food approaching its expiration date, a camera means for photographing food in a physical store, an image analysis means for analyzing images photographed by the camera means in the physical store to recognize food, a shopping list generation means for generating and notifying a shopping list based on the recognized food information in the physical store, a recipe suggestion means for suggesting and notifying recipes based on food available in the physical store, and a sale information notification means for recognizing and notifying sale information in the physical store. This enables efficient food management at home and in physical stores, providing nutritionally balanced meals, preventing food waste, and reducing economic burdens.

[2039] The "camera means" is a device for taking pictures of food in a refrigerator and in a physical store.

[2040] The "image analysis means" is a device for analyzing the image captured by the camera means and recognizing food.

[2041] The "recipe suggestion means" is a device for suggesting currently available recipes based on recognized food information.

[2042] The "nutritional management means" is a device that calculates the nutrients of each food and manages a nutritionally balanced diet.

[2043] The "food waste prevention device" is a device that estimates the expiration date based on the purchase date of food and generates an alert for food that is approaching its expiration date.

[2044] "Camera means in a physical store" refers to a device for photographing food in a physical store.

[2045] The "image analysis means in the physical store" is a device for recognizing food by analyzing images taken by a camera means in the physical store.

[2046] The "shopping list generation means" is a device for generating and notifying a shopping list based on the food information in the recognized physical store.

[2047] The "in-store recipe suggestion means" is a device for suggesting and notifying recipes based on food items that can be purchased in the store.

[2048] The "sale information notification means" is a device for recognizing and notifying sale information within a physical store.

[2049] The system of the present invention efficiently manages food in refrigerators and brick-and-mortar stores, and solves social issues such as preventive medicine, reducing the burden of housework, and preventing food waste. The system includes a camera, image analysis, recipe suggestion, nutritional management, and food waste prevention, as well as brick-and-mortar store-exclusive camera, image analysis, shopping list generation, in-store recipe suggestion, and sale information notification.

[2050] Hardware and software used:

[2051] Camera means: A camera installed in a refrigerator or in a physical store, which is a device for taking pictures of food.

[2052] Image analysis means: A device that analyzes captured images and recognizes the type of food, using common computer vision software such as OpenCV.

[2053] Recipe suggestion unit: A device that searches for possible recipe suggestions based on the recognized food information and notifies the user. It also works in conjunction with a recipe database.

[2054] Nutrition management tool: A device that calculates the nutritional value of food and supports balanced diet management. It uses an AI-based nutrition analysis tool.

[2055] Food waste prevention method: A device that estimates the expiration date of food from the purchase date information and generates an alert to notify the user when food is approaching its expiration date.

[2056] In-store camera means: A device installed in a physical store that takes pictures of food products being sold.

[2057] In-store image analysis means: A device that analyzes images taken by camera means in a physical store and recognizes the food being sold.

[2058] Shopping list generation means: A device that generates a shopping list based on the recognized food information in the physical store, along with the current refrigerator inventory, and notifies the user.

[2059] In-store recipe suggestion means: A device that suggests recipes based on food items available for purchase in a physical store and notifies the user.

[2060] Sale information notification means: A device that analyzes sale information in physical stores and notifies users.

[2061] Example of a system:

[2062] The system constantly monitors food inventory in the refrigerator and automatically generates a shopping list as needed. For example, if there is only one liter of milk, two tomatoes, and three peppers left in the refrigerator, the refrigerator's camera captures this information and analyzes it using image analysis. Based on this information, the recipe suggestion module suggests recipes using these foods to the user and automatically adds any missing ingredients to the shopping list.

[2063] When a user visits a physical store, a camera in the store takes a photo of the food on the shelves, and the image analysis means analyzes the information and notifies the user in real time. Furthermore, sale information is also analyzed, and economical sale items are suggested. In this way, users can comfortably and efficiently purchase the food they need.

[2064] Example prompt sentence:

[2065] "The current inventory in your refrigerator is:

[2066] 1 liter of milk

[2067] 2 tomatoes

[2068] 3 bell peppers

[2069] In-store specials include:

[2070] 2 tomatoes 100 yen

[2071] 20% off 1 pack of milk

[2072] Use this to generate recipe recommendations and shopping lists."

[2073] This invention allows for food management, nutritional balance and economical shopping at home and in physical stores.

[2074] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[2075] Step 1:

[2076] The server periodically takes pictures of the food using a camera in the refrigerator. The input is the image obtained from the camera, and the output is an image file of the photographed food. This image file is then handed over to the next analysis process.

[2077] Step 2:

[2078] The server uses image analysis means to analyze the photographed food image and recognize the type and quantity of food. The input is the photographed image file, and the output is a list of the recognized food types and quantities. This list is stored in a database.

[2079] Step 3:

[2080] A user sends a recipe request using a smartphone. The input is the user's request, and the output is a list of suitable recipe candidates produced by the recipe suggestion means. This candidate list is then sent to the user's terminal.

[2081] Step 4:

[2082] The server uses the nutrition management tool to calculate the nutritional value of each food item based on the proposed recipe. The input is the ingredient information of the proposed recipe, and the output is a list of the nutrients contained in each recipe. This list is saved as the user's health management information and can be viewed by the user on their smartphone.

[2083] Step 5:

[2084] The server uses a food waste prevention method to estimate expiration dates based on food purchase date information and identify foods that are approaching their expiration date. The input is the food purchase date information, and the output is a list of foods that are approaching their expiration date. This list is notified to the user's device as an alert.

[2085] Step 6:

[2086] The server takes pictures of food items on shelves using a camera in the physical store and analyzes them using an image analysis means. The input is the image obtained from the camera in the physical store, and the output is a list of the types and quantities of recognized food items. This list is sent to the physical store management system.

[2087] Step 7:

[2088] The server compares the recognized food information in the physical store with the user's refrigerator inventory and generates and notifies the user of a shopping list containing the necessary foods. The input is the food information in the physical store and the refrigerator inventory information, and the output is the generated shopping list. This list is notified to the user's smartphone.

[2089] Step 8:

[2090] The server proposes recipes based on information about food items available in physical stores and notifies the user's device. The input is information about food items in the store and inventory in the refrigerator, and the output is a list of suggested recipes. This list is then sent to the user's smartphone.

[2091] Step 9:

[2092] The server analyzes sales information in physical stores and notifies the user. The input is sales information, and the output is a list of foods on sale. This list is sent to the user's smartphone to assist them in their shopping.

[2093] This series of processing steps allows users to efficiently manage and purchase food, enabling them to maintain a nutritionally balanced lifestyle.

[2094] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[2095] The present invention is a system that has a camera means, image analysis means, recipe suggestion means, nutritional management means, food waste prevention means, and emotion engine means built into a refrigerator. This system efficiently manages food in the refrigerator, helping to maintain health, reduce the burden of housework, and reduce food waste, as well as suggesting recipes and managing nutrition based on the user's emotions.

[2096] Ingredient recognition processing

[2097] The camera means periodically takes pictures of the inside of the refrigerator and sends the image data to a server via the Internet. The server uses image analysis means to recognize food from the images and stores information about the food (e.g., 3 tomatoes, 1 carton of milk, 6 eggs) in a database.

[2098] Recipe suggestion processing

[2099] When a user requests a dinner recipe, the server retrieves the latest food information and searches for available recipes. The emotion engine also collects the user's emotion data (e.g., stress, joy). The server selects an appropriate recipe based on this emotion data and notifies the user's smartphone of a recipe such as "tomato cream pasta."

[2100] Nutritional balance management

[2101] The recipes selected by the user and information about the meals they actually made are recorded on the server. Based on this, the server calculates the nutrients contained in each meal and stores the data in a database. It analyzes nutritional balance based on the dietary history over a certain period (for example, one week) and identifies any nutrient deficiencies. At this time, it also takes into account data from the emotion engine and creates a detailed report, such as "On days with high stress, the required intake of vitamin C is low." The report is notified to the user via the app.

[2102] Preventing food waste

[2103] The server estimates the expiration date based on the food's purchase date information and generates an alert for food that is approaching its expiration date. For example, if the expiration date of milk is approaching in three days, the server creates an alert saying, "The milk's expiration date is approaching. Please consume it soon," and sends it to the user's smartphone. The content and timing of the alert can be adjusted to suit the user's emotional state using data from the emotion engine means.

[2104] Specific processing examples of emotion engine means

[2105] 1. While the user is preparing dinner, the emotion engine means analyzes facial expressions and tone of voice through the smartphone's camera and microphone.

[2106] 2. If the emotion engine means determines that the user is feeling stressed, the server will preferentially suggest recipes that use foods that have a relaxing effect.

[2107] 3. When a meal lacks balance, the server will include in the suggestions foods that contain nutrients that help relieve stress.

[2108] 4. When food in the refrigerator is nearing its expiration date, the server will quickly alert the user if they are feeling down, along with an encouraging message.

[2109] The above is a specific processing flow using the system of the present invention. This system optimizes food management in the refrigerator while taking into consideration the user's feelings, making it possible to manage health and reduce the burden of housework, as well as reducing food waste.

[2110] The processing flow will be explained below.

[2111] Ingredient recognition processing

[2112] Step 1:

[2113] The camera takes pictures of the inside of the refrigerator.

[2114] A camera means periodically takes images of the food shelves in the refrigerator.

[2115] Step 2:

[2116] The image data is sent to a server.

[2117] Image data captured by the camera means is transmitted to a server via the Internet.

[2118] Step 3:

[2119] The server analyzes the image data.

[2120] The server uses image analysis to recognize food items from the image, for example, distinguishing between tomatoes, milk, and eggs.

[2121] Step 4:

[2122] The recognized food information is stored in a database.

[2123] The server stores the recognition results (3 tomatoes, 1 carton of milk, and 6 eggs) in a database.

[2124] Recipe suggestion processing

[2125] Step 1:

[2126] A user requests recipe suggestions.

[2127] A user requests a dinner recipe on a smartphone app.

[2128] Step 2:

[2129] The server acquires information about the food in the refrigerator.

[2130] The server retrieves current food information from the database.

[2131] Step 3:

[2132] The emotion engine analyzes the user's emotions.

[2133] The emotion engine means analyzes the user's facial expressions and tone of voice through the smartphone's camera and microphone to determine the user's current emotional state.

[2134] Step 4:

[2135] The server searches the recipe database.

[2136] The server searches for appropriate recipes based on the food information in the refrigerator and the user's emotional data. For example, if the user is feeling stressed, it will select recipes using foods that have a relaxing effect.

[2137] Step 5:

[2138] The server creates recipe suggestions.

[2139] Servers create recipe suggestions that include required ingredients, cooking instructions, nutritional information, and emotionally sensitive comments.

[2140] Step 6:

[2141] Recipe suggestions are notified to the device.

[2142] The server notifies the user's smartphone of the recipe suggestions and displays detailed information within the app.

[2143] Nutritional balance management

[2144] Step 1:

[2145] Record the user's dietary history.

[2146] The app records the recipes the user has cooked and sends that information to the server.

[2147] Step 2:

[2148] The server calculates the nutrients.

[2149] The server calculates the nutrients contained in each meal and stores them in a database.

[2150] Step 3:

[2151] The server analyzes the nutritional balance.

[2152] The server analyzes the user's nutritional balance based on their dietary history over a certain period of time (for example, one week).

[2153] Step 4:

[2154] The server generates the report.

[2155] The server creates a detailed report based on the results of the nutritional balance analysis, taking into account the user's emotional data.

[2156] Step 5:

[2157] The report is sent to the device.

[2158] The server sends a detailed report to the user's smartphone, where the user can check the report using the app.

[2159] Preventing food waste

[2160] Step 1:

[2161] The server estimates the expiration date.

[2162] The server estimates the expiration date of each food item based on the purchase date information of the food item.

[2163] Step 2:

[2164] Save expiration date information in a database.

[2165] The server registers and manages the estimated expiration date in a database.

[2166] Step 3:

[2167] Generate an alert.

[2168] The server generates an alert message for food products that are approaching their expiration date.

[2169] Step 4:

[2170] Sends alerts to the device.

[2171] The server sends an alert to the user's smartphone, informing them that the expiration date is approaching.

[2172] Specific processing of emotion engine means

[2173] Step 1:

[2174] Collect user emotion data.

[2175] The emotion engine means uses the smartphone's camera and microphone to collect the user's facial expressions, tone of voice, and other emotion indicators.

[2176] Step 2:

[2177] Analyze emotion data.

[2178] Based on the data collected by the emotion engine means, the user's current emotional state is analyzed and determined.

[2179] Step 3:

[2180] The determined emotion is sent to the server.

[2181] The emotion engine means transmits the analysis results to the server.

[2182] Step 4:

[2183] Adjust recipes and alerts based on sentiment data.

[2184] The server takes emotional data into consideration and adjusts the content and timing of recipe suggestions and alerts to provide users with the most appropriate information.

[2185] The above is a specific processing flow using the system of the present invention. By combining it with an emotion engine means, recipe suggestions and food waste prevention that take into account the user's emotional state can be realized, further improving the user experience.

[2186] Example 2

[2187] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2188] Managing food in the refrigerator poses the problem of making it difficult to understand expiration dates and manage nutritional balance, making household diet management, which relies on this, cumbersome. Another issue is the inability to provide recipe suggestions or alert notifications that take into account the user's emotional state. The objective of this invention is to solve these issues and achieve health maintenance, reduce the burden of housework, and reduce food waste.

[2189] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[2190] In this invention, the server includes an image acquisition means for capturing images of food in the refrigerator, an image analysis means for analyzing images captured by the image acquisition means to recognize the food, a recipe suggestion means for suggesting currently available recipes based on the recognized food information, a nutrition management means for calculating the nutrients contained in each meal and managing a nutritionally balanced diet, a food waste prevention means for estimating expiration dates based on food purchase date information and generating alerts for food approaching its expiration date, and an emotion analysis means for analyzing user emotion data and adjusting the content and timing of recipe suggestions and alerts based on the user's emotional state. This allows for optimized refrigerator food management while taking user emotions into consideration, thereby reducing the burden of health management and housework and reducing food waste.

[2191] "Image acquisition means" refers to a camera and associated equipment for photographing food in the refrigerator.

[2192] "Image analysis means" refers to algorithms or software for analyzing the images obtained by the image acquisition means and recognizing food.

[2193] "Recipe suggestion means" refers to a system or method for suggesting available recipes based on recognized food information.

[2194] "Nutrition management measures" refers to systems and methods for calculating the nutrients contained in each meal and managing nutritional balance.

[2195] "Food waste prevention measures" refers to systems and methods for estimating expiration dates based on food purchase date information and generating alerts for food that is approaching its expiration date.

[2196] "Emotion analysis means" refers to a system or method for analyzing a user's emotional data and adjusting the content and timing of recipe suggestions and alerts based on the user's emotional state.

[2197] "Notification means" refers to a system or method for sending recipe suggestions and alert notifications to a user terminal.

[2198] "Report generation means" refers to a system or method for creating a report based on the results of the nutritional balance analysis and presenting it to the user.

[2199] This system efficiently manages food in the refrigerator, helping maintain health, reduce the burden of housework, and reduce food waste. Its unique feature is its ability to suggest recipes and manage nutrition based on the user's emotions.

[2200] System configuration

[2201] The system includes the following main components:

[2202] 1. Image acquisition method:

[2203] A camera installed inside the refrigerator automatically captures images when the refrigerator is opened or closed.

[2204] 2. Image analysis methods:

[2205] The captured images are analyzed and food is recognized, specifically using Google Vision API and other image analysis algorithms.

[2206] 3. Recipe suggestion methods:

[2207] Based on the recognized food information, the system suggests currently available recipes. The recipe information is stored in a database, and the system searches for recipes that match the user's ingredients.

[2208] 4. Nutritional Management Measures:

[2209] Calculates the nutrients contained in each meal and manages a nutritionally balanced diet. Calculates the content of various nutrients based on a food composition database.

[2210] 5. Food waste prevention measures:

[2211] The system estimates the expiration date based on the purchase date of the food item and generates an alert for food items approaching their expiration date. The alert is sent to a smartphone.

[2212] 6. Emotion analysis means:

[2213] Analyzes user emotional data and adjusts the content and timing of recipe suggestions and alerts based on the user's emotional state. Sentiment analysis is performed using machine learning models such as TensorFlow.

[2214] Example of operation

[2215] Ingredient recognition processing

[2216] The camera unit takes an image of the inside of the refrigerator when it is opened, and the terminal transmits the image data to a server via the Internet.

[2217] The server uses image analysis to recognize the food items and stores their information in a database, for example, three tomatoes, one carton of milk, and six eggs.

[2218] Recipe suggestion processing

[2219] A user requests a dinner recipe through a smartphone app, and the device sends the request to a server.

[2220] The server selects appropriate recipes based on food information and the user's emotional data. Using an emotional engine, it suggests dishes such as "tomato cream pasta" that have a relaxing effect to users who are under stress.

[2221] Nutritional balance management

[2222] The user enters the recipe they selected and the information about the meal they actually made into the app, and the device sends that information to the server.

[2223] The server calculates the nutrients contained in each meal and stores them in a database. It analyzes nutritional balance based on a week's diet history and identifies nutrient deficiencies. A report containing the results is sent to the user via the app.

[2224] Preventing food waste

[2225] The server estimates expiration dates based on the purchase date information of food items registered in the database, and generates an alert for food items approaching their expiration date and notifies the user's smartphone.

[2226] The alerts are tailored in content and timing based on emotional data, and if the user is feeling down, they'll be accompanied by an encouraging message.

[2227] Specific examples

[2228] I took a picture of the food in my refrigerator. I can see three tomatoes, one carton of milk, and six eggs. Can you suggest a dinner recipe? The user is stressed.

[2229] This system optimizes food management in the refrigerator while taking into consideration the user's emotions, making it possible to manage health and reduce the burden of housework, as well as reduce food waste.

[2230] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2231] Ingredient recognition processing

[2232] Step 1:

[2233] The camera means periodically takes images of the inside of the refrigerator.

[2234] Input: Interior image of a refrigerator when it is opened.

[2235] Output: Image data of the inside of the refrigerator.

[2236] What it does: The camera takes a picture as soon as the refrigerator door opens, capturing a photo of the food inside.

[2237] Step 2:

[2238] The terminal transmits the captured image data to a server via the Internet.

[2239] Input: Image data of the inside of a refrigerator.

[2240] Output: Image data sent to the server.

[2241] Specific operation: The device uploads image data to the server via Wi-Fi. Once the upload is complete, the server confirms receipt of the image.

[2242] Step 3:

[2243] The server uses image analysis means to recognize the food from the image.

[2244] Input: Image data of the inside of the refrigerator sent to the server.

[2245] Output: Food information in the refrigerator (e.g., 3 tomatoes, 1 carton of milk, 6 eggs).

[2246] Specific operation: The server uses the Google Vision API to analyze objects in the image, recognize food, and compile information about it.

[2247] Step 4:

[2248] The server stores the recognized food information in a database.

[2249] Input: Food information in the refrigerator.

[2250] Output: Food information stored in a database.

[2251] Specific operation: The server connects to a MySQL database and accurately stores the recognized food information.

[2252] Recipe suggestion processing

[2253] Step 1:

[2254] A user requests a dinner recipe via a smartphone app.

[2255] Input: A recipe request from a user.

[2256] Output: Request information from the smartphone app.

[2257] What it does: A user uses the app and taps or voice commands to say, "Suggest recipes for dinner."

[2258] Step 2:

[2259] The terminal sends a recipe request to the server.

[2260] Input: Request information from the smartphone app.

[2261] Output: The recipe request sent to the server.

[2262] Specific operation: The device sends the request information to the server via the Internet. It notifies the user that the transmission was successful.

[2263] Step 3:

[2264] The server retrieves the latest food information from the database.

[2265] Input: The recipe request sent to the server.

[2266] Output: Food information retrieved from the database.

[2267] What happens: The server runs an SQL query to pull food information from the database.

[2268] Step 4:

[2269] The server uses emotion analysis means to collect emotion data of the user.

[2270] Input: User emotion data when requesting a recipe.

[2271] Output: Analyzed user emotion data (e.g. stress level).

[2272] Specific operation: The emotion analysis means analyzes facial expressions and tone of voice through the smartphone's camera and microphone, and determines emotions using a TensorFlow model.

[2273] Step 5:

[2274] The server selects the optimal recipe based on food information and emotional data.

[2275] Input: Retrieved food information and parsed emotion data.

[2276] Output: The best recipe information to be suggested to the user.

[2277] Specific operation: The server uses an algorithm to select the recipe that best suits the food inventory and the user's emotions, and assembles the recipe information.

[2278] Step 6:

[2279] The server notifies the user of the selected recipe on their smartphone.

[2280] Input: Selected recipe information.

[2281] Output: Notification to the user's smartphone.

[2282] Specific operation: The server uses push notifications to send recipe information to the user's smartphone app.

[2283] Nutritional balance management

[2284] Step 1:

[2285] The user enters the recipe they selected and information about the meal they actually made into the app.

[2286] Input: Meal information entered by the user.

[2287] Output: Meal information sent to the server.

[2288] What it does: The app is designed to make it easy to record your meal information by scanning barcodes or entering your voice.

[2289] Step 2:

[2290] The terminal transmits the information to the server.

[2291] Input: Meal information entered by the user into the app.

[2292] Output: Meal information sent to the server.

[2293] Specific operation: The device sends the input information to the server via the Internet and notifies the user when the transmission is complete.

[2294] Step 3:

[2295] The server calculates the nutrients contained in each meal based on the input dietary information.

[2296] Input: Meal information sent to the server.

[2297] Output: Calculated nutrient information for each meal.

[2298] Specific operation: The server uses Python's pandas library to calculate the nutrients for each food.

[2299] Step 4:

[2300] The server stores the calculated nutrient information in a database.

[2301] Input: Calculated nutrient information.

[2302] Output: Nutrient information stored in a database.

[2303] Specific operation: The server records the nutrient information in a database and manages it for each user.

[2304] Step 5:

[2305] The server analyzes nutritional balance based on a week's dietary history and identifies any nutrient deficiencies.

[2306] Input: A week's worth of dietary history recorded in a database.

[2307] Output: Nutritional balance analysis results.

[2308] Specific operation: Analyzes dietary history using Python's scikit-learn library and evaluates nutritional balance.

[2309] Step 6:

[2310] The server creates a nutritional balance report and notifies the user through the app.

[2311] Input: Nutritional balance analysis results.

[2312] Output: Report notification to user.

[2313] Specific operation: The server creates a report based on the analysis results and sends a push notification to the app.

[2314] Preventing food waste

[2315] Step 1:

[2316] The server estimates the expiration date based on the purchase date information of the food product.

[2317] Input: Food purchase date information recorded in the database.

[2318] Output: Estimated expiration date.

[2319] What it does: Calculates expiration dates using Python's datetime library.

[2320] Step 2:

[2321] The server generates alerts for food items approaching their expiration date.

[2322] Input: Estimated expiration date information.

[2323] Output: Alert information.

[2324] Specific behavior: The server generates an alert message based on the expiration date and adds it to the alert list.

[2325] Step 3:

[2326] The server notifies the user of the generated alert on their smartphone.

[2327] Input: Alert information.

[2328] Output: Notification to the user's smartphone.

[2329] What it does: Sends an alert message to the user's smartphone app using push notification.

[2330] Examples of application of emotion analysis methods

[2331] Step 1:

[2332] While the user prepares dinner, the emotion analysis means analyzes facial expressions and tone of voice through the smartphone's camera and microphone.

[2333] Input: User's facial expression and voice data.

[2334] Output: Parsed emotion data.

[2335] How it works: The emotion analyzer analyzes the user's facial expressions and tone of voice in real time to assess their stress and happiness levels.

[2336] Step 2:

[2337] If the emotion analysis means determines that the user is feeling stressed, the server preferentially suggests recipes that use foods that have a relaxing effect.

[2338] Input: Parsed emotion data.

[2339] Output: Relaxing recipe suggestions.

[2340] What it does: The server searches the database for relaxing recipes and updates the list of suggestions.

[2341] Step 3:

[2342] When your diet is lacking in balance, the server will suggest foods that contain nutrients that help relieve stress.

[2343] Input: Nutritional balance analysis results and emotional data.

[2344] Output: Nutrition-conscious recipe suggestions.

[2345] What it does: Based on the missing nutritional information, the server recommends recipes that include foods that help relieve stress.

[2346] Step 4:

[2347] When food in the refrigerator is nearing its expiration date, the server will quickly alert the user if they are feeling down, along with an encouraging message.

[2348] Input: Emotion data and expiration date information.

[2349] Output: Emotion-sensitive alert notification.

[2350] What it does: The server adjusts the content and timing of alert notifications based on emotion data and adds encouraging messages.

[2351] (Application example 2)

[2352] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2353] Conventional refrigerator systems require complicated management of food items in the refrigerator and expiration dates, and do not adequately achieve health management for users, reduce the burden of housework, or reduce food waste. Furthermore, they do not suggest or manage food based on the user's emotional state, and do not provide detailed support to users. This results in insufficient management of users' health and efficiency in housework, and often leads to food waste.

[2354] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[2355] In this invention, the server includes a camera for taking pictures of food in the refrigerator, an image analysis unit for analyzing the images taken by the camera and recognizing the food, a suggestion unit for suggesting currently available recipes based on the recognized food information and the user's emotional information, a nutrition calculation unit for calculating the nutrients of each food and managing a nutritionally balanced diet, a food consumption management unit for estimating expiration dates based on food purchase dates and generating alerts for food items approaching their expiration date, and an emotional response unit for providing messages and suggestions based on the user's emotional state. This improves the efficiency of refrigerator food management, reduces the user's health management, eases housework burden, and reduces food waste. Furthermore, by suggesting recipes and generating alerts based on the user's emotional state, the server provides detailed support and improves user satisfaction.

[2356] 1. "Photographing means" refers to a photographing device placed to photograph food items in a refrigerator.

[2357] 2. "Image analysis means" refers to a means that has the function of analyzing the image acquired by the photographing means and recognizing food.

[2358] 3. "Suggestion means" refers to a means that has the function of suggesting currently available recipes based on recognized food information and user emotion information.

[2359] 4. "Nutrition calculation tool" refers to a tool that has the function of calculating the nutrients of each food and managing a nutritionally balanced diet.

[2360] 5. "Food expiration date management means" refers to a means that has the function of estimating the expiration date based on the purchase date information of food and generating an alert for food that is approaching its expiration date.

[2361] 6. "Emotion response means" refers to means that have the function of sending messages or suggestions according to the user's emotional state.

[2362] DETAILED DESCRIPTION OF THE INVENTION

[2363] The following describes an embodiment of the present invention: The system mainly operates using a camera means built into the refrigerator, a server, and a user's smartphone or other user terminal.

[2364] First, a camera installed inside the refrigerator periodically takes images of the inside of the refrigerator. This image data is sent to a server via the Internet. The server then uses image analysis means (such as image analysis software) to recognize food from the images and store the food information in a database. Specific image analysis uses the widely used OpenCV library, etc.

[2365] Next, when the user requests a dinner recipe, the server retrieves the latest food information and generates the optimal recipe using a suggestion method (recipe suggestion algorithm). This also takes the user's emotional information into consideration. Emotion recognition software such as EmotionEngine is used as an emotional response method to analyze the user's emotions. Based on the ingredient data and the user's emotional state, a recipe such as "tomato cream pasta" is sent to the user's smartphone.

[2366] The nutritional calculation tool calculates the nutritional value of each recipe based on the recipe selected by the user and the information on the meal they have actually made. This information is recorded on a server, and a report is created that analyzes the nutritional balance based on a week's worth of meal history and identifies any nutrient deficiencies. The report is sent to the user via the app.

[2367] Furthermore, the food consumption period management means estimates the expiration date based on the purchase date information of the food, and generates an alert for food that is approaching its expiration date. This alert is also notified to the user at an appropriate time using the emotion response means.

[2368] For example, for milk that is nearing its expiration date, a specific alert could be generated such as "Your milk is nearing its expiration date. Please consume it soon," accompanied by an uplifting message that reflects the user's emotional state.

[2369] Examples:

[2370] 1. Images taken by the refrigerator camera are periodically sent to the server.

[2371] 2. Using image analysis software (e.g., OpenCV), the food items in the refrigerator are recognized. For example, data such as "3 tomatoes, 1 carton of milk, and 6 eggs" is obtained.

[2372] 3. When a user requests a dinner recipe on their smartphone, the server uses the Emotion Engine to analyze the user's emotional information (e.g., stress level).

[2373] 4. Based on the user’s emotions and the food information in the refrigerator, the suggestion tool will suggest a recipe for “Tomato Cream Pasta” and notify the smartphone.

[2374] 5. The nutrition calculation means calculates the nutrients in the tomato cream pasta and creates a weekly nutritional balance report based on the user's dietary history.

[2375] 6. An alert will be generated for food products (such as milk) that are nearing their expiration date, and a notification will be sent to your smartphone saying, "Your milk is nearing its expiration date. Please consume it as soon as possible."

[2376] Example prompt sentence:

[2377] "Analyze images of the inside of the refrigerator and tell me the current food inventory."

[2378] "Please suggest a relaxing recipe based on my mood."

[2379] "Get alerts for foods that are nearing their expiration date."

[2380] This system will improve the efficiency of food management in the refrigerator, enabling users to manage their health, reduce household chores, and reduce food waste. It will also provide detailed support by suggesting recipes and generating alerts based on the user's emotional state, improving user satisfaction.

[2381] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2382] Specific operation steps:

[2383] Step 1:

[2384] The refrigerator's camera periodically captures images of the inside of the refrigerator, and the server receives this image data via the Internet.

[2385] Input: Image data of the inside of the refrigerator

[2386] Data processing: Image data is sent to a server via the Internet

[2387] Output: Image data received by the server

[2388] Step 2:

[2389] The server uses image analysis means (image analysis software) to analyze the received image data and identify the food items in the refrigerator. The information about the identified food items is stored in a database.

[2390] Input: Image data received by the server

[2391] Data analysis: Recognize food using image analysis software (e.g., OpenCV)

[2392] Output: Recognized food information (e.g., 3 tomatoe...

Claims

1. a camera means for photographing food in the refrigerator; image analysis means for analyzing an image captured by the camera means and recognizing food; a recipe suggestion means for suggesting currently available recipes based on the recognized food information; A nutritional management tool that calculates the nutrients of each food and manages a nutritionally balanced diet; A food waste prevention measure that estimates the expiration date based on the purchase date of food and generates an alert for food that is approaching its expiration date. A system including:

2. The system according to claim 1 , wherein the recipe suggesting means includes a notifying means for notifying a user terminal of a recipe suggestion based on the food information.

3. 2. The system according to claim 1, wherein the nutritional management means includes a report generation means for recording the user's dietary history, analyzing the nutritional balance, and generating a report.

Images