System

Abstract

A system is provided.SOLUTION: A system including a camera means for monitoring a state in a refrigerator, a means for processing image data captured using the camera means and recognizing a type of food ingredient from the image data, a means for managing an expiration date of the recognized food ingredient, a means for notifying a user of a food ingredient whose expiration date is approaching, a means for generating a recipe corresponding to the food ingredient and proposing the recipe to the user, and a means for making a personalized proposal based on input information of the user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In daily diet management, it is difficult to know the expiration dates of ingredients in the refrigerator, which often results in wasteful disposal or deterioration of ingredients. Another issue is that not knowing which recipes are suitable for each ingredient leads to a lack of variety in meals. These problems not only lower the quality of users' diets, but also cause food waste and economic losses. Furthermore, there is a lack of appropriate information to maintain healthy and balanced eating habits. To solve these issues, there is a need for a system that can efficiently manage ingredients in the refrigerator and suggest appropriate recipes. [Means for solving the problem]

[0005] The present invention solves this problem with a system including a camera for monitoring the status inside a refrigerator, a means for processing image data captured by the camera and recognizing the types of ingredients from the image data, a means for managing the expiration dates of the recognized ingredients, a means for notifying the user of ingredients approaching their expiration dates, a means for generating recipes corresponding to the ingredients and suggesting them to the user, and a means for making personalized suggestions based on information input by the user. Specifically, the system includes a means for periodically acquiring image data of ingredients inside the refrigerator, a means for transmitting the acquired image data to a server, a means for the server to recognize ingredients from the image data and store them in a database, a means for notifying the user when the expiration date is approaching, a means for suggesting recipes based on the ingredients, and a means for suggesting recipes based on the user's eating habits and preferences. This allows for efficient food management and supports the user's eating habits.

[0006] "Conditions inside the refrigerator" refers to physical and environmental factors such as temperature, humidity, odor, and the type and location of ingredients inside the refrigerator.

[0007] "Camera means" refers to a camera for capturing images of the interior of the refrigerator and the hardware and software associated with its operation and control.

[0008] "Image data" refers to data of still images or videos taken by a camera means.

[0009] "Type of food" refers to the specific types of food stored in the refrigerator, such as meat, vegetables, dairy products, etc.

[0010] "Means of recognition" refers to algorithms or AI models that identify specific ingredients from image data.

[0011] "Best before" refers to the date by which food can be eaten tastily, and is usually the date and time printed on the package at the time of purchase.

[0012] "Means of management" refers to software functions and databases that track the expiration dates of recognized ingredients and take action at the appropriate time.

[0013] "Notification means" refers to hardware and software used to notify users of the status of ingredients and expiration dates. Examples include push notifications to smartphones and email notifications.

[0014] A "recipe" refers to information about the steps and ingredients needed to cook a dish using specific ingredients.

[0015] "Means for generating" refers to a software algorithm for automatically generating a recipe based on ingredient information.

[0016] "User input information" refers to information that a user inputs through an application or device, including personal data such as eating habits, preferences, and allergy information.

[0017] "Personalized suggestions" refers to suggesting individually optimized recipes and management information based on the user's input information and past behavioral history.

[0018] The term "system" refers to a comprehensive combination of devices and software in which a camera means, image processing means, database, notification means, recipe generation means, user information management means, etc. work in cooperation with one another. [Brief explanation of the drawings]

[0019] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4]FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0020] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0021] First, the terms used in the following description will be explained.

[0022] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0023] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0024] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0025] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0026] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0027] [First embodiment]

[0028] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0029] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0030] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0031] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0032] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0033] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0034] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0035] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0036] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0037] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0038] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0039] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0040] This invention is a system that efficiently manages ingredients in a refrigerator and suggests suitable recipes. This system consists of a camera for monitoring the status of the refrigerator, a server for processing image data, and a terminal that notifies the user and suggests recipes. Each component and its operation are described in detail below.

[0041] System configuration

[0042] 1. Camera Means

[0043] A camera installed inside the refrigerator periodically takes pictures of the inside of the refrigerator. The camera has high resolution and can clearly capture each food item inside the refrigerator. The captured image data is sent to a server.

[0044] 2. Image data processing by the server

[0045] The server receives the image data sent from the refrigerator and uses an AI model to recognize the type of food. The recognized food is stored in a database. The server also estimates the expiration date for each food item and records this information in the database.

[0046] 3. Best before date management

[0047] The server periodically checks the expiration dates of ingredients stored in the database, lists ingredients that are approaching their expiration date, and prepares to notify the user of the listed ingredients.

[0048] 4. Means of notification

[0049] The server sends information about food items approaching their expiration date to the user's device, which can be a smartphone or tablet, and can send push notifications or email notifications to the user.

[0050] 5. Recipe generation and suggestions

[0051] The server generates recipes based on ingredients that are approaching their expiration date. The recipe generation uses a database built in the past and information entered by the user (preferences, allergies, etc.). The generated recipes are presented to the user as multiple options.

[0052] 6. Personalization with user input

[0053] Users input their eating habits, preferences, allergy information, etc. through an application or web interface, and the server uses this information to suggest individually optimized recipes for the user.

[0054] Example of program processing

[0055] 1. Recognizing ingredients

[0056] The user places a new ingredient (e.g., cabbage) in the refrigerator.

[0057] The server analyzes the image sent from the camera inside the refrigerator and recognizes the cabbage. The recognition result and the expiration date of the cabbage are recorded in a database.

[0058] 2. Best before date management

[0059] The server periodically checks the database and finds that the cabbage's expiration date is two days away.

[0060] The server sends a notification to the terminal such as "The cabbage's expiration date is in two days."

[0061] 3. Recipe suggestions

[0062] The server generates recipes that use cabbage (e.g., cabbage rolls, cabbage salad).

[0063] The device suggests these recipes to the user.

[0064] When a user selects a recipe for cabbage rolls, detailed instructions and ingredients are displayed on the device.

[0065] In this way, the system of the present invention efficiently manages ingredients and supports the user's eating habits through ingredient recognition, expiration date management, notifications, recipe suggestions, and personalization based on user-entered information.

[0066] The processing flow will be explained below.

[0067] Step 1:

[0068] The user places a new ingredient (e.g., cabbage) in the refrigerator.

[0069] The refrigerator periodically takes pictures of the interior using a built-in camera.

[0070] Step 2:

[0071] Image data captured by the camera means is transmitted to the server.

[0072] The server receives the image data.

[0073] Step 3:

[0074] The server processes the image data and uses an AI model to recognize the type of food ingredient (e.g., cabbage).

[0075] The recognition results are recorded in a database.

[0076] Step 4:

[0077] The server automatically estimates the expiration date of the recognized ingredients and stores it in a database.

[0078] The server periodically checks the database to identify ingredients that are approaching their expiration date.

[0079] Step 5:

[0080] The server prepares a notification to the user for ingredients that are nearing their expiration date.

[0081] A notification such as "The cabbage's expiration date is in two days" is sent to the device.

[0082] Step 6:

[0083] The device displays a push or email notification to the user.

[0084] The user checks the notification and becomes aware of the presence of food items (e.g., cabbage) that are nearing their expiration date.

[0085] Step 7:

[0086] The server runs an algorithm to suggest recipes using cabbage (e.g., cabbage rolls, cabbage salad).

[0087] A proposal list is created and sent to the terminal.

[0088] Step 8:

[0089] The terminal displays a list of suggestions to the user.

[0090] The user selects the cabbage rolls recipe from the list of suggestions.

[0091] Step 9:

[0092] The server sends detailed recipe information (steps and ingredients) for the cabbage rolls to the terminal.

[0093] The terminal displays the detailed information to the user.

[0094] Step 10:

[0095] Users enter their eating habits, preferences, allergy information, etc. through the application.

[0096] The server stores this input information in a database and reflects it in future recipe suggestions.

[0097] Through this series of processing steps, users can efficiently manage and use ingredients, and improve the quality of their diet by utilizing optimal recipes.

[0098] Example 1

[0099] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0100] There is a need for efficient management of food in refrigerators, reducing food waste, and providing users with appropriate and timely recipe suggestions. However, existing refrigerator management systems have issues with food recognition accuracy, expiration date management, user notifications, and recipe suggestions, making it difficult to perform multiple different functions in an integrated manner.

[0101] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0102] In this invention, the server includes an image acquisition means, a means for processing image data captured using the image acquisition means and recognizing the type of food from the image data, a means for managing the expiration dates of the recognized foods, a means for periodically checking the database to confirm the expiration dates of the foods and notifying the user when the expiration date is approaching, a means for generating cooking methods based on the notified foods with an approaching expiration date and providing them to the user, and a means for making personalized suggestions based on information input by the user. This allows multiple different functions to be executed in an integrated manner as a series of processes, making it possible to improve the efficiency of food management, reduce waste, and even suggest recipes optimized for the user.

[0103] The "image acquisition means" is a device such as a camera or sensor that takes a picture of the food in the refrigerator and acquires the image data.

[0104] "Means for recognizing food types" refers to algorithms or AI models that analyze acquired image data and identify the food in the image.

[0105] A "best-before date management tool" is software or algorithms that record the best-before dates of recognized food products in a database and periodically check those dates.

[0106] "Means of checking the database to verify expiration dates" refers to the process or method of periodically checking food information in the database to identify foods that are approaching their expiration date.

[0107] The "notification means" refers to a push notification, email notification, or other notification method for informing the user about food products that are approaching their expiration date.

[0108] "Method for generating cooking instructions" refers to an algorithm or AI model that automatically generates recipes and cooking instructions based on specific foods.

[0109] The "means for providing to the user" refers to a terminal or application interface for displaying the generated recipes and cooking methods to the user.

[0110] "Means for personalized suggestions" refers to customization features and algorithms that provide optimized recipes and cooking methods based on the user's eating habits and preferences.

[0111] This invention is a system that efficiently manages ingredients in a refrigerator and suggests suitable recipes. The system consists of an image acquisition unit for monitoring the status of the refrigerator, a server for processing image data, and a terminal that notifies the user and suggests recipes.

[0112] Image Acquisition Method

[0113] A camera is installed inside the refrigerator and periodically takes pictures of the inside of the refrigerator. The camera has high resolution and can clearly capture each ingredient. The captured image data is sent to a server via wireless or wired communication.

[0114] Image data processing by the server

[0115] The server receives the image data sent from the refrigerator and uses an AI model (e.g., a model trained with TensorFlow or PyTorch) to recognize the type of food. The recognized food information is stored in a database. The server also estimates the expiration date of each food item and records this information in the database.

[0116] Best before date management

[0117] The server periodically checks the expiration dates of ingredients stored in the database. It lists ingredients whose expiration dates are approaching and prepares to notify users based on that information. The expiration date management algorithm estimates expiration dates based on, for example, the type of ingredient and the date of purchase.

[0118] Notification means

[0119] The server sends information about food items approaching their expiration date to a device (e.g., a smartphone or tablet). The device then sends a push notification or email notification to the user based on the received information.

[0120] Recipe generation and suggestions

[0121] The server generates recipes based on ingredients that are approaching their expiration date. The recipe generation uses a database built in the past and information entered by the user (e.g., preferences and allergies). The generated recipes are sent to the terminal and provided to the user as multiple options.

[0122] Personalization based on user input

[0123] Users input their eating habits, preferences, and allergies through an application or web interface, and the server uses this information to generate and suggest individually optimized recipes to the user.

[0124] Specific examples of operation

[0125] When a user places a new ingredient (e.g., cabbage) in the refrigerator, the camera takes a picture of it and sends it to the server. The server analyzes the image to recognize the cabbage and records its information and expiration date in a database. When the expiration date approaches, the server sends a notification to the device stating, "The cabbage will expire in two days." The device receives this notification and notifies the user as a push notification. The server then generates recipes using cabbage (e.g., cabbage rolls, cabbage salad) and sends them to the device. The device presents these recipes to the user, and if the user selects a cabbage roll recipe, it displays detailed instructions and the necessary ingredients.

[0126] Example prompts for generative AI models

[0127] "I'm storing some cabbage and it will expire in two days. Can you suggest some recipes using cabbage? Please take into consideration recipes that users have liked in the past."

[0128] In this way, the system of the present invention can effectively manage ingredients and support the user's eating habits through ingredient recognition, expiration date management, user notifications, recipe suggestions, and personalization based on user-entered information.

[0129] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0130] Step 1:

[0131] The camera periodically takes high-resolution images of the inside of the refrigerator. The input is an image of the food inside the refrigerator, and the output is the captured image data. The captured image data is sent to a server via a wireless or wired network. Specifically, the camera automatically releases the shutter and transmits the resulting image as digital data.

[0132] Step 2:

[0133] The server processes the image data received from the camera. The input is the image data of the food sent from the camera, and the output is the recognized food information (e.g., food name, quantity, etc.). Specifically, the server analyzes the image using an AI model (e.g., TensorFlow or PyTorch) and extracts the identified ingredient information. This analysis uses image processing algorithms and machine learning models.

[0134] Step 3:

[0135] The server estimates the expiration date based on the recognized food ingredient information. The input is the recognized food information, and the output is the estimated expiration date for each food item. Specifically, the algorithm calculates the expiration date based on the type of food ingredient, storage conditions, and past data. As a result, the expiration date information is recorded in a database.

[0136] Step 4:

[0137] The server periodically checks the database to identify ingredients that are approaching their expiration date. The input is all food information in the database, and the output is a list of ingredients that are approaching their expiration date. Specifically, a scheduling program traverses the database and lists ingredients that are within three days of their expiration date.

[0138] Step 5:

[0139] The server sends information about ingredients that are approaching their expiration date to the device. The input is a list of ingredients that are approaching their expiration date, and the output is a notification message to be sent to the user. Specifically, the server creates a push notification or email notification and sends it to the device. This notification contains a specific message, such as "The cabbage will expire in two days."

[0140] Step 6:

[0141] The device displays the notification message received from the server to the user. The input is the notification message sent from the server, and the output is the notification information displayed to the user. Specifically, the message is displayed on the device screen as a push notification or email notification.

[0142] Step 7:

[0143] The server generates recipes based on ingredients that are approaching their expiration date. The input is information about ingredients that are approaching their expiration date, and the output is a list of generated recipes. Specifically, the server generates multiple recipes based on a database of past recipes and user preferences. This process uses AI models and algorithms.

[0144] Step 8:

[0145] The server sends the generated recipes to the terminal. The input is a list of generated recipes, and the output is recipe information provided to the user. In concrete terms, the server sends recipe information to the terminal, and the terminal receives it.

[0146] Step 9:

[0147] The device presents the received recipe information to the user. The input is the recipe information sent from the server, and the output is a list of recipes displayed to the user. Specifically, the device displays multiple recipe options on the screen and displays detailed instructions and required ingredients for the recipe selected by the user.

[0148] Step 10:

[0149] A user provides input information, such as preferences and allergies, through an application or web interface. The input is the information entered by the user, and the output is the personalized information sent to the server. Specifically, the user enters information through a digital interface, and the information is sent to the server.

[0150] This series of processes realizes an overall flow from food recognition to expiration date management and recipe suggestions, effectively supporting the user's food management and eating habits.

[0151] (Application example 1)

[0152] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0153] Conventional refrigerator food management systems were limited to managing food ingredients within the home, and were unable to adequately manage ingredients or suggest recipes based on ingredients approaching their expiration date when purchasing at a physical store (such as a supermarket). Furthermore, it was not possible to track the history of ingredients purchased by the user and manage expiration dates or suggest recipes based on that information. This made food management cumbersome and led to the risk of food waste.

[0154] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0155] In this invention, the server includes a camera means for monitoring the status inside the refrigerator, a means for processing image data captured by the camera means and recognizing the type of ingredients from the image data, a means for managing the expiration dates of the recognized ingredients, a means for notifying the user of ingredients approaching their expiration dates, a means for generating recipes corresponding to the ingredients and suggesting them to the user, a means for making personalized suggestions based on information input by the user, and a means for managing ingredient purchase histories and presenting ingredients approaching their expiration dates in purchasing activities at physical stores. This enables ingredient expiration date management and recipe suggestions not only at home but also in purchasing activities at physical stores, enabling efficient ingredient management and reduction of food waste.

[0156] The "camera means" is a device for taking pictures of the inside of the refrigerator and acquiring the image data.

[0157] "Means for processing image data and recognizing the type of food ingredient from the image data" refers to technology for analyzing image data obtained from a camera means and identifying the type of food ingredient using an AI model, etc.

[0158] The "means for managing the expiration dates of recognized ingredients" is a system for storing the expiration dates of recognized ingredients in a database and periodically checking the expiration dates.

[0159] The "means for notifying users about food ingredients that are approaching their expiration date" is a system for notifying users about food ingredients that are approaching their expiration date, and is a mechanism for sending notifications to smartphones or tablets.

[0160] "Means for generating recipes corresponding to ingredients and suggesting them to users" refers to technology for generating appropriate recipes based on the types of ingredients and expiration dates and providing them to users.

[0161] "Means for making personalized suggestions based on user input" refers to a system that takes into account the user's preferences, eating habits, allergy information, etc., and suggests individually optimized recipes based on that information.

[0162] "Means for managing food purchase history and presenting food items approaching their expiration date during purchasing activities at physical stores" is a technology that stores the history of food items purchased at physical stores in a database and presents food items approaching their expiration date to the user.

[0163] The "means for periodically checking expiration dates" is a system that periodically checks the expiration dates of ingredients stored in a database.

[0164] The "means for generating a recipe including ingredients to be used based on a notification" is a technology that automatically generates a recipe including ingredients to be used based on ingredients whose expiration date is approaching.

[0165] The "terminal means for providing the generated recipe to the user" is a device for displaying the generated recipe to the user, such as a smartphone or tablet.

[0166] This system efficiently manages ingredients in a refrigerator and suggests suitable recipes, and also enables ingredient management and recipe suggestions for purchases in physical stores. This system consists of three main components: a camera device in the refrigerator, a server, and a user terminal.

[0167] System Components

[0168] 1. Camera Means

[0169] A camera installed inside the refrigerator periodically takes pictures of the interior. The high-resolution camera captures clear images of each ingredient. The captured image data is sent to a server.

[0170] 2. Image data processing by the server

[0171] The server receives the image data sent from the refrigerator and uses the generative AI model to recognize the type of food. The recognition results are recorded in a database. The server also estimates the expiration date, and this information is also stored in the database.

[0172] 3. Best before date management

[0173] The server periodically checks the expiration dates of ingredients stored in the database and lists ingredients that are approaching their expiration date, and prepares to notify the user of this information.

[0174] 4. Means of notification

[0175] The server sends notifications about food items approaching their expiration date to the user's device, such as a smartphone or tablet, via push notifications or email.

[0176] 5. Recipe generation and suggestions

[0177] The server generates recipes based on ingredients that are approaching their expiration date. The recipes are generated using a database built in the past and user input information (preferences, allergies, etc.). The generated recipes are provided to the user.

[0178] 6. Personalization with user input

[0179] Users input their eating habits, preferences, allergy information, etc. through the application, and the server uses this information to suggest individually optimized recipes.

[0180] 7. Managing food purchase history

[0181] The system also manages the history of food purchases at physical stores, and uses the purchase history database to suggest recipes based on ingredients that are close to their expiration date.

[0182] Hardware and software used

[0183] Hardware

[0184] Camera Method: High resolution camera installed inside the refrigerator.

[0185] User devices: smartphones, tablets.

[0186] Server: A cloud server such as AWS or GCP.

[0187] software

[0188] Image processing: Python, OpenCV, TensorFlow.

[0189] Server side: Django framework.

[0190] Database Management: Django ORM.

[0191] Communication method: Communication using REST API.

[0192] Specific examples of use

[0193] For example, if a user places "tomatoes" and "lettuce" in the refrigerator, the camera takes a picture of them and the server recognizes the type of food. The recognized food is stored in a database, and the user is notified when the expiration date approaches. At the same time, a recipe for "tomato and lettuce salad" is suggested. If the user purchases "chicken" at a physical store, the purchase history is stored in the database and used to suggest recipes for the next time.

[0194] Prompt Sentence Examples

[0195] "Please tell me some recipes that use the tomatoes, lettuce, and chicken I have in my fridge. The user prefers low-calorie recipes, and is allergic to nuts."

[0196] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0197] Step 1:

[0198] A camera means in the refrigerator periodically takes pictures of ingredients. The captured image data is sent to a server. The input is the image of the ingredients in the refrigerator, and the output is the image data sent to the server. The camera takes pictures when the camera button is pressed, or automatically at set intervals.

[0199] Step 2:

[0200] The server analyzes the received image data using a generative AI model to recognize the type of ingredient. The input is the image data sent in step 1, and the output is the recognized ingredient information. The analysis process is performed using Python and TensorFlow.

[0201] Step 3:

[0202] The server stores the information of the recognized ingredients in a database. The database records the type of ingredient, quantity, purchase date, and expiration date. The input is the ingredient information recognized in step 2, and the output is an organized ingredient database. It is saved to the database using Django ORM.

[0203] Step 4:

[0204] The server periodically checks the database and lists ingredients that are approaching their expiration date. The input is the ingredient information in the database, and the output is a list of ingredients that are approaching their expiration date. The checking process is performed by a Python script.

[0205] Step 5:

[0206] The server sends notifications to the user's device about ingredients that are approaching their expiration date. The input is the ingredient information listed in step 4, and the output is a notification sent to the user's smartphone or tablet. Notifications are sent via push notifications or email.

[0207] Step 6:

[0208] The server generates recipes based on ingredients that are approaching their expiration date. Multiple recipes are created using the generative AI model and the user's eating habits, preferences, and allergy information. The input is the ingredient information from step 4 and the user's personal information, and the output is the generated recipe. Prompt statements can be used to generate recipes.

[0209] Step 7:

[0210] The server provides the generated recipe to the user's device. The input is the recipe generated in step 6, and the output is the recipe displayed on the user's smartphone or tablet through the application interface.

[0211] Step 8:

[0212] When a user buys ingredients in a physical store, the purchase history is stored in a database. The input is the user's purchase information, and the output is an updated ingredient purchase history database. The database is managed using Django ORM.

[0213] Step 9:

[0214] The server uses the purchase history database to create a list of food items purchased in physical stores that are nearing their expiration date, and presents this to the user. The input is the updated purchase history database, and the output is a new notification sent to the user's device. The notification is displayed on a smartphone or tablet.

[0215] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0216] This invention is a system that efficiently manages ingredients in a refrigerator and suggests appropriate recipes, and by combining it with an emotion engine that recognizes the user's emotions, it provides more personalized dietary support. This system is composed of a camera for monitoring the status of the refrigerator, a server that processes image data, a terminal that notifies the user and suggests recipes, and an emotion engine that recognizes the user's emotions. Each component and its operation are described in detail below.

[0217] System configuration

[0218] 1. Camera Means

[0219] A camera installed inside the refrigerator periodically takes pictures of the inside of the refrigerator. The camera has high resolution and can clearly capture each food item inside the refrigerator. The captured image data is sent to a server.

[0220] 2. Image data processing by the server

[0221] The server receives the image data sent from the refrigerator and uses an AI model to identify the type of food. The identified food is then stored in a database. The server also estimates the expiration date for each food item and records this information in the database.

[0222] 3. Best before date management

[0223] The server periodically checks the expiration dates of ingredients stored in the database, lists ingredients that are approaching their expiration date, and prepares to notify the user of the listed ingredients.

[0224] 4. Means of notification

[0225] The server sends information about food items approaching their expiration date to the user's device, which can be a smartphone or tablet, and can send push notifications or email notifications to the user.

[0226] 5. Recipe generation and suggestions

[0227] The server generates recipes based on ingredients that are approaching their expiration date. The recipe generation uses a database built in the past and information entered by the user (preferences, allergies, etc.). The generated recipes are presented to the user as multiple options.

[0228] 6. Personalization with user input

[0229] Users input their eating habits, preferences, allergy information, etc. through an application or web interface, and the server uses this information to suggest individually optimized recipes for the user.

[0230] 7. Emotion Engine

[0231] The emotion engine recognizes emotions from the user's voice, facial expressions, text input, etc. The user's emotion information recognized by the emotion engine is also stored in the database.

[0232] Example of program processing

[0233] 1. Recognizing ingredients

[0234] The user places a new ingredient (e.g., cabbage) in the refrigerator.

[0235] The server analyzes the image sent from the camera inside the refrigerator and recognizes the cabbage. The recognition result and the expiration date of the cabbage are recorded in a database.

[0236] 2. Best before date management

[0237] The server periodically checks the database and finds that the cabbage's expiration date is two days away.

[0238] The server sends a notification to the terminal such as "The cabbage's expiration date is in two days."

[0239] 3. Recipe suggestions

[0240] The server generates recipes that use cabbage (e.g., cabbage rolls, cabbage salad).

[0241] The device suggests these recipes to the user.

[0242] When a user selects a recipe for cabbage rolls, detailed instructions and ingredients are displayed on the device.

[0243] 4. Personalization with user input

[0244] Users enter their eating habits, preferences, allergy information, etc. through the application.

[0245] The server stores this information in a database and reflects it in future recipe suggestions.

[0246] 5. Emotion recognition and suggestion optimization

[0247] The emotion engine recognizes emotions from the user's voice and facial expressions and sends this information to the server.

[0248] The server then tailors recipes and notifications to suit the user based on their emotional information, for example, suggesting easy-to-make recipes if the user is tired.

[0249] In this way, the system of the present invention provides optimal dietary support to users by combining ingredient recognition, expiration date management, notifications, recipe suggestions, personalization based on user-entered information, and emotion recognition using an emotion engine.

[0250] The processing flow will be explained below.

[0251] Step 1:

[0252] The user places a new ingredient (e.g., cabbage) in the refrigerator.

[0253] The refrigerator periodically takes pictures of the interior using a built-in camera.

[0254] Step 2:

[0255] Image data captured by the camera means is transmitted to the server.

[0256] The server receives the image data.

[0257] Step 3:

[0258] The server analyzes the image data and uses an AI model to recognize the type of food ingredient (e.g., cabbage).

[0259] The recognition results are recorded in a database.

[0260] Step 4:

[0261] The server automatically estimates the expiration date of the recognized ingredients and stores it in a database.

[0262] The server periodically checks the database to identify ingredients that are approaching their expiration date.

[0263] Step 5:

[0264] The server prepares a notification to the user for listed ingredients that are close to their expiration date.

[0265] A notification such as "The cabbage's expiration date is in two days" is sent to the device.

[0266] Step 6:

[0267] The device displays the push notification to the user.

[0268] The user checks the notification and recognizes ingredients (e.g., cabbage) that are nearing their expiration date.

[0269] Step 7:

[0270] The server requests data to check the user's emotions with an emotion engine.

[0271] The device collects the user's voice and facial expression data and sends it to the server.

[0272] Step 8:

[0273] The emotion engine analyzes the user's emotions (e.g., if the user is tired).

[0274] The server records the user's emotional information recognized by the emotion engine in a database and uses it to adjust the recipe.

[0275] Step 9:

[0276] The server executes a recipe algorithm based on the user's ingredient data and emotion data.

[0277] For example, if the cabbage is nearing its expiration date and the user is tired, the app will suggest an easy recipe (e.g., easy stir-fried cabbage).

[0278] Step 10:

[0279] The server generates recipe suggestions and sends them to the device.

[0280] The device displays recipe suggestions to the user.

[0281] When a user selects a recipe, detailed instructions and required ingredients are displayed.

[0282] Step 11:

[0283] Users enter their eating habits, preferences, allergy information, etc. through the application.

[0284] The server stores this information in a database and reflects it in future recipe suggestions.

[0285] In this way, the system provides optimal dietary support to users by recognizing ingredients, managing expiration dates, providing notifications, suggesting recipes, personalizing with user-entered information, and even recognizing emotions using an emotion engine.

[0286] Example 2

[0287] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0288] In modern households, managing food in the refrigerator is complicated, and food that has passed its expiration date is often wasted. Furthermore, when it comes to effectively utilizing ingredients and suggesting recipes, there is a lack of personalization that takes into account individual users' preferences and allergy information. Furthermore, there is no system that can make appropriate suggestions based on the user's emotional state.

[0289] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a camera means for monitoring the state inside the refrigerator, a means for processing image data captured by the camera means and recognizing the types of ingredients from the image data, a means for estimating and managing the expiration dates of the recognized ingredients, a means for notifying the user of ingredients whose expiration dates are approaching, a means for generating recipes based on the ingredients and suggesting them to the user, a means for making personalized suggestions based on information input by the user, and a means for recognizing the user's emotions from their voice, facial expressions, text input, etc., and adjusting the suggestions. This reduces ingredient waste, makes recipe suggestions that meet the user's individual preferences, and enables optimal suggestions based on the user's emotional state.

[0290] The "camera means" refers to a camera device installed to monitor the state inside the refrigerator, and has the function of taking pictures of ingredients.

[0291] "Image data" refers to visual information of the inside of a refrigerator photographed by a camera means, which is recorded and stored in digital format.

[0292] The "means for recognizing the type of food ingredient" is a function that analyzes the captured image data and identifies the name and attributes of the food ingredient using AI or machine learning models.

[0293] The "means for estimating and managing expiration dates" has the function of predicting and managing expiration dates for recognized food ingredients based on the purchase date and general storage period.

[0294] The "notification means" has a function for transmitting information about ingredients approaching their expiration date, recipe suggestions, etc. to the user's terminal.

[0295] The "means for generating recipes" has the function of automatically creating cooking instructions and steps based on the recognized ingredient information.

[0296] The "means for suggesting to the user" has a function for displaying the generated recipe and notification content to the user.

[0297] The "means for making personalized suggestions" has the function of generating recipes and notification content that take into account individual preferences and allergies based on the user's input information and past data.

[0298] The "means for recognizing emotions and adjusting suggested content" has the function of analyzing the user's current emotional state from their voice, facial expressions, text input, etc., and providing recipes and notification content accordingly.

[0299] This invention is a system that efficiently manages ingredients in the refrigerator and suggests suitable recipes, and by combining it with an emotion engine that recognizes the user's emotions, it provides more personalized dietary support. This system operates based on the following components.

[0300] 1. Camera Means

[0301] The camera installed inside the refrigerator takes high-resolution images of the interior of the refrigerator. The camera periodically (for example, every hour) takes a picture of the overall situation inside the refrigerator and sends this image data to a server. It is desirable to use a network camera with a resolution of 1080p or higher.

[0302] 2. Image data processing by the server

[0303] The server receives the image data sent from the camera and performs image analysis. This image analysis uses AI models such as TensorFlow and PyTorch. This automatically recognizes the types of ingredients in the refrigerator and stores the results in a database. Specifically, it uses the YOLOv3 model to perform high-speed, high-precision object recognition.

[0304] 3. Estimation and management of expiration dates

[0305] The server estimates the expiration date for the recognized ingredients. For example, if it recognizes cabbage, it calculates the expiration date based on the typical storage period for cabbage (about 5 days in the refrigerator) and records this information in the database. The database is a relational database such as MySQL or PostgreSQL.

[0306] 4. Sending notifications

[0307] The server periodically checks the information in the database and sends notifications to the user's device about ingredients that are approaching their expiration date. This notification is done via push notification or email. For example, if the expiration date of cabbage is approaching in two days, a message such as "The expiration date of the cabbage is in two days" is generated and sent to the user's smartphone.

[0308] 5. Recipe generation and suggestions

[0309] The server uses a database of past recipes and information entered by the user (such as preferences and allergies) to generate recipes based on ingredients approaching their expiration date. The generated recipes are presented to the user as multiple options. For example, recipes such as "stuffed cabbage" and "cabbage salad" using cabbage may be generated.

[0310] 6. Personalization with user input

[0311] Users input their eating habits, preferences, allergies, and other information through the application or web interface. The server stores this information in a database and uses it to suggest future recipes, allowing users to receive recipes tailored to their individual needs.

[0312] 7. Emotion Recognition with Emotion Engine

[0313] The emotion engine recognizes emotions from the user's voice, facial expressions, and text input. This emotion information is also sent to the server and stored in a database. The server uses this information to tailor recipes and notification content to suit the user. For example, if the emotion engine recognizes that the user is tired, it will prioritize suggesting easy recipes.

[0314] Examples and prompts

[0315] Example: When a user puts a new cabbage in the refrigerator, the camera recognizes this and sends data to the server to calculate the expiration date of the cabbage. When the expiration date approaches, the server sends a notification and generates several recipes using the cabbage and displays them on the device.

[0316] Example prompt: "Recognize the new ingredients in your refrigerator and suggest a recipe using them."

[0317] In this way, the system of the present invention comprehensively supports everything from managing ingredients in the refrigerator to suggesting recipes for each user, and even suggesting the best recipes based on the user's emotional state. Introducing this system will reduce food waste and lead to a richer and healthier diet for users.

[0318] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0319] Step 1:

[0320] Taking pictures with a camera

[0321] Input: Condition inside the refrigerator

[0322] Process: The camera takes high-resolution images of the inside of the refrigerator periodically (e.g., every hour). Use a network camera with a resolution of 1080p or higher.

[0323] Output: Image data of the inside of the refrigerator

[0324] Specific operation: The location and condition of all food items in the refrigerator are clearly photographed and data is generated.

[0325] Step 2:

[0326] Sending image data

[0327] Input: Image data of the inside of the refrigerator

[0328] Processing: Image data generated by the camera is sent to the server in real time via Wi-Fi.

[0329] Output: Image data sent to the server

[0330] Specific operation: The image file taken by the camera is compressed and a protocol (e.g. HTTP) is executed to send it to the server.

[0331] Step 3:

[0332] Receiving and analyzing image data

[0333] Input: Image data sent from the camera

[0334] Processing: The server receives the image data and uses an AI model (e.g., TensorFlow or PyTorch) to recognize the type of food. Specifically, the YOLOv3 model is used.

[0335] Output: Recognized food type and location information

[0336] Specific operation: The server analyzes the received image data, identifies the name and location of each ingredient, and records it in a database.

[0337] Step 4:

[0338] Estimating expiration dates and storing

[0339] Input: Recognized food type and location information

[0340] Processing: The server estimates the shelf life of each ingredient based on its type. For example, cabbage can be stored in the refrigerator for 5 days.

[0341] Output: Best before date data for each ingredient

[0342] What it does: Calculates the expiration date for each ingredient based on the purchase date and shelf life, and records that information in a database.

[0343] Step 5:

[0344] Regular check of expiration dates

[0345] Input: Best before date data stored in the database

[0346] Processing: The server periodically (e.g., every night at midnight) scans the database and lists ingredients that are nearing their expiration date.

[0347] Output: A list of ingredients that are nearing their expiration date

[0348] What it does: Query the database to find ingredients with a shelf life of less than 3 days.

[0349] Step 6:

[0350] Creating a notification message and preparing it for sending

[0351] Input: A list of ingredients that are nearing their expiration date

[0352] Processing: The server generates a notification message for the listed ingredients, for example, "The cabbage will expire in 2 days."

[0353] Output: The generated notification message

[0354] Specific operation: Automatically generate notification content using a message template and prepare to send it to the user's device.

[0355] Step 7:

[0356] Sending notifications

[0357] Input: The generated notification message

[0358] Process: The server sends a notification message to the user's device via push notification or email notification.

[0359] Output: Notification message received by the user

[0360] Specific operation: The server sends a message to the user's device via push notification or email system.

[0361] Step 8:

[0362] Receiving notifications

[0363] Input: Notification message sent by the server

[0364] Processing: The device receives the notification and displays it on the user's screen as a popup or email.

[0365] Output: The notification message displayed to the user

[0366] Specific behavior: The device's notification system receives the message from the server and immediately displays it to the user.

[0367] Step 9:

[0368] Recipe Generation

[0369] Input: List of ingredients approaching expiration date and user database

[0370] Processing: The server generates recipes based on ingredients that are nearing their expiration date, using a database of past data and user preferences and allergy information.

[0371] Output: A list of generated recipes

[0372] Specific operation: The system generates multiple appropriate recipes based on the ingredients data and the user's profile. For example, it suggests "cabbage rolls" and "cabbage salad" using cabbage.

[0373] Step 10:

[0374] Recipe Suggestions

[0375] Input: A list of generated recipes

[0376] What happens: The device presents the suggested recipes to the user. When the user selects a specific recipe, detailed instructions and required ingredients are displayed.

[0377] Output: Recipe details

[0378] Specific behavior: Displays a list of recipes on the device screen and provides detailed information about the recipe selected by the user.

[0379] Step 11:

[0380] Entering user information

[0381] Input: User-provided dietary habits, preferences, and allergy information

[0382] Processing: The user enters information through an application or web interface and sends it to the server.

[0383] Output: User information stored in the database

[0384] What happens: The user uses the interface to enter their information and presses the submit button. The server receives this and records it in the database.

[0385] Step 12:

[0386] Storage and Use of User Information

[0387] Input: User-submitted dietary habits, preferences, and allergy information

[0388] Processing: The server stores this information in a database and uses it to suggest recipes from the next time onwards.

[0389] Output: Updated user profile

[0390] Specific operation: The server uses the user's input information to personalize future recipe suggestions and notifications.

[0391] Step 13:

[0392] emotion recognition

[0393] Input: User voice, facial expressions, and text input

[0394] Processing: The emotion engine grasps the user's emotion and sends it to the server.

[0395] Output: Recognized user emotion information

[0396] Specific operation: The emotion engine analyzes the user's input data, grasps the user's emotional state, and sends that information to the server.

[0397] Step 14:

[0398] Use of emotional information

[0399] Input: Recognized user emotion information

[0400] Processing: The server records the emotion information in a database and adjusts the suggestions to the user based on this information.

[0401] Output: Adjusted proposal

[0402] Specific operation: The server refers to the emotional information and provides the user with the most appropriate content according to their emotions, such as suggesting simple recipes if the user is tired.

[0403] (Application example 2)

[0404] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0405] In recent years, refrigerator food management and food waste issues have been attracting attention, but traditional manual and simple digital management methods have their limitations. Furthermore, managing ingredients in brick-and-mortar stores and effectively suggesting recipes to customers requires a great deal of effort. Furthermore, while there is a demand for services that take customer emotions into account, no effective system exists to achieve this. To solve these issues, it is necessary to streamline refrigerator food management, provide appropriate information to store staff and customers, and provide personalized services that take customer emotions into account.

[0406] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0407] In this invention, the server includes means for processing image data captured by a camera means and recognizing the type of ingredient from the image data, means for managing the expiration dates of the recognized ingredients, means for notifying the user of ingredients approaching their expiration dates, means for generating recipes corresponding to the ingredients and suggesting them to the user, means for making personalized suggestions based on information input by the user, means for efficiently managing ingredients in a refrigerator and monitoring products stored in refrigerators in the store, means for notifying store staff of products approaching their expiration dates, and means for recognizing customer emotions and suggesting recipes or promoting purchases based on the emotions. This enables efficient management of ingredients in a refrigerator, appropriate management of ingredients in a physical store, personalized recipe suggestions to customers, and service provision based on customer emotions.

[0408] The "camera means" is a photographing device installed to monitor the state inside the refrigerator, and periodically takes pictures of the food ingredients.

[0409] "Image data" refers to video information of ingredients photographed using a camera, and is digital data that is analyzed by the server.

[0410] A "means for recognizing" is a method or device capable of processing image data and identifying and classifying objects in the image.

[0411] The "means for managing expiration dates" refers to a method or device for recording the expiration dates of recognized food ingredients and for notifying users when the expiration date is approaching.

[0412] "Notification means" refers to a method or device used to notify a user or store staff of specific information.

[0413] The "means for generating and suggesting recipes" refers to a method or device that creates a cooking method based on recognized ingredients and provides it to the user.

[0414] The "means for making personalized suggestions" refers to a method or device that makes individually optimized suggestions based on information input by the user, past data, and the like.

[0415] "Means for monitoring products" refers to a method or device for monitoring the condition of products stored in refrigerators in a store and understanding product trends and inventory status.

[0416] A "means for recognizing customer emotions" is a method or device that determines a customer's emotions from voice, facial expressions, text input, etc., and utilizes that information.

[0417] "Means for promoting purchases" are methods and devices for making individually optimized product suggestions based on customer emotions and behavioral data, thereby increasing purchasing motivation.

[0418] The present invention provides a system for improving the efficiency of food ingredient management in a refrigerator and product management in a physical store, and for suggesting personalized recipes to customers and promoting purchases. The following describes in detail an embodiment of the present invention.

[0419] First, regarding the hardware configuration of the entire system, a high-resolution camera is installed inside the refrigerator. This camera periodically takes pictures of the ingredients and products inside the store refrigerator and sends the image data to a server. A configuration using a virtual server on AWS EC2 is suitable for this server.

[0420] The server receives image data sent from the refrigerator and uses an image recognition AI model using TensorFlow to identify ingredients and product types. The identified ingredients are then stored in a database (AWS RDS).

[0421] The server then periodically checks the database and lists ingredients and products that are approaching their expiration date. Store staff are notified of these ingredients and products via push notifications sent via smartphones or tablets. The server also generates recipes based on the products that are approaching their expiration date.

[0422] Recipe generation utilizes previously constructed databases and information entered by users (such as preferences and allergies). The generated recipes are provided to customers via a sales promotion touch panel display and a customer application.

[0423] Furthermore, an emotion engine will be used to recognize emotions from the customer's voice and facial expressions. This emotion information will also be sent to the server, and personalized recipe suggestions and purchase promotions will be made based on the emotion. Microsoft Azure Cognitive Services could be used for the emotion engine.

[0424] The system includes the following specific processing steps:

[0425] 1. The camera means takes an image of the food in the refrigerator.

[0426] 2. The captured image data is sent to the server.

[0427] 3. The server analyzes the image using a TensorFlow model and recognizes the ingredients.

[0428] 4. The recognition results and expiration date information are stored in the database.

[0429] 5. The server periodically checks the database and notifies store staff of any food items that are nearing their expiration date.

[0430] 6. The server generates recipes for customers using ingredients that are close to their expiration date and provides them via a touch panel display or app.

[0431] 7. The emotion recognition engine recognizes the customer's emotions and sends the emotional information to the server.

[0432] 8. Based on the emotional information, the server will suggest recipes and promote purchases according to the emotions.

[0433] For example, consider the following prompt:

[0434] "A new cabbage is placed in the refrigerator. Use an image recognition AI model to recognize the cabbage and add it to the expiration date management system. When the cabbage's expiration date approaches, notify the customer via the touch panel display with a recommended recipe. Also, recognize the customer's emotions based on their facial expressions and suggest recipes that match those emotions."

[0435] This will enable efficient management of food in the refrigerator, proper notification to store staff, and the provision of services that respond to customer emotions.

[0436] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0437] Step 1:

[0438] A camera means within the refrigerator takes images of the food items.

[0439] Input: Refrigerator status

[0440] Data processing: image data capture

[0441] Output: High-resolution food image data

[0442] Step 2:

[0443] The captured image data is sent to a server.

[0444] Input: High-resolution food image data

[0445] Data processing: Sending image data

[0446] Output: Image data received by the server

[0447] Step 3:

[0448] The server uses a TensorFlow model to analyze the image data and recognize the type of ingredient.

[0449] Input: Received food image data

[0450] Data processing: image recognition, food classification

[0451] Output: Recognized ingredients (e.g., tomato, cabbage)

[0452] Step 4:

[0453] The recognized ingredient information and expiration date information are stored in a database on the server.

[0454] Input: Recognized ingredient information

[0455] Data processing: Recording in database, estimating expiration date

[0456] Output: Stored ingredients and expiration date data

[0457] Step 5:

[0458] The server periodically checks the database and notifies store staff of food items that are nearing their expiration date.

[0459] Input: Stored ingredients and expiration date data

[0460] Data processing: Expiration date check, notification information creation

[0461] Output: Notification information (e.g. "The expiration date for the tomatoes is in 2 days")

[0462] Step 6:

[0463] The server generates recipes using ingredients that are close to their expiration date and transmits them to a terminal for serving to the customer.

[0464] Input: Information about ingredients that are close to expiry date, as well as user preferences and allergy information

[0465] Data processing: Recipe generation, recipe data creation

[0466] Output: Recipe suggestions (e.g., tomato salad, tomato pasta)

[0467] Step 7:

[0468] The generated recipe information is provided to the customer via a touch panel display or a customer application.

[0469] Input: Recipe to suggest

[0470] Data processing: Preparing to display recipe information

[0471] Output: Display of recipe (e.g., recipe displayed on a touch panel or smartphone)

[0472] Step 8:

[0473] The emotion engine recognizes emotions from the customer's voice and facial expressions and sends that information to the server.

[0474] Input: Customer voice and facial expression data

[0475] Data processing: Emotion recognition, creating data of the recognition results

[0476] Output: Recognized emotion information (e.g., "The customer is tired")

[0477] Step 9:

[0478] Based on the emotional information, the server makes personalized recipe suggestions or purchase promotions according to the emotions.

[0479] Input: Recognized emotion information

[0480] Data processing: Recipe suggestions readjustment, purchase promotion information creation

[0481] Output: Personalized recipe suggestions and promotional notifications (e.g., "Easy recipes perfect for a tiring day")

[0482] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0483] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0484] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0485] [Second embodiment]

[0486] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0487] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0488] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0489] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0490] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0491] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0492] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0493] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0494] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0495] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0496] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0497] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0498] This invention is a system that efficiently manages ingredients in a refrigerator and suggests suitable recipes. This system consists of a camera for monitoring the status of the refrigerator, a server for processing image data, and a terminal that notifies the user and suggests recipes. Each component and its operation are described in detail below.

[0499] System configuration

[0500] 1. Camera Means

[0501] A camera installed inside the refrigerator periodically takes pictures of the inside of the refrigerator. The camera has high resolution and can clearly capture each food item inside the refrigerator. The captured image data is sent to a server.

[0502] 2. Image data processing by the server

[0503] The server receives the image data sent from the refrigerator and uses an AI model to identify the type of food. The identified food is then stored in a database. The server also estimates the expiration date for each food item and records this information in the database.

[0504] 3. Best before date management

[0505] The server periodically checks the expiration dates of ingredients stored in the database, lists ingredients that are approaching their expiration date, and prepares to notify the user of the listed ingredients.

[0506] 4. Means of notification

[0507] The server sends information about food items approaching their expiration date to the user's device, which can be a smartphone or tablet, and can send push notifications or email notifications to the user.

[0508] 5. Recipe generation and suggestions

[0509] The server generates recipes based on ingredients that are approaching their expiration date. The recipe generation uses a database built in the past and information entered by the user (preferences, allergies, etc.). The generated recipes are presented to the user as multiple options.

[0510] 6. Personalization with user input

[0511] Users input their eating habits, preferences, allergy information, etc. through an application or web interface, and the server uses this information to suggest individually optimized recipes for the user.

[0512] Example of program processing

[0513] 1. Recognizing ingredients

[0514] The user places a new ingredient (e.g., cabbage) in the refrigerator.

[0515] The server analyzes the image sent from the camera inside the refrigerator and recognizes the cabbage. The recognition result and the expiration date of the cabbage are recorded in a database.

[0516] 2. Best before date management

[0517] The server periodically checks the database and finds that the cabbage's expiration date is two days away.

[0518] The server sends a notification to the terminal such as "The cabbage's expiration date is in two days."

[0519] 3. Recipe suggestions

[0520] The server generates recipes that use cabbage (e.g., cabbage rolls, cabbage salad).

[0521] The device suggests these recipes to the user.

[0522] When a user selects a recipe for cabbage rolls, detailed instructions and ingredients are displayed on the device.

[0523] In this way, the system of the present invention efficiently manages ingredients and supports the user's eating habits through ingredient recognition, expiration date management, notifications, recipe suggestions, and personalization based on user-entered information.

[0524] The processing flow will be explained below.

[0525] Step 1:

[0526] The user places a new ingredient (e.g., cabbage) in the refrigerator.

[0527] The refrigerator periodically takes pictures of the interior using a built-in camera.

[0528] Step 2:

[0529] Image data captured by the camera means is transmitted to the server.

[0530] The server receives the image data.

[0531] Step 3:

[0532] The server processes the image data and uses an AI model to recognize the type of food ingredient (e.g., cabbage).

[0533] The recognition results are recorded in a database.

[0534] Step 4:

[0535] The server automatically estimates the expiration date of the recognized ingredients and stores it in a database.

[0536] The server periodically checks the database to identify ingredients that are approaching their expiration date.

[0537] Step 5:

[0538] The server prepares a notification to the user for ingredients that are nearing their expiration date.

[0539] A notification such as "The cabbage's expiration date is in two days" is sent to the device.

[0540] Step 6:

[0541] The device displays a push or email notification to the user.

[0542] The user checks the notification and becomes aware of the presence of food items (e.g., cabbage) that are nearing their expiration date.

[0543] Step 7:

[0544] The server runs an algorithm to suggest recipes using cabbage (e.g., cabbage rolls, cabbage salad).

[0545] A proposal list is created and sent to the terminal.

[0546] Step 8:

[0547] The terminal displays a list of suggestions to the user.

[0548] The user selects the cabbage rolls recipe from the list of suggestions.

[0549] Step 9:

[0550] The server sends detailed recipe information (steps and ingredients) for the cabbage rolls to the terminal.

[0551] The terminal displays the detailed information to the user.

[0552] Step 10:

[0553] Users enter their eating habits, preferences, allergy information, etc. through the application.

[0554] The server stores this input information in a database and reflects it in future recipe suggestions.

[0555] Through this series of processing steps, users can efficiently manage and use ingredients, and improve the quality of their diet by utilizing optimal recipes.

[0556] Example 1

[0557] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0558] There is a need for efficient management of food in refrigerators, reducing food waste, and providing users with appropriate and timely recipe suggestions. However, existing refrigerator management systems have issues with food recognition accuracy, expiration date management, user notifications, and recipe suggestions, making it difficult to perform multiple different functions in an integrated manner.

[0559] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0560] In this invention, the server includes an image acquisition means, a means for processing image data captured using the image acquisition means and recognizing the type of food from the image data, a means for managing the expiration dates of the recognized foods, a means for periodically checking the database to confirm the expiration dates of the foods and notifying the user when the expiration date is approaching, a means for generating cooking methods based on the notified foods with an approaching expiration date and providing them to the user, and a means for making personalized suggestions based on information input by the user. This allows multiple different functions to be executed in an integrated manner as a series of processes, making it possible to improve the efficiency of food management, reduce waste, and even suggest recipes optimized for the user.

[0561] The "image acquisition means" is a device such as a camera or sensor that takes a picture of the food in the refrigerator and acquires the image data.

[0562] "Means for recognizing food types" refers to algorithms or AI models that analyze acquired image data and identify the food in the image.

[0563] A "best-before date management tool" is software or algorithms that record the best-before dates of recognized food products in a database and periodically check those dates.

[0564] "Means of checking the database to verify expiration dates" refers to the process or method of periodically checking food information in the database to identify foods that are approaching their expiration date.

[0565] The "notification means" refers to a push notification, email notification, or other notification method for informing the user about food products that are approaching their expiration date.

[0566] "Method for generating cooking instructions" refers to an algorithm or AI model that automatically generates recipes and cooking instructions based on specific foods.

[0567] The "means for providing to the user" refers to a terminal or application interface for displaying the generated recipes and cooking methods to the user.

[0568] "Means for personalized suggestions" refers to customization features and algorithms that provide optimized recipes and cooking methods based on the user's eating habits and preferences.

[0569] This invention is a system that efficiently manages ingredients in a refrigerator and suggests suitable recipes. The system consists of an image acquisition unit for monitoring the status of the refrigerator, a server for processing image data, and a terminal that notifies the user and suggests recipes.

[0570] Image Acquisition Method

[0571] A camera is installed inside the refrigerator and periodically takes pictures of the inside of the refrigerator. The camera has high resolution and can clearly capture each ingredient. The captured image data is sent to a server via wireless or wired communication.

[0572] Image data processing by the server

[0573] The server receives the image data sent from the refrigerator and uses an AI model (e.g., a model trained with TensorFlow or PyTorch) to recognize the type of food. The recognized food information is stored in a database. The server also estimates the expiration date of each food item and records this information in the database.

[0574] Best before date management

[0575] The server periodically checks the expiration dates of ingredients stored in the database. It lists ingredients whose expiration dates are approaching and prepares to notify users based on that information. The expiration date management algorithm estimates expiration dates based on, for example, the type of ingredient and the date of purchase.

[0576] Notification means

[0577] The server sends information about food items approaching their expiration date to a device (e.g., a smartphone or tablet). The device then sends a push notification or email notification to the user based on the received information.

[0578] Recipe generation and suggestions

[0579] The server generates recipes based on ingredients that are approaching their expiration date. The recipe generation uses a database built in the past and information entered by the user (e.g., preferences and allergies). The generated recipes are sent to the terminal and provided to the user as multiple options.

[0580] Personalization based on user input

[0581] Users input their eating habits, preferences, and allergies through an application or web interface, and the server uses this information to generate and suggest individually optimized recipes to the user.

[0582] Specific examples of operation

[0583] When a user places a new ingredient (e.g., cabbage) in the refrigerator, the camera takes a picture of it and sends it to the server. The server analyzes the image to recognize the cabbage and records its information and expiration date in a database. When the expiration date approaches, the server sends a notification to the device stating, "The cabbage will expire in two days." The device receives this notification and notifies the user as a push notification. The server then generates recipes using cabbage (e.g., cabbage rolls, cabbage salad) and sends them to the device. The device presents these recipes to the user, and if the user selects a cabbage roll recipe, it displays detailed instructions and the necessary ingredients.

[0584] Example prompts for generative AI models

[0585] "I'm storing some cabbage and it will expire in two days. Can you suggest some recipes using cabbage? Please take into consideration recipes that users have liked in the past."

[0586] In this way, the system of the present invention can effectively manage ingredients and support the user's eating habits through ingredient recognition, expiration date management, user notifications, recipe suggestions, and personalization based on user-entered information.

[0587] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0588] Step 1:

[0589] The camera periodically takes high-resolution images of the inside of the refrigerator. The input is an image of the food inside the refrigerator, and the output is the captured image data. The captured image data is sent to a server via a wireless or wired network. Specifically, the camera automatically releases the shutter and transmits the resulting image as digital data.

[0590] Step 2:

[0591] The server processes the image data received from the camera. The input is the image data of the food sent from the camera, and the output is the recognized food information (e.g., food name, quantity, etc.). Specifically, the server analyzes the image using an AI model (e.g., TensorFlow or PyTorch) and extracts the identified ingredient information. This analysis uses image processing algorithms and machine learning models.

[0592] Step 3:

[0593] The server estimates the expiration date based on the recognized food ingredient information. The input is the recognized food information, and the output is the estimated expiration date for each food item. Specifically, the algorithm calculates the expiration date based on the type of food ingredient, storage conditions, and past data. As a result, the expiration date information is recorded in a database.

[0594] Step 4:

[0595] The server periodically checks the database to identify ingredients that are approaching their expiration date. The input is all food information in the database, and the output is a list of ingredients that are approaching their expiration date. Specifically, a scheduling program traverses the database and lists ingredients that are within three days of their expiration date.

[0596] Step 5:

[0597] The server sends information about ingredients that are approaching their expiration date to the device. The input is a list of ingredients that are approaching their expiration date, and the output is a notification message to be sent to the user. Specifically, the server creates a push notification or email notification and sends it to the device. This notification contains a specific message, such as "The cabbage will expire in two days."

[0598] Step 6:

[0599] The device displays the notification message received from the server to the user. The input is the notification message sent from the server, and the output is the notification information displayed to the user. Specifically, the message is displayed on the device screen as a push notification or email notification.

[0600] Step 7:

[0601] The server generates recipes based on ingredients that are approaching their expiration date. The input is information about ingredients that are approaching their expiration date, and the output is a list of generated recipes. Specifically, the server generates multiple recipes based on a database of past recipes and user preferences. This process uses AI models and algorithms.

[0602] Step 8:

[0603] The server sends the generated recipes to the terminal. The input is a list of generated recipes, and the output is recipe information provided to the user. In concrete terms, the server sends recipe information to the terminal, and the terminal receives it.

[0604] Step 9:

[0605] The device presents the received recipe information to the user. The input is the recipe information sent from the server, and the output is a list of recipes displayed to the user. Specifically, the device displays multiple recipe options on the screen and displays detailed instructions and required ingredients for the recipe selected by the user.

[0606] Step 10:

[0607] A user provides input information, such as preferences and allergies, through an application or web interface. The input is the information entered by the user, and the output is the personalized information sent to the server. Specifically, the user enters information through a digital interface, and the information is sent to the server.

[0608] This series of processes realizes an overall flow from food recognition to expiration date management and recipe suggestions, effectively supporting the user's food management and eating habits.

[0609] (Application example 1)

[0610] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0611] Conventional refrigerator food management systems were limited to managing food ingredients within the home, and were unable to adequately manage ingredients or suggest recipes based on ingredients approaching their expiration date when purchasing at a physical store (such as a supermarket). Furthermore, it was not possible to track the history of ingredients purchased by the user and manage expiration dates or suggest recipes based on that information. This made food management cumbersome and led to the risk of food waste.

[0612] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0613] In this invention, the server includes a camera means for monitoring the status inside the refrigerator, a means for processing image data captured by the camera means and recognizing the type of ingredients from the image data, a means for managing the expiration dates of the recognized ingredients, a means for notifying the user of ingredients approaching their expiration dates, a means for generating recipes corresponding to the ingredients and suggesting them to the user, a means for making personalized suggestions based on information input by the user, and a means for managing ingredient purchase histories and presenting ingredients approaching their expiration dates in purchasing activities at physical stores. This enables ingredient expiration date management and recipe suggestions not only at home but also in purchasing activities at physical stores, enabling efficient ingredient management and reduction of food waste.

[0614] The "camera means" is a device for taking pictures of the inside of the refrigerator and acquiring the image data.

[0615] "Means for processing image data and recognizing the type of food ingredient from the image data" refers to technology for analyzing image data obtained from a camera means and identifying the type of food ingredient using an AI model, etc.

[0616] The "means for managing the expiration dates of recognized ingredients" is a system for storing the expiration dates of recognized ingredients in a database and periodically checking the expiration dates.

[0617] The "means for notifying users about food ingredients that are approaching their expiration date" is a system for notifying users about food ingredients that are approaching their expiration date, and is a mechanism for sending notifications to smartphones or tablets.

[0618] "Means for generating recipes corresponding to ingredients and suggesting them to users" refers to technology for generating appropriate recipes based on the types of ingredients and expiration dates and providing them to users.

[0619] "Means for making personalized suggestions based on user input" refers to a system that takes into account the user's preferences, eating habits, allergy information, etc., and suggests individually optimized recipes based on that information.

[0620] "Means for managing food purchase history and presenting food items approaching their expiration date during purchasing activities at physical stores" is a technology that stores the history of food items purchased at physical stores in a database and presents food items approaching their expiration date to the user.

[0621] The "means for periodically checking expiration dates" is a system that periodically checks the expiration dates of ingredients stored in a database.

[0622] The "means for generating a recipe including ingredients to be used based on a notification" is a technology that automatically generates a recipe including ingredients to be used based on ingredients whose expiration date is approaching.

[0623] The "terminal means for providing the generated recipe to the user" is a device for displaying the generated recipe to the user, such as a smartphone or tablet.

[0624] This system efficiently manages ingredients in a refrigerator and suggests suitable recipes, and also enables ingredient management and recipe suggestions for purchases in physical stores. This system consists of three main components: a camera device in the refrigerator, a server, and a user terminal.

[0625] System Components

[0626] 1. Camera Means

[0627] A camera installed inside the refrigerator periodically takes pictures of the interior. The high-resolution camera captures clear images of each ingredient. The captured image data is sent to a server.

[0628] 2. Image data processing by the server

[0629] The server receives the image data sent from the refrigerator and uses the generative AI model to recognize the type of food. The recognition results are recorded in a database. The server also estimates the expiration date, and this information is also stored in the database.

[0630] 3. Best before date management

[0631] The server periodically checks the expiration dates of ingredients stored in the database and lists ingredients that are approaching their expiration date, and prepares to notify the user of this information.

[0632] 4. Means of notification

[0633] The server sends notifications about food items approaching their expiration date to the user's device, such as a smartphone or tablet, via push notifications or email.

[0634] 5. Recipe generation and suggestions

[0635] The server generates recipes based on ingredients that are approaching their expiration date. The recipes are generated using a database built in the past and user input information (preferences, allergies, etc.). The generated recipes are provided to the user.

[0636] 6. Personalization with user input

[0637] Users input their eating habits, preferences, allergy information, etc. through the application, and the server uses this information to suggest individually optimized recipes.

[0638] 7. Managing food purchase history

[0639] The system also manages the history of food purchases at physical stores, and uses the purchase history database to suggest recipes based on ingredients that are close to their expiration date.

[0640] Hardware and software used

[0641] Hardware

[0642] Camera Method: High resolution camera installed inside the refrigerator.

[0643] User devices: smartphones, tablets.

[0644] Server: A cloud server such as AWS or GCP.

[0645] software

[0646] Image processing: Python, OpenCV, TensorFlow.

[0647] Server side: Django framework.

[0648] Database Management: Django ORM.

[0649] Communication method: Communication using REST API.

[0650] Specific examples of use

[0651] For example, if a user places "tomatoes" and "lettuce" in the refrigerator, the camera takes a picture of them and the server recognizes the type of food. The recognized food is stored in a database, and the user is notified when the expiration date approaches. At the same time, a recipe for "tomato and lettuce salad" is suggested. If the user purchases "chicken" at a physical store, the purchase history is stored in the database and used to suggest recipes for the next time.

[0652] Prompt Sentence Examples

[0653] "Please tell me some recipes that use the tomatoes, lettuce, and chicken I have in my fridge. The user prefers low-calorie recipes, and is allergic to nuts."

[0654] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0655] Step 1:

[0656] A camera means in the refrigerator periodically takes pictures of ingredients. The captured image data is sent to a server. The input is the image of the ingredients in the refrigerator, and the output is the image data sent to the server. The camera takes pictures when the camera button is pressed, or automatically at set intervals.

[0657] Step 2:

[0658] The server analyzes the received image data using a generative AI model to recognize the type of ingredient. The input is the image data sent in step 1, and the output is the recognized ingredient information. The analysis process is performed using Python and TensorFlow.

[0659] Step 3:

[0660] The server stores the information of the recognized ingredients in a database. The database records the type of ingredient, quantity, purchase date, and expiration date. The input is the ingredient information recognized in step 2, and the output is an organized ingredient database. It is saved to the database using Django ORM.

[0661] Step 4:

[0662] The server periodically checks the database and lists ingredients that are approaching their expiration date. The input is the ingredient information in the database, and the output is a list of ingredients that are approaching their expiration date. The checking process is performed by a Python script.

[0663] Step 5:

[0664] The server sends notifications to the user's device about ingredients that are approaching their expiration date. The input is the ingredient information listed in step 4, and the output is a notification sent to the user's smartphone or tablet. Notifications are sent via push notifications or email.

[0665] Step 6:

[0666] The server generates recipes based on ingredients that are approaching their expiration date. Multiple recipes are created using the generative AI model and the user's eating habits, preferences, and allergy information. The input is the ingredient information from step 4 and the user's personal information, and the output is the generated recipe. Prompt statements can be used to generate recipes.

[0667] Step 7:

[0668] The server provides the generated recipe to the user's device. The input is the recipe generated in step 6, and the output is the recipe displayed on the user's smartphone or tablet through the application interface.

[0669] Step 8:

[0670] When a user buys ingredients in a physical store, the purchase history is stored in a database. The input is the user's purchase information, and the output is an updated ingredient purchase history database. The database is managed using Django ORM.

[0671] Step 9:

[0672] The server uses the purchase history database to create a list of food items purchased in physical stores that are nearing their expiration date, and presents this to the user. The input is the updated purchase history database, and the output is a new notification sent to the user's device. The notification is displayed on a smartphone or tablet.

[0673] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0674] This invention is a system that efficiently manages ingredients in a refrigerator and suggests appropriate recipes, and by combining it with an emotion engine that recognizes the user's emotions, it provides more personalized dietary support. This system is composed of a camera for monitoring the status of the refrigerator, a server that processes image data, a terminal that notifies the user and suggests recipes, and an emotion engine that recognizes the user's emotions. Each component and its operation are described in detail below.

[0675] System configuration

[0676] 1. Camera Means

[0677] A camera installed inside the refrigerator periodically takes pictures of the inside of the refrigerator. The camera has high resolution and can clearly capture each food item inside the refrigerator. The captured image data is sent to a server.

[0678] 2. Image data processing by the server

[0679] The server receives the image data sent from the refrigerator and uses an AI model to identify the type of food. The identified food is then stored in a database. The server also estimates the expiration date for each food item and records this information in the database.

[0680] 3. Best before date management

[0681] The server periodically checks the expiration dates of ingredients stored in the database, lists ingredients that are approaching their expiration date, and prepares to notify the user of the listed ingredients.

[0682] 4. Means of notification

[0683] The server sends information about food items approaching their expiration date to the user's device, which can be a smartphone or tablet, and can send push notifications or email notifications to the user.

[0684] 5. Recipe generation and suggestions

[0685] The server generates recipes based on ingredients that are approaching their expiration date. The recipe generation uses a database built in the past and information entered by the user (preferences, allergies, etc.). The generated recipes are presented to the user as multiple options.

[0686] 6. Personalization with user input

[0687] Users input their eating habits, preferences, allergy information, etc. through an application or web interface, and the server uses this information to suggest individually optimized recipes for the user.

[0688] 7. Emotion Engine

[0689] The emotion engine recognizes emotions from the user's voice, facial expressions, text input, etc. The user's emotion information recognized by the emotion engine is also stored in the database.

[0690] Example of program processing

[0691] 1. Recognizing ingredients

[0692] The user places a new ingredient (e.g., cabbage) in the refrigerator.

[0693] The server analyzes the image sent from the camera inside the refrigerator and recognizes the cabbage. The recognition result and the expiration date of the cabbage are recorded in a database.

[0694] 2. Best before date management

[0695] The server periodically checks the database and finds that the cabbage's expiration date is two days away.

[0696] The server sends a notification to the terminal such as "The cabbage's expiration date is in two days."

[0697] 3. Recipe suggestions

[0698] The server generates recipes that use cabbage (e.g., cabbage rolls, cabbage salad).

[0699] The device suggests these recipes to the user.

[0700] When a user selects a recipe for cabbage rolls, detailed instructions and ingredients are displayed on the device.

[0701] 4. Personalization with user input

[0702] Users enter their eating habits, preferences, allergy information, etc. through the application.

[0703] The server stores this information in a database and reflects it in future recipe suggestions.

[0704] 5. Emotion recognition and suggestion optimization

[0705] The emotion engine recognizes emotions from the user's voice and facial expressions and sends this information to the server.

[0706] The server then tailors recipes and notifications to suit the user based on their emotional information, for example, suggesting easy-to-make recipes if the user is tired.

[0707] In this way, the system of the present invention provides optimal dietary support to users by combining ingredient recognition, expiration date management, notifications, recipe suggestions, personalization based on user-entered information, and emotion recognition using an emotion engine.

[0708] The processing flow will be explained below.

[0709] Step 1:

[0710] The user places a new ingredient (e.g., cabbage) in the refrigerator.

[0711] The refrigerator periodically takes pictures of the interior using a built-in camera.

[0712] Step 2:

[0713] Image data captured by the camera means is transmitted to the server.

[0714] The server receives the image data.

[0715] Step 3:

[0716] The server analyzes the image data and uses an AI model to recognize the type of food ingredient (e.g., cabbage).

[0717] The recognition results are recorded in a database.

[0718] Step 4:

[0719] The server automatically estimates the expiration date of the recognized ingredients and stores it in a database.

[0720] The server periodically checks the database to identify ingredients that are approaching their expiration date.

[0721] Step 5:

[0722] The server prepares a notification to the user for listed ingredients that are close to their expiration date.

[0723] A notification such as "The cabbage's expiration date is in two days" is sent to the device.

[0724] Step 6:

[0725] The device displays the push notification to the user.

[0726] The user checks the notification and recognizes ingredients (e.g., cabbage) that are nearing their expiration date.

[0727] Step 7:

[0728] The server requests data to check the user's emotions with an emotion engine.

[0729] The device collects the user's voice and facial expression data and sends it to the server.

[0730] Step 8:

[0731] The emotion engine analyzes the user's emotions (e.g., if the user is tired).

[0732] The server records the user's emotional information recognized by the emotion engine in a database and uses it to adjust the recipe.

[0733] Step 9:

[0734] The server executes a recipe algorithm based on the user's ingredient data and emotion data.

[0735] For example, if the cabbage is nearing its expiration date and the user is tired, the app will suggest an easy recipe (e.g., easy stir-fried cabbage).

[0736] Step 10:

[0737] The server generates recipe suggestions and sends them to the device.

[0738] The device displays recipe suggestions to the user.

[0739] When a user selects a recipe, detailed instructions and required ingredients are displayed.

[0740] Step 11:

[0741] Users enter their eating habits, preferences, allergy information, etc. through the application.

[0742] The server stores this information in a database and reflects it in future recipe suggestions.

[0743] In this way, the system provides optimal dietary support to users by recognizing ingredients, managing expiration dates, providing notifications, suggesting recipes, personalizing with user-entered information, and even recognizing emotions using an emotion engine.

[0744] Example 2

[0745] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0746] In modern households, managing food in the refrigerator is complicated, and food that has passed its expiration date is often wasted. Furthermore, when it comes to effectively utilizing ingredients and suggesting recipes, there is a lack of personalization that takes into account individual users' preferences and allergy information. Furthermore, there is no system that can make appropriate suggestions based on the user's emotional state.

[0747] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a camera means for monitoring the state inside the refrigerator, a means for processing image data captured by the camera means and recognizing the types of ingredients from the image data, a means for estimating and managing the expiration dates of the recognized ingredients, a means for notifying the user of ingredients whose expiration dates are approaching, a means for generating recipes based on the ingredients and suggesting them to the user, a means for making personalized suggestions based on information input by the user, and a means for recognizing the user's emotions from their voice, facial expressions, text input, etc., and adjusting the suggestions. This reduces ingredient waste, makes recipe suggestions that meet the user's individual preferences, and enables optimal suggestions based on the user's emotional state.

[0748] The "camera means" refers to a camera device installed to monitor the state inside the refrigerator, and has the function of taking pictures of ingredients.

[0749] "Image data" refers to visual information of the inside of a refrigerator photographed by a camera means, which is recorded and stored in digital format.

[0750] The "means for recognizing the type of food ingredient" is a function that analyzes the captured image data and identifies the name and attributes of the food ingredient using AI or machine learning models.

[0751] The "means for estimating and managing expiration dates" has the function of predicting and managing expiration dates for recognized food ingredients based on the purchase date and general storage period.

[0752] The "notification means" has a function for transmitting information about ingredients approaching their expiration date, recipe suggestions, etc. to the user's terminal.

[0753] The "means for generating recipes" has the function of automatically creating cooking instructions and steps based on the recognized ingredient information.

[0754] The "means for suggesting to the user" has a function for displaying the generated recipe and notification content to the user.

[0755] The "means for making personalized suggestions" has the function of generating recipes and notification content that take into account individual preferences and allergies based on the user's input information and past data.

[0756] The "means for recognizing emotions and adjusting suggested content" has the function of analyzing the user's current emotional state from their voice, facial expressions, text input, etc., and providing recipes and notification content accordingly.

[0757] This invention is a system that efficiently manages ingredients in the refrigerator and suggests suitable recipes, and by combining it with an emotion engine that recognizes the user's emotions, it provides more personalized dietary support. This system operates based on the following components.

[0758] 1. Camera Means

[0759] The camera installed inside the refrigerator takes high-resolution images of the interior of the refrigerator. The camera periodically (for example, every hour) takes a picture of the overall situation inside the refrigerator and sends this image data to a server. It is desirable to use a network camera with a resolution of 1080p or higher.

[0760] 2. Image data processing by the server

[0761] The server receives the image data sent from the camera and performs image analysis. This image analysis uses AI models such as TensorFlow and PyTorch. This automatically recognizes the types of ingredients in the refrigerator and stores the results in a database. Specifically, it uses the YOLOv3 model to perform high-speed, high-precision object recognition.

[0762] 3. Estimation and management of expiration dates

[0763] The server estimates the expiration date for the recognized ingredients. For example, if it recognizes cabbage, it calculates the expiration date based on the typical storage period for cabbage (about 5 days in the refrigerator) and records this information in the database. The database is a relational database such as MySQL or PostgreSQL.

[0764] 4. Sending notifications

[0765] The server periodically checks the information in the database and sends notifications to the user's device about ingredients that are approaching their expiration date. This notification is done via push notification or email. For example, if the expiration date of cabbage is approaching in two days, a message such as "The expiration date of the cabbage is in two days" is generated and sent to the user's smartphone.

[0766] 5. Recipe generation and suggestions

[0767] The server uses a database of past recipes and information entered by the user (such as preferences and allergies) to generate recipes based on ingredients approaching their expiration date. The generated recipes are presented to the user as multiple options. For example, recipes such as "stuffed cabbage" and "cabbage salad" using cabbage may be generated.

[0768] 6. Personalization with user input

[0769] Users input their eating habits, preferences, allergies, and other information through the application or web interface. The server stores this information in a database and uses it to suggest future recipes, allowing users to receive recipes tailored to their individual needs.

[0770] 7. Emotion Recognition with Emotion Engine

[0771] The emotion engine recognizes emotions from the user's voice, facial expressions, and text input. This emotion information is also sent to the server and stored in a database. The server uses this information to tailor recipes and notification content to suit the user. For example, if the emotion engine recognizes that the user is tired, it will prioritize suggesting easy recipes.

[0772] Examples and prompts

[0773] Example: When a user puts a new cabbage in the refrigerator, the camera recognizes this and sends data to the server to calculate the expiration date of the cabbage. When the expiration date approaches, the server sends a notification and generates several recipes using the cabbage and displays them on the device.

[0774] Example prompt: "Recognize the new ingredients in your refrigerator and suggest a recipe using them."

[0775] In this way, the system of the present invention comprehensively supports everything from managing ingredients in the refrigerator to suggesting recipes for each user, and even suggesting the best recipes based on the user's emotional state. Introducing this system will reduce food waste and lead to a richer and healthier diet for users.

[0776] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0777] Step 1:

[0778] Taking pictures with a camera

[0779] Input: Condition inside the refrigerator

[0780] Process: The camera takes high-resolution images of the inside of the refrigerator periodically (e.g., every hour). Use a network camera with a resolution of 1080p or higher.

[0781] Output: Image data of the inside of the refrigerator

[0782] Specific operation: The location and condition of all food items in the refrigerator are clearly photographed and data is generated.

[0783] Step 2:

[0784] Sending image data

[0785] Input: Image data of the inside of the refrigerator

[0786] Processing: Image data generated by the camera is sent to the server in real time via Wi-Fi.

[0787] Output: Image data sent to the server

[0788] Specific operation: The image file taken by the camera is compressed and a protocol (e.g. HTTP) is executed to send it to the server.

[0789] Step 3:

[0790] Receiving and analyzing image data

[0791] Input: Image data sent from the camera

[0792] Processing: The server receives the image data and uses an AI model (e.g., TensorFlow or PyTorch) to recognize the type of food. Specifically, the YOLOv3 model is used.

[0793] Output: Recognized food type and location information

[0794] Specific operation: The server analyzes the received image data, identifies the name and location of each ingredient, and records it in a database.

[0795] Step 4:

[0796] Estimating expiration dates and storing

[0797] Input: Recognized food type and location information

[0798] Processing: The server estimates the shelf life of each ingredient based on its type. For example, cabbage can be stored in the refrigerator for 5 days.

[0799] Output: Best before date data for each ingredient

[0800] What it does: Calculates the expiration date for each ingredient based on the purchase date and shelf life, and records that information in a database.

[0801] Step 5:

[0802] Regular check of expiration dates

[0803] Input: Best before date data stored in the database

[0804] Processing: The server periodically (e.g., every night at midnight) scans the database and lists ingredients that are nearing their expiration date.

[0805] Output: A list of ingredients that are nearing their expiration date

[0806] What it does: Query the database to find ingredients with a shelf life of less than 3 days.

[0807] Step 6:

[0808] Creating a notification message and preparing it for sending

[0809] Input: A list of ingredients that are nearing their expiration date

[0810] Processing: The server generates a notification message for the listed ingredients, for example, "The cabbage will expire in 2 days."

[0811] Output: The generated notification message

[0812] Specific operation: Automatically generate notification content using a message template and prepare to send it to the user's device.

[0813] Step 7:

[0814] Sending notifications

[0815] Input: The generated notification message

[0816] Process: The server sends a notification message to the user's device via push notification or email notification.

[0817] Output: Notification message received by the user

[0818] Specific operation: The server sends a message to the user's device via push notification or email system.

[0819] Step 8:

[0820] Receiving notifications

[0821] Input: Notification message sent by the server

[0822] Processing: The device receives the notification and displays it on the user's screen as a popup or email.

[0823] Output: The notification message displayed to the user

[0824] Specific behavior: The device's notification system receives the message from the server and immediately displays it to the user.

[0825] Step 9:

[0826] Recipe Generation

[0827] Input: List of ingredients approaching expiration date and user database

[0828] Processing: The server generates recipes based on ingredients that are nearing their expiration date, using a database of past data and user preferences and allergy information.

[0829] Output: A list of generated recipes

[0830] Specific operation: The system generates multiple appropriate recipes based on the ingredients data and the user's profile. For example, it suggests "cabbage rolls" and "cabbage salad" using cabbage.

[0831] Step 10:

[0832] Recipe Suggestions

[0833] Input: A list of generated recipes

[0834] What happens: The device presents the suggested recipes to the user. When the user selects a specific recipe, detailed instructions and required ingredients are displayed.

[0835] Output: Recipe details

[0836] Specific behavior: Displays a list of recipes on the device screen and provides detailed information about the recipe selected by the user.

[0837] Step 11:

[0838] Entering user information

[0839] Input: User-provided dietary habits, preferences, and allergy information

[0840] Processing: The user enters information through an application or web interface and sends it to the server.

[0841] Output: User information stored in the database

[0842] What happens: The user uses the interface to enter their information and presses the submit button. The server receives this and records it in the database.

[0843] Step 12:

[0844] Storage and Use of User Information

[0845] Input: User-submitted dietary habits, preferences, and allergy information

[0846] Processing: The server stores this information in a database and uses it to suggest recipes from the next time onwards.

[0847] Output: Updated user profile

[0848] Specific operation: The server uses the user's input information to personalize future recipe suggestions and notifications.

[0849] Step 13:

[0850] emotion recognition

[0851] Input: User voice, facial expressions, and text input

[0852] Processing: The emotion engine grasps the user's emotion and sends it to the server.

[0853] Output: Recognized user emotion information

[0854] Specific operation: The emotion engine analyzes the user's input data, grasps the user's emotional state, and sends that information to the server.

[0855] Step 14:

[0856] Use of emotional information

[0857] Input: Recognized user emotion information

[0858] Processing: The server records the emotion information in a database and adjusts the suggestions to the user based on this information.

[0859] Output: Adjusted proposal

[0860] Specific operation: The server refers to the emotional information and provides the user with the most appropriate content according to their emotions, such as suggesting simple recipes if the user is tired.

[0861] (Application example 2)

[0862] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0863] In recent years, refrigerator food management and food waste issues have been attracting attention, but traditional manual and simple digital management methods have their limitations. Furthermore, managing ingredients in brick-and-mortar stores and effectively suggesting recipes to customers requires a great deal of effort. Furthermore, while there is a demand for services that take customer emotions into account, no effective system exists to achieve this. To solve these issues, it is necessary to streamline refrigerator food management, provide appropriate information to store staff and customers, and provide personalized services that take customer emotions into account.

[0864] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0865] In this invention, the server includes means for processing image data captured by a camera means and recognizing the type of ingredient from the image data, means for managing the expiration dates of the recognized ingredients, means for notifying the user of ingredients approaching their expiration dates, means for generating recipes corresponding to the ingredients and suggesting them to the user, means for making personalized suggestions based on information input by the user, means for efficiently managing ingredients in a refrigerator and monitoring products stored in refrigerators in the store, means for notifying store staff of products approaching their expiration dates, and means for recognizing customer emotions and suggesting recipes or promoting purchases based on the emotions. This enables efficient management of ingredients in a refrigerator, appropriate management of ingredients in a physical store, personalized recipe suggestions to customers, and service provision based on customer emotions.

[0866] The "camera means" is a photographing device installed to monitor the state inside the refrigerator, and periodically takes pictures of the food ingredients.

[0867] "Image data" refers to video information of ingredients photographed using a camera, and is digital data that is analyzed by the server.

[0868] A "means for recognizing" is a method or device capable of processing image data and identifying and classifying objects in the image.

[0869] The "means for managing expiration dates" refers to a method or device for recording the expiration dates of recognized food ingredients and for notifying users when the expiration date is approaching.

[0870] "Notification means" refers to a method or device used to notify a user or store staff of specific information.

[0871] The "means for generating and suggesting recipes" refers to a method or device that creates a cooking method based on recognized ingredients and provides it to the user.

[0872] The "means for making personalized suggestions" refers to a method or device that makes individually optimized suggestions based on information input by the user, past data, and the like.

[0873] "Means for monitoring products" refers to a method or device for monitoring the condition of products stored in refrigerators in a store and understanding product trends and inventory status.

[0874] A "means for recognizing customer emotions" is a method or device that determines a customer's emotions from voice, facial expressions, text input, etc., and utilizes that information.

[0875] "Means for promoting purchases" are methods and devices for making individually optimized product suggestions based on customer emotions and behavioral data, thereby increasing purchasing motivation.

[0876] The present invention provides a system for improving the efficiency of food ingredient management in a refrigerator and product management in a physical store, and for suggesting personalized recipes to customers and promoting purchases. The following describes in detail an embodiment of the present invention.

[0877] First, regarding the hardware configuration of the entire system, a high-resolution camera is installed inside the refrigerator. This camera periodically takes pictures of the ingredients and products inside the store refrigerator and sends the image data to a server. A configuration using a virtual server on AWS EC2 is suitable for this server.

[0878] The server receives image data sent from the refrigerator and uses an image recognition AI model using TensorFlow to identify ingredients and product types. The identified ingredients are then stored in a database (AWS RDS).

[0879] The server then periodically checks the database and lists ingredients and products that are approaching their expiration date. Store staff are notified of these ingredients and products via push notifications sent via smartphones or tablets. The server also generates recipes based on the products that are approaching their expiration date.

[0880] Recipe generation utilizes previously constructed databases and information entered by users (such as preferences and allergies). The generated recipes are provided to customers via a sales promotion touch panel display and a customer application.

[0881] Furthermore, an emotion engine will be used to recognize emotions from the customer's voice and facial expressions. This emotion information will also be sent to the server, and personalized recipe suggestions and purchase promotions will be made based on the emotion. Microsoft Azure Cognitive Services could be used for the emotion engine.

[0882] The system includes the following specific processing steps:

[0883] 1. The camera means takes an image of the food in the refrigerator.

[0884] 2. The captured image data is sent to the server.

[0885] 3. The server analyzes the image using a TensorFlow model and recognizes the ingredients.

[0886] 4. The recognition results and expiration date information are stored in the database.

[0887] 5. The server periodically checks the database and notifies store staff of any food items that are nearing their expiration date.

[0888] 6. The server generates recipes for customers using ingredients that are close to their expiration date and provides them via a touch panel display or app.

[0889] 7. The emotion recognition engine recognizes the customer's emotions and sends the emotional information to the server.

[0890] 8. Based on the emotional information, the server will suggest recipes and promote purchases according to the emotions.

[0891] For example, consider the following prompt:

[0892] "A new cabbage is placed in the refrigerator. Use an image recognition AI model to recognize the cabbage and add it to the expiration date management system. When the cabbage's expiration date approaches, notify the customer via the touch panel display with a recommended recipe. Also, recognize the customer's emotions based on their facial expressions and suggest recipes that match those emotions."

[0893] This will enable efficient management of food in the refrigerator, proper notification to store staff, and the provision of services that respond to customer emotions.

[0894] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0895] Step 1:

[0896] A camera means within the refrigerator takes images of the food items.

[0897] Input: Refrigerator status

[0898] Data processing: image data capture

[0899] Output: High-resolution food image data

[0900] Step 2:

[0901] The captured image data is sent to a server.

[0902] Input: High-resolution food image data

[0903] Data processing: Sending image data

[0904] Output: Image data received by the server

[0905] Step 3:

[0906] The server uses a TensorFlow model to analyze the image data and recognize the type of ingredient.

[0907] Input: Received food image data

[0908] Data processing: image recognition, food classification

[0909] Output: Recognized ingredients (e.g., tomato, cabbage)

[0910] Step 4:

[0911] The recognized ingredient information and expiration date information are stored in a database on the server.

[0912] Input: Recognized ingredient information

[0913] Data processing: Recording in database, estimating expiration date

[0914] Output: Stored ingredients and expiration date data

[0915] Step 5:

[0916] The server periodically checks the database and notifies store staff of food items that are nearing their expiration date.

[0917] Input: Stored ingredients and expiration date data

[0918] Data processing: Expiration date check, notification information creation

[0919] Output: Notification information (e.g. "The expiration date for the tomatoes is in 2 days")

[0920] Step 6:

[0921] The server generates recipes using ingredients that are close to their expiration date and transmits them to a terminal for serving to the customer.

[0922] Input: Information about ingredients that are close to expiry date, as well as user preferences and allergy information

[0923] Data processing: Recipe generation, recipe data creation

[0924] Output: Recipe suggestions (e.g., tomato salad, tomato pasta)

[0925] Step 7:

[0926] The generated recipe information is provided to the customer via a touch panel display or a customer application.

[0927] Input: Recipe to suggest

[0928] Data processing: Preparing to display recipe information

[0929] Output: Display of recipe (e.g., recipe displayed on a touch panel or smartphone)

[0930] Step 8:

[0931] The emotion engine recognizes emotions from the customer's voice and facial expressions and sends that information to the server.

[0932] Input: Customer voice and facial expression data

[0933] Data processing: Emotion recognition, creating data of the recognition results

[0934] Output: Recognized emotion information (e.g., "The customer is tired")

[0935] Step 9:

[0936] Based on the emotional information, the server makes personalized recipe suggestions or purchase promotions according to the emotions.

[0937] Input: Recognized emotion information

[0938] Data processing: Recipe suggestions readjustment, purchase promotion information creation

[0939] Output: Personalized recipe suggestions and promotional notifications (e.g., "Easy recipes perfect for a tiring day")

[0940] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0941] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0942] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0943] [Third embodiment]

[0944] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0945] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0946] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0947] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0948] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0949] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0950] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0951] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0952] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0953] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0954] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0955] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0956] This invention is a system that efficiently manages ingredients in a refrigerator and suggests suitable recipes. This system consists of a camera for monitoring the status of the refrigerator, a server for processing image data, and a terminal that notifies the user and suggests recipes. Each component and its operation are described in detail below.

[0957] System configuration

[0958] 1. Camera Means

[0959] A camera installed inside the refrigerator periodically takes pictures of the inside of the refrigerator. The camera has high resolution and can clearly capture each food item inside the refrigerator. The captured image data is sent to a server.

[0960] 2. Image data processing by the server

[0961] The server receives the image data sent from the refrigerator and uses an AI model to identify the type of food. The identified food is then stored in a database. The server also estimates the expiration date for each food item and records this information in the database.

[0962] 3. Best before date management

[0963] The server periodically checks the expiration dates of ingredients stored in the database, lists ingredients that are approaching their expiration date, and prepares to notify the user of the listed ingredients.

[0964] 4. Means of notification

[0965] The server sends information about food items approaching their expiration date to the user's device, which can be a smartphone or tablet, and can send push notifications or email notifications to the user.

[0966] 5. Recipe generation and suggestions

[0967] The server generates recipes based on ingredients that are approaching their expiration date. The recipe generation uses a database built in the past and information entered by the user (preferences, allergies, etc.). The generated recipes are presented to the user as multiple options.

[0968] 6. Personalization with user input

[0969] Users input their eating habits, preferences, allergy information, etc. through an application or web interface, and the server uses this information to suggest individually optimized recipes for the user.

[0970] Example of program processing

[0971] 1. Recognizing ingredients

[0972] The user places a new ingredient (e.g., cabbage) in the refrigerator.

[0973] The server analyzes the image sent from the camera inside the refrigerator and recognizes the cabbage. The recognition result and the expiration date of the cabbage are recorded in a database.

[0974] 2. Best before date management

[0975] The server periodically checks the database and finds that the cabbage's expiration date is two days away.

[0976] The server sends a notification to the terminal such as "The cabbage's expiration date is in two days."

[0977] 3. Recipe suggestions

[0978] The server generates recipes that use cabbage (e.g., cabbage rolls, cabbage salad).

[0979] The device suggests these recipes to the user.

[0980] When a user selects a recipe for cabbage rolls, detailed instructions and ingredients are displayed on the device.

[0981] In this way, the system of the present invention efficiently manages ingredients and supports the user's eating habits through ingredient recognition, expiration date management, notifications, recipe suggestions, and personalization based on user-entered information.

[0982] The processing flow will be explained below.

[0983] Step 1:

[0984] The user places a new ingredient (e.g., cabbage) in the refrigerator.

[0985] The refrigerator periodically takes pictures of the interior using a built-in camera.

[0986] Step 2:

[0987] Image data captured by the camera means is transmitted to the server.

[0988] The server receives the image data.

[0989] Step 3:

[0990] The server processes the image data and uses an AI model to recognize the type of food ingredient (e.g., cabbage).

[0991] The recognition results are recorded in a database.

[0992] Step 4:

[0993] The server automatically estimates the expiration date of the recognized ingredients and stores it in a database.

[0994] The server periodically checks the database to identify ingredients that are approaching their expiration date.

[0995] Step 5:

[0996] The server prepares a notification to the user for ingredients that are nearing their expiration date.

[0997] A notification such as "The cabbage's expiration date is in two days" is sent to the device.

[0998] Step 6:

[0999] The device displays a push or email notification to the user.

[1000] The user checks the notification and becomes aware of the presence of food items (e.g., cabbage) that are nearing their expiration date.

[1001] Step 7:

[1002] The server runs an algorithm to suggest recipes using cabbage (e.g., cabbage rolls, cabbage salad).

[1003] A proposal list is created and sent to the terminal.

[1004] Step 8:

[1005] The terminal displays a list of suggestions to the user.

[1006] The user selects the cabbage rolls recipe from the list of suggestions.

[1007] Step 9:

[1008] The server sends detailed recipe information (steps and ingredients) for the cabbage rolls to the terminal.

[1009] The terminal displays the detailed information to the user.

[1010] Step 10:

[1011] Users enter their eating habits, preferences, allergy information, etc. through the application.

[1012] The server stores this input information in a database and reflects it in future recipe suggestions.

[1013] Through this series of processing steps, users can efficiently manage and use ingredients, and improve the quality of their diet by utilizing optimal recipes.

[1014] Example 1

[1015] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1016] There is a need for efficient management of food in refrigerators, reducing food waste, and providing users with appropriate and timely recipe suggestions. However, existing refrigerator management systems have issues with food recognition accuracy, expiration date management, user notifications, and recipe suggestions, making it difficult to perform multiple different functions in an integrated manner.

[1017] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1018] In this invention, the server includes an image acquisition means, a means for processing image data captured using the image acquisition means and recognizing the type of food from the image data, a means for managing the expiration dates of the recognized foods, a means for periodically checking the database to confirm the expiration dates of the foods and notifying the user when the expiration date is approaching, a means for generating cooking methods based on the notified foods with an approaching expiration date and providing them to the user, and a means for making personalized suggestions based on information input by the user. This allows multiple different functions to be executed in an integrated manner as a series of processes, making it possible to improve the efficiency of food management, reduce waste, and even suggest recipes optimized for the user.

[1019] The "image acquisition means" is a device such as a camera or sensor that takes a picture of the food in the refrigerator and acquires the image data.

[1020] "Means for recognizing food types" refers to algorithms or AI models that analyze acquired image data and identify the food in the image.

[1021] A "best-before date management tool" is software or algorithms that record the best-before dates of recognized food products in a database and periodically check those dates.

[1022] "Means of checking the database to verify expiration dates" refers to the process or method of periodically checking food information in the database to identify foods that are approaching their expiration date.

[1023] The "notification means" refers to a push notification, email notification, or other notification method for informing the user about food products that are approaching their expiration date.

[1024] "Method for generating cooking instructions" refers to an algorithm or AI model that automatically generates recipes and cooking instructions based on specific foods.

[1025] The "means for providing to the user" refers to a terminal or application interface for displaying the generated recipes and cooking methods to the user.

[1026] "Means for personalized suggestions" refers to customization features and algorithms that provide optimized recipes and cooking methods based on the user's eating habits and preferences.

[1027] This invention is a system that efficiently manages ingredients in a refrigerator and suggests suitable recipes. The system consists of an image acquisition unit for monitoring the status of the refrigerator, a server for processing image data, and a terminal that notifies the user and suggests recipes.

[1028] Image Acquisition Method

[1029] A camera is installed inside the refrigerator and periodically takes pictures of the inside of the refrigerator. The camera has high resolution and can clearly capture each ingredient. The captured image data is sent to a server via wireless or wired communication.

[1030] Image data processing by the server

[1031] The server receives the image data sent from the refrigerator and uses an AI model (e.g., a model trained with TensorFlow or PyTorch) to recognize the type of food. The recognized food information is stored in a database. The server also estimates the expiration date of each food item and records this information in the database.

[1032] Best before date management

[1033] The server periodically checks the expiration dates of ingredients stored in the database. It lists ingredients whose expiration dates are approaching and prepares to notify users based on that information. The expiration date management algorithm estimates expiration dates based on, for example, the type of ingredient and the date of purchase.

[1034] Notification means

[1035] The server sends information about food items approaching their expiration date to a device (e.g., a smartphone or tablet). The device then sends a push notification or email notification to the user based on the received information.

[1036] Recipe generation and suggestions

[1037] The server generates recipes based on ingredients that are approaching their expiration date. The recipe generation uses a database built in the past and information entered by the user (e.g., preferences and allergies). The generated recipes are sent to the terminal and provided to the user as multiple options.

[1038] Personalization based on user input

[1039] Users input their eating habits, preferences, and allergies through an application or web interface, and the server uses this information to generate and suggest individually optimized recipes to the user.

[1040] Specific examples of operation

[1041] When a user places a new ingredient (e.g., cabbage) in the refrigerator, the camera takes a picture of it and sends it to the server. The server analyzes the image to recognize the cabbage and records its information and expiration date in a database. When the expiration date approaches, the server sends a notification to the device stating, "The cabbage will expire in two days." The device receives this notification and notifies the user as a push notification. The server then generates recipes using cabbage (e.g., cabbage rolls, cabbage salad) and sends them to the device. The device presents these recipes to the user, and if the user selects a cabbage roll recipe, it displays detailed instructions and the necessary ingredients.

[1042] Example prompts for generative AI models

[1043] "I'm storing some cabbage and it will expire in two days. Can you suggest some recipes using cabbage? Please take into consideration recipes that users have liked in the past."

[1044] In this way, the system of the present invention can effectively manage ingredients and support the user's eating habits through ingredient recognition, expiration date management, user notifications, recipe suggestions, and personalization based on user-entered information.

[1045] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1046] Step 1:

[1047] The camera periodically takes high-resolution images of the inside of the refrigerator. The input is an image of the food inside the refrigerator, and the output is the captured image data. The captured image data is sent to a server via a wireless or wired network. Specifically, the camera automatically releases the shutter and transmits the resulting image as digital data.

[1048] Step 2:

[1049] The server processes the image data received from the camera. The input is the image data of the food sent from the camera, and the output is the recognized food information (e.g., food name, quantity, etc.). Specifically, the server analyzes the image using an AI model (e.g., TensorFlow or PyTorch) and extracts the identified ingredient information. This analysis uses image processing algorithms and machine learning models.

[1050] Step 3:

[1051] The server estimates the expiration date based on the recognized food ingredient information. The input is the recognized food information, and the output is the estimated expiration date for each food item. Specifically, the algorithm calculates the expiration date based on the type of food ingredient, storage conditions, and past data. As a result, the expiration date information is recorded in a database.

[1052] Step 4:

[1053] The server periodically checks the database to identify ingredients that are approaching their expiration date. The input is all food information in the database, and the output is a list of ingredients that are approaching their expiration date. Specifically, a scheduling program traverses the database and lists ingredients that are within three days of their expiration date.

[1054] Step 5:

[1055] The server sends information about ingredients that are approaching their expiration date to the device. The input is a list of ingredients that are approaching their expiration date, and the output is a notification message to be sent to the user. Specifically, the server creates a push notification or email notification and sends it to the device. This notification contains a specific message, such as "The cabbage will expire in two days."

[1056] Step 6:

[1057] The device displays the notification message received from the server to the user. The input is the notification message sent from the server, and the output is the notification information displayed to the user. Specifically, the message is displayed on the device screen as a push notification or email notification.

[1058] Step 7:

[1059] The server generates recipes based on ingredients that are approaching their expiration date. The input is information about ingredients that are approaching their expiration date, and the output is a list of generated recipes. Specifically, the server generates multiple recipes based on a database of past recipes and user preferences. This process uses AI models and algorithms.

[1060] Step 8:

[1061] The server sends the generated recipes to the terminal. The input is a list of generated recipes, and the output is recipe information provided to the user. In concrete terms, the server sends recipe information to the terminal, and the terminal receives it.

[1062] Step 9:

[1063] The device presents the received recipe information to the user. The input is the recipe information sent from the server, and the output is a list of recipes displayed to the user. Specifically, the device displays multiple recipe options on the screen and displays detailed instructions and required ingredients for the recipe selected by the user.

[1064] Step 10:

[1065] A user provides input information, such as preferences and allergies, through an application or web interface. The input is the information entered by the user, and the output is the personalized information sent to the server. Specifically, the user enters information through a digital interface, and the information is sent to the server.

[1066] This series of processes realizes an overall flow from food recognition to expiration date management and recipe suggestions, effectively supporting the user's food management and eating habits.

[1067] (Application example 1)

[1068] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1069] Conventional refrigerator food management systems were limited to managing food ingredients within the home, and were unable to adequately manage ingredients or suggest recipes based on ingredients approaching their expiration date when purchasing at a physical store (such as a supermarket). Furthermore, it was not possible to track the history of ingredients purchased by the user and manage expiration dates or suggest recipes based on that information. This made food management cumbersome and led to the risk of food waste.

[1070] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1071] In this invention, the server includes a camera means for monitoring the status inside the refrigerator, a means for processing image data captured by the camera means and recognizing the type of ingredients from the image data, a means for managing the expiration dates of the recognized ingredients, a means for notifying the user of ingredients approaching their expiration dates, a means for generating recipes corresponding to the ingredients and suggesting them to the user, a means for making personalized suggestions based on information input by the user, and a means for managing ingredient purchase histories and presenting ingredients approaching their expiration dates in purchasing activities at physical stores. This enables ingredient expiration date management and recipe suggestions not only at home but also in purchasing activities at physical stores, enabling efficient ingredient management and reduction of food waste.

[1072] The "camera means" is a device for taking pictures of the inside of the refrigerator and acquiring the image data.

[1073] "Means for processing image data and recognizing the type of food ingredient from the image data" refers to technology for analyzing image data obtained from a camera means and identifying the type of food ingredient using an AI model, etc.

[1074] The "means for managing the expiration dates of recognized ingredients" is a system for storing the expiration dates of recognized ingredients in a database and periodically checking the expiration dates.

[1075] The "means for notifying users about food ingredients that are approaching their expiration date" is a system for notifying users about food ingredients that are approaching their expiration date, and is a mechanism for sending notifications to smartphones or tablets.

[1076] "Means for generating recipes corresponding to ingredients and suggesting them to users" refers to technology for generating appropriate recipes based on the types of ingredients and expiration dates and providing them to users.

[1077] "Means for making personalized suggestions based on user input" refers to a system that takes into account the user's preferences, eating habits, allergy information, etc., and suggests individually optimized recipes based on that information.

[1078] "Means for managing food purchase history and presenting food items approaching their expiration date during purchasing activities at physical stores" is a technology that stores the history of food items purchased at physical stores in a database and presents food items approaching their expiration date to the user.

[1079] The "means for periodically checking expiration dates" is a system that periodically checks the expiration dates of ingredients stored in a database.

[1080] The "means for generating a recipe including ingredients to be used based on a notification" is a technology that automatically generates a recipe including ingredients to be used based on ingredients whose expiration date is approaching.

[1081] The "terminal means for providing the generated recipe to the user" is a device for displaying the generated recipe to the user, such as a smartphone or tablet.

[1082] This system efficiently manages ingredients in a refrigerator and suggests suitable recipes, and also enables ingredient management and recipe suggestions for purchases in physical stores. This system consists of three main components: a camera device in the refrigerator, a server, and a user terminal.

[1083] System Components

[1084] 1. Camera Means

[1085] A camera installed inside the refrigerator periodically takes pictures of the interior. The high-resolution camera captures clear images of each ingredient. The captured image data is sent to a server.

[1086] 2. Image data processing by the server

[1087] The server receives the image data sent from the refrigerator and uses the generative AI model to recognize the type of food. The recognition results are recorded in a database. The server also estimates the expiration date, and this information is also stored in the database.

[1088] 3. Best before date management

[1089] The server periodically checks the expiration dates of ingredients stored in the database and lists ingredients that are approaching their expiration date, and prepares to notify the user of this information.

[1090] 4. Means of notification

[1091] The server sends notifications about food items approaching their expiration date to the user's device, such as a smartphone or tablet, via push notifications or email.

[1092] 5. Recipe generation and suggestions

[1093] The server generates recipes based on ingredients that are approaching their expiration date. The recipes are generated using a database built in the past and user input information (preferences, allergies, etc.). The generated recipes are provided to the user.

[1094] 6. Personalization with user input

[1095] Users input their eating habits, preferences, allergy information, etc. through the application, and the server uses this information to suggest individually optimized recipes.

[1096] 7. Managing food purchase history

[1097] The system also manages the history of food purchases at physical stores, and uses the purchase history database to suggest recipes based on ingredients that are close to their expiration date.

[1098] Hardware and software used

[1099] Hardware

[1100] Camera Method: High resolution camera installed inside the refrigerator.

[1101] User devices: smartphones, tablets.

[1102] Server: A cloud server such as AWS or GCP.

[1103] software

[1104] Image processing: Python, OpenCV, TensorFlow.

[1105] Server side: Django framework.

[1106] Database Management: Django ORM.

[1107] Communication method: Communication using REST API.

[1108] Specific examples of use

[1109] For example, if a user places "tomatoes" and "lettuce" in the refrigerator, the camera takes a picture of them and the server recognizes the type of food. The recognized food is stored in a database, and the user is notified when the expiration date approaches. At the same time, a recipe for "tomato and lettuce salad" is suggested. If the user purchases "chicken" at a physical store, the purchase history is stored in the database and used to suggest recipes for the next time.

[1110] Prompt Sentence Examples

[1111] "Please tell me some recipes that use the tomatoes, lettuce, and chicken I have in my fridge. The user prefers low-calorie recipes, and is allergic to nuts."

[1112] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1113] Step 1:

[1114] A camera means in the refrigerator periodically takes pictures of ingredients. The captured image data is sent to a server. The input is the image of the ingredients in the refrigerator, and the output is the image data sent to the server. The camera takes pictures when the camera button is pressed, or automatically at set intervals.

[1115] Step 2:

[1116] The server analyzes the received image data using a generative AI model to recognize the type of ingredient. The input is the image data sent in step 1, and the output is the recognized ingredient information. The analysis process is performed using Python and TensorFlow.

[1117] Step 3:

[1118] The server stores the information of the recognized ingredients in a database. The database records the type of ingredient, quantity, purchase date, and expiration date. The input is the ingredient information recognized in step 2, and the output is an organized ingredient database. It is saved to the database using Django ORM.

[1119] Step 4:

[1120] The server periodically checks the database and lists ingredients that are approaching their expiration date. The input is the ingredient information in the database, and the output is a list of ingredients that are approaching their expiration date. The checking process is performed by a Python script.

[1121] Step 5:

[1122] The server sends notifications to the user's device about ingredients that are approaching their expiration date. The input is the ingredient information listed in step 4, and the output is a notification sent to the user's smartphone or tablet. Notifications are sent via push notifications or email.

[1123] Step 6:

[1124] The server generates recipes based on ingredients that are approaching their expiration date. Multiple recipes are created using the generative AI model and the user's eating habits, preferences, and allergy information. The input is the ingredient information from step 4 and the user's personal information, and the output is the generated recipe. Prompt statements can be used to generate recipes.

[1125] Step 7:

[1126] The server provides the generated recipe to the user's device. The input is the recipe generated in step 6, and the output is the recipe displayed on the user's smartphone or tablet through the application interface.

[1127] Step 8:

[1128] When a user buys ingredients in a physical store, the purchase history is stored in a database. The input is the user's purchase information, and the output is an updated ingredient purchase history database. The database is managed using Django ORM.

[1129] Step 9:

[1130] The server uses the purchase history database to create a list of food items purchased in physical stores that are nearing their expiration date, and presents this to the user. The input is the updated purchase history database, and the output is a new notification sent to the user's device. The notification is displayed on a smartphone or tablet.

[1131] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1132] This invention is a system that efficiently manages ingredients in a refrigerator and suggests appropriate recipes, and by combining it with an emotion engine that recognizes the user's emotions, it provides more personalized dietary support. This system is composed of a camera for monitoring the status of the refrigerator, a server that processes image data, a terminal that notifies the user and suggests recipes, and an emotion engine that recognizes the user's emotions. Each component and its operation are described in detail below.

[1133] System configuration

[1134] 1. Camera Means

[1135] A camera installed inside the refrigerator periodically takes pictures of the inside of the refrigerator. The camera has high resolution and can clearly capture each food item inside the refrigerator. The captured image data is sent to a server.

[1136] 2. Image data processing by the server

[1137] The server receives the image data sent from the refrigerator and uses an AI model to identify the type of food. The identified food is then stored in a database. The server also estimates the expiration date for each food item and records this information in the database.

[1138] 3. Best before date management

[1139] The server periodically checks the expiration dates of ingredients stored in the database, lists ingredients that are approaching their expiration date, and prepares to notify the user of the listed ingredients.

[1140] 4. Means of notification

[1141] The server sends information about food items approaching their expiration date to the user's device, which can be a smartphone or tablet, and can send push notifications or email notifications to the user.

[1142] 5. Recipe generation and suggestions

[1143] The server generates recipes based on ingredients that are approaching their expiration date. The recipe generation uses a database built in the past and information entered by the user (preferences, allergies, etc.). The generated recipes are presented to the user as multiple options.

[1144] 6. Personalization with user input

[1145] Users input their eating habits, preferences, allergy information, etc. through an application or web interface, and the server uses this information to suggest individually optimized recipes for the user.

[1146] 7. Emotion Engine

[1147] The emotion engine recognizes emotions from the user's voice, facial expressions, text input, etc. The user's emotion information recognized by the emotion engine is also stored in the database.

[1148] Example of program processing

[1149] 1. Recognizing ingredients

[1150] The user places a new ingredient (e.g., cabbage) in the refrigerator.

[1151] The server analyzes the image sent from the camera inside the refrigerator and recognizes the cabbage. The recognition result and the expiration date of the cabbage are recorded in a database.

[1152] 2. Best before date management

[1153] The server periodically checks the database and finds that the cabbage's expiration date is two days away.

[1154] The server sends a notification to the terminal such as "The cabbage's expiration date is in two days."

[1155] 3. Recipe suggestions

[1156] The server generates recipes that use cabbage (e.g., cabbage rolls, cabbage salad).

[1157] The device suggests these recipes to the user.

[1158] When a user selects a recipe for cabbage rolls, detailed instructions and ingredients are displayed on the device.

[1159] 4. Personalization with user input

[1160] Users enter their eating habits, preferences, allergy information, etc. through the application.

[1161] The server stores this information in a database and reflects it in future recipe suggestions.

[1162] 5. Emotion recognition and suggestion optimization

[1163] The emotion engine recognizes emotions from the user's voice and facial expressions and sends this information to the server.

[1164] The server then tailors recipes and notifications to suit the user based on their emotional information, for example, suggesting easy-to-make recipes if the user is tired.

[1165] In this way, the system of the present invention provides optimal dietary support to users by combining ingredient recognition, expiration date management, notifications, recipe suggestions, personalization based on user-entered information, and emotion recognition using an emotion engine.

[1166] The processing flow will be explained below.

[1167] Step 1:

[1168] The user places a new ingredient (e.g., cabbage) in the refrigerator.

[1169] The refrigerator periodically takes pictures of the interior using a built-in camera.

[1170] Step 2:

[1171] Image data captured by the camera means is transmitted to the server.

[1172] The server receives the image data.

[1173] Step 3:

[1174] The server analyzes the image data and uses an AI model to recognize the type of food ingredient (e.g., cabbage).

[1175] The recognition results are recorded in a database.

[1176] Step 4:

[1177] The server automatically estimates the expiration date of the recognized ingredients and stores it in a database.

[1178] The server periodically checks the database to identify ingredients that are approaching their expiration date.

[1179] Step 5:

[1180] The server prepares a notification to the user for listed ingredients that are close to their expiration date.

[1181] A notification such as "The cabbage's expiration date is in two days" is sent to the device.

[1182] Step 6:

[1183] The device displays the push notification to the user.

[1184] The user checks the notification and recognizes ingredients (e.g., cabbage) that are nearing their expiration date.

[1185] Step 7:

[1186] The server requests data to check the user's emotions with an emotion engine.

[1187] The device collects the user's voice and facial expression data and sends it to the server.

[1188] Step 8:

[1189] The emotion engine analyzes the user's emotions (e.g., if the user is tired).

[1190] The server records the user's emotional information recognized by the emotion engine in a database and uses it to adjust the recipe.

[1191] Step 9:

[1192] The server executes a recipe algorithm based on the user's ingredient data and emotion data.

[1193] For example, if the cabbage is nearing its expiration date and the user is tired, the app will suggest an easy recipe (e.g., easy stir-fried cabbage).

[1194] Step 10:

[1195] The server generates recipe suggestions and sends them to the device.

[1196] The device displays recipe suggestions to the user.

[1197] When a user selects a recipe, detailed instructions and required ingredients are displayed.

[1198] Step 11:

[1199] Users enter their eating habits, preferences, allergy information, etc. through the application.

[1200] The server stores this information in a database and reflects it in future recipe suggestions.

[1201] In this way, the system provides optimal dietary support to users by recognizing ingredients, managing expiration dates, providing notifications, suggesting recipes, personalizing with user-entered information, and even recognizing emotions using an emotion engine.

[1202] Example 2

[1203] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1204] In modern households, managing food in the refrigerator is complicated, and food that has passed its expiration date is often wasted. Furthermore, when it comes to effectively utilizing ingredients and suggesting recipes, there is a lack of personalization that takes into account individual users' preferences and allergy information. Furthermore, there is no system that can make appropriate suggestions based on the user's emotional state.

[1205] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a camera means for monitoring the state inside the refrigerator, a means for processing image data captured by the camera means and recognizing the types of ingredients from the image data, a means for estimating and managing the expiration dates of the recognized ingredients, a means for notifying the user of ingredients whose expiration dates are approaching, a means for generating recipes based on the ingredients and suggesting them to the user, a means for making personalized suggestions based on information input by the user, and a means for recognizing the user's emotions from their voice, facial expressions, text input, etc., and adjusting the suggestions. This reduces ingredient waste, makes recipe suggestions that meet the user's individual preferences, and enables optimal suggestions based on the user's emotional state.

[1206] The "camera means" refers to a camera device installed to monitor the state inside the refrigerator, and has the function of taking pictures of ingredients.

[1207] "Image data" refers to visual information of the inside of a refrigerator photographed by a camera means, which is recorded and stored in digital format.

[1208] The "means for recognizing the type of food ingredient" is a function that analyzes the captured image data and identifies the name and attributes of the food ingredient using AI or machine learning models.

[1209] The "means for estimating and managing expiration dates" has the function of predicting and managing expiration dates for recognized food ingredients based on the purchase date and general storage period.

[1210] The "notification means" has a function for transmitting information about ingredients approaching their expiration date, recipe suggestions, etc. to the user's terminal.

[1211] The "means for generating recipes" has the function of automatically creating cooking instructions and steps based on the recognized ingredient information.

[1212] The "means for suggesting to the user" has a function for displaying the generated recipe and notification content to the user.

[1213] The "means for making personalized suggestions" has the function of generating recipes and notification content that take into account individual preferences and allergies based on the user's input information and past data.

[1214] The "means for recognizing emotions and adjusting suggested content" has the function of analyzing the user's current emotional state from their voice, facial expressions, text input, etc., and providing recipes and notification content accordingly.

[1215] This invention is a system that efficiently manages ingredients in the refrigerator and suggests suitable recipes, and by combining it with an emotion engine that recognizes the user's emotions, it provides more personalized dietary support. This system operates based on the following components.

[1216] 1. Camera Means

[1217] The camera installed inside the refrigerator takes high-resolution images of the interior of the refrigerator. The camera periodically (for example, every hour) takes a picture of the overall situation inside the refrigerator and sends this image data to a server. It is desirable to use a network camera with a resolution of 1080p or higher.

[1218] 2. Image data processing by the server

[1219] The server receives the image data sent from the camera and performs image analysis. This image analysis uses AI models such as TensorFlow and PyTorch. This automatically recognizes the types of ingredients in the refrigerator and stores the results in a database. Specifically, it uses the YOLOv3 model to perform high-speed, high-precision object recognition.

[1220] 3. Estimation and management of expiration dates

[1221] The server estimates the expiration date for the recognized ingredients. For example, if it recognizes cabbage, it calculates the expiration date based on the typical storage period for cabbage (about 5 days in the refrigerator) and records this information in the database. The database is a relational database such as MySQL or PostgreSQL.

[1222] 4. Sending notifications

[1223] The server periodically checks the information in the database and sends notifications to the user's device about ingredients that are approaching their expiration date. This notification is done via push notification or email. For example, if the expiration date of cabbage is approaching in two days, a message such as "The expiration date of the cabbage is in two days" is generated and sent to the user's smartphone.

[1224] 5. Recipe generation and suggestions

[1225] The server uses a database of past recipes and information entered by the user (such as preferences and allergies) to generate recipes based on ingredients approaching their expiration date. The generated recipes are presented to the user as multiple options. For example, recipes such as "stuffed cabbage" and "cabbage salad" using cabbage may be generated.

[1226] 6. Personalization with user input

[1227] Users input their eating habits, preferences, allergies, and other information through the application or web interface. The server stores this information in a database and uses it to suggest future recipes, allowing users to receive recipes tailored to their individual needs.

[1228] 7. Emotion Recognition with Emotion Engine

[1229] The emotion engine recognizes emotions from the user's voice, facial expressions, and text input. This emotion information is also sent to the server and stored in a database. The server uses this information to tailor recipes and notification content to suit the user. For example, if the emotion engine recognizes that the user is tired, it will prioritize suggesting easy recipes.

[1230] Examples and prompts

[1231] Example: When a user puts a new cabbage in the refrigerator, the camera recognizes this and sends data to the server to calculate the expiration date of the cabbage. When the expiration date approaches, the server sends a notification and generates several recipes using the cabbage and displays them on the device.

[1232] Example prompt: "Recognize the new ingredients in your refrigerator and suggest a recipe using them."

[1233] In this way, the system of the present invention comprehensively supports everything from managing ingredients in the refrigerator to suggesting recipes for each user, and even suggesting the best recipes based on the user's emotional state. Introducing this system will reduce food waste and lead to a richer and healthier diet for users.

[1234] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1235] Step 1:

[1236] Taking pictures with a camera

[1237] Input: Condition inside the refrigerator

[1238] Process: The camera takes high-resolution images of the inside of the refrigerator periodically (e.g., every hour). Use a network camera with a resolution of 1080p or higher.

[1239] Output: Image data of the inside of the refrigerator

[1240] Specific operation: The location and condition of all food items in the refrigerator are clearly photographed and data is generated.

[1241] Step 2:

[1242] Sending image data

[1243] Input: Image data of the inside of the refrigerator

[1244] Processing: Image data generated by the camera is sent to the server in real time via Wi-Fi.

[1245] Output: Image data sent to the server

[1246] Specific operation: The image file taken by the camera is compressed and a protocol (e.g. HTTP) is executed to send it to the server.

[1247] Step 3:

[1248] Receiving and analyzing image data

[1249] Input: Image data sent from the camera

[1250] Processing: The server receives the image data and uses an AI model (e.g., TensorFlow or PyTorch) to recognize the type of food. Specifically, the YOLOv3 model is used.

[1251] Output: Recognized food type and location information

[1252] Specific operation: The server analyzes the received image data, identifies the name and location of each ingredient, and records it in a database.

[1253] Step 4:

[1254] Estimating expiration dates and storing

[1255] Input: Recognized food type and location information

[1256] Processing: The server estimates the shelf life of each ingredient based on its type. For example, cabbage can be stored in the refrigerator for 5 days.

[1257] Output: Best before date data for each ingredient

[1258] What it does: Calculates the expiration date for each ingredient based on the purchase date and shelf life, and records that information in a database.

[1259] Step 5:

[1260] Regular check of expiration dates

[1261] Input: Best before date data stored in the database

[1262] Processing: The server periodically (e.g., every night at midnight) scans the database and lists ingredients that are nearing their expiration date.

[1263] Output: A list of ingredients that are nearing their expiration date

[1264] What it does: Query the database to find ingredients with a shelf life of less than 3 days.

[1265] Step 6:

[1266] Creating a notification message and preparing it for sending

[1267] Input: A list of ingredients that are nearing their expiration date

[1268] Processing: The server generates a notification message for the listed ingredients, for example, "The cabbage will expire in 2 days."

[1269] Output: The generated notification message

[1270] Specific operation: Automatically generate notification content using a message template and prepare to send it to the user's device.

[1271] Step 7:

[1272] Sending notifications

[1273] Input: The generated notification message

[1274] Process: The server sends a notification message to the user's device via push notification or email notification.

[1275] Output: Notification message received by the user

[1276] Specific operation: The server sends a message to the user's device via push notification or email system.

[1277] Step 8:

[1278] Receiving notifications

[1279] Input: Notification message sent by the server

[1280] Processing: The device receives the notification and displays it on the user's screen as a popup or email.

[1281] Output: The notification message displayed to the user

[1282] Specific behavior: The device's notification system receives the message from the server and immediately displays it to the user.

[1283] Step 9:

[1284] Recipe Generation

[1285] Input: List of ingredients approaching expiration date and user database

[1286] Processing: The server generates recipes based on ingredients that are nearing their expiration date, using a database of past data and user preferences and allergy information.

[1287] Output: A list of generated recipes

[1288] Specific operation: The system generates multiple appropriate recipes based on the ingredients data and the user's profile. For example, it suggests "cabbage rolls" and "cabbage salad" using cabbage.

[1289] Step 10:

[1290] Recipe Suggestions

[1291] Input: A list of generated recipes

[1292] What happens: The device presents the suggested recipes to the user. When the user selects a specific recipe, detailed instructions and required ingredients are displayed.

[1293] Output: Recipe details

[1294] Specific behavior: Displays a list of recipes on the device screen and provides detailed information about the recipe selected by the user.

[1295] Step 11:

[1296] Entering user information

[1297] Input: User-provided dietary habits, preferences, and allergy information

[1298] Processing: The user enters information through an application or web interface and sends it to the server.

[1299] Output: User information stored in the database

[1300] What happens: The user uses the interface to enter their information and presses the submit button. The server receives this and records it in the database.

[1301] Step 12:

[1302] Storage and Use of User Information

[1303] Input: User-submitted dietary habits, preferences, and allergy information

[1304] Processing: The server stores this information in a database and uses it to suggest recipes from the next time onwards.

[1305] Output: Updated user profile

[1306] Specific operation: The server uses the user's input information to personalize future recipe suggestions and notifications.

[1307] Step 13:

[1308] emotion recognition

[1309] Input: User voice, facial expressions, and text input

[1310] Processing: The emotion engine grasps the user's emotion and sends it to the server.

[1311] Output: Recognized user emotion information

[1312] Specific operation: The emotion engine analyzes the user's input data, grasps the user's emotional state, and sends that information to the server.

[1313] Step 14:

[1314] Use of emotional information

[1315] Input: Recognized user emotion information

[1316] Processing: The server records the emotion information in a database and adjusts the suggestions to the user based on this information.

[1317] Output: Adjusted proposal

[1318] Specific operation: The server refers to the emotional information and provides the user with the most appropriate content according to their emotions, such as suggesting simple recipes if the user is tired.

[1319] (Application example 2)

[1320] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1321] In recent years, refrigerator food management and food waste issues have been attracting attention, but traditional manual and simple digital management methods have their limitations. Furthermore, managing ingredients in brick-and-mortar stores and effectively suggesting recipes to customers requires a great deal of effort. Furthermore, while there is a demand for services that take customer emotions into account, no effective system exists to achieve this. To solve these issues, it is necessary to streamline refrigerator food management, provide appropriate information to store staff and customers, and provide personalized services that take customer emotions into account.

[1322] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1323] In this invention, the server includes means for processing image data captured by a camera means and recognizing the type of ingredient from the image data, means for managing the expiration dates of the recognized ingredients, means for notifying the user of ingredients approaching their expiration dates, means for generating recipes corresponding to the ingredients and suggesting them to the user, means for making personalized suggestions based on information input by the user, means for efficiently managing ingredients in a refrigerator and monitoring products stored in refrigerators in the store, means for notifying store staff of products approaching their expiration dates, and means for recognizing customer emotions and suggesting recipes or promoting purchases based on the emotions. This enables efficient management of ingredients in a refrigerator, appropriate management of ingredients in a physical store, personalized recipe suggestions to customers, and service provision based on customer emotions.

[1324] The "camera means" is a photographing device installed to monitor the state inside the refrigerator, and periodically takes pictures of the food ingredients.

[1325] "Image data" refers to video information of ingredients photographed using a camera, and is digital data that is analyzed by the server.

[1326] A "means for recognizing" is a method or device capable of processing image data and identifying and classifying objects in the image.

[1327] The "means for managing expiration dates" refers to a method or device for recording the expiration dates of recognized food ingredients and for notifying users when the expiration date is approaching.

[1328] "Notification means" refers to a method or device used to notify a user or store staff of specific information.

[1329] The "means for generating and suggesting recipes" refers to a method or device that creates a cooking method based on recognized ingredients and provides it to the user.

[1330] The "means for making personalized suggestions" refers to a method or device that makes individually optimized suggestions based on information input by the user, past data, and the like.

[1331] "Means for monitoring products" refers to a method or device for monitoring the condition of products stored in refrigerators in a store and understanding product trends and inventory status.

[1332] A "means for recognizing customer emotions" is a method or device that determines a customer's emotions from voice, facial expressions, text input, etc., and utilizes that information.

[1333] "Means for promoting purchases" are methods and devices for making individually optimized product suggestions based on customer emotions and behavioral data, thereby increasing purchasing motivation.

[1334] The present invention provides a system for improving the efficiency of food ingredient management in a refrigerator and product management in a physical store, and for suggesting personalized recipes to customers and promoting purchases. The following describes in detail an embodiment of the present invention.

[1335] First, regarding the hardware configuration of the entire system, a high-resolution camera is installed inside the refrigerator. This camera periodically takes pictures of the ingredients and products inside the store refrigerator and sends the image data to a server. A configuration using a virtual server on AWS EC2 is suitable for this server.

[1336] The server receives image data sent from the refrigerator and uses an image recognition AI model using TensorFlow to identify ingredients and product types. The identified ingredients are then stored in a database (AWS RDS).

[1337] The server then periodically checks the database and lists ingredients and products that are approaching their expiration date. Store staff are notified of these ingredients and products via push notifications sent via smartphones or tablets. The server also generates recipes based on the products that are approaching their expiration date.

[1338] Recipe generation utilizes previously constructed databases and information entered by users (such as preferences and allergies). The generated recipes are provided to customers via a sales promotion touch panel display and a customer application.

[1339] Furthermore, an emotion engine will be used to recognize emotions from the customer's voice and facial expressions. This emotion information will also be sent to the server, and personalized recipe suggestions and purchase promotions will be made based on the emotion. Microsoft Azure Cognitive Services could be used for the emotion engine.

[1340] The system includes the following specific processing steps:

[1341] 1. The camera means takes an image of the food in the refrigerator.

[1342] 2. The captured image data is sent to the server.

[1343] 3. The server analyzes the image using a TensorFlow model and recognizes the ingredients.

[1344] 4. The recognition results and expiration date information are stored in the database.

[1345] 5. The server periodically checks the database and notifies store staff of any food items that are nearing their expiration date.

[1346] 6. The server generates recipes for customers using ingredients that are close to their expiration date and provides them via a touch panel display or app.

[1347] 7. The emotion recognition engine recognizes the customer's emotions and sends the emotional information to the server.

[1348] 8. Based on the emotional information, the server will suggest recipes and promote purchases according to the emotions.

[1349] For example, consider the following prompt:

[1350] "A new cabbage is placed in the refrigerator. Use an image recognition AI model to recognize the cabbage and add it to the expiration date management system. When the cabbage's expiration date approaches, notify the customer via the touch panel display with a recommended recipe. Also, recognize the customer's emotions based on their facial expressions and suggest recipes that match those emotions."

[1351] This will enable efficient management of food in the refrigerator, proper notification to store staff, and the provision of services that respond to customer emotions.

[1352] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1353] Step 1:

[1354] A camera means within the refrigerator takes images of the food items.

[1355] Input: Refrigerator status

[1356] Data processing: image data capture

[1357] Output: High-resolution food image data

[1358] Step 2:

[1359] The captured image data is sent to a server.

[1360] Input: High-resolution food image data

[1361] Data processing: Sending image data

[1362] Output: Image data received by the server

[1363] Step 3:

[1364] The server uses a TensorFlow model to analyze the image data and recognize the type of ingredient.

[1365] Input: Received food image data

[1366] Data processing: image recognition, food classification

[1367] Output: Recognized ingredients (e.g., tomato, cabbage)

[1368] Step 4:

[1369] The recognized ingredient information and expiration date information are stored in a database on the server.

[1370] Input: Recognized ingredient information

[1371] Data processing: Recording in database, estimating expiration date

[1372] Output: Stored ingredients and expiration date data

[1373] Step 5:

[1374] The server periodically checks the database and notifies store staff of food items that are nearing their expiration date.

[1375] Input: Stored ingredients and expiration date data

[1376] Data processing: Expiration date check, notification information creation

[1377] Output: Notification information (e.g. "The expiration date for the tomatoes is in 2 days")

[1378] Step 6:

[1379] The server generates recipes using ingredients that are close to their expiration date and transmits them to a terminal for serving to the customer.

[1380] Input: Information about ingredients that are close to expiry date, as well as user preferences and allergy information

[1381] Data processing: Recipe generation, recipe data creation

[1382] Output: Recipe suggestions (e.g., tomato salad, tomato pasta)

[1383] Step 7:

[1384] The generated recipe information is provided to the customer via a touch panel display or a customer application.

[1385] Input: Recipe to suggest

[1386] Data processing: Preparing to display recipe information

[1387] Output: Display of recipe (e.g., recipe displayed on a touch panel or smartphone)

[1388] Step 8:

[1389] The emotion engine recognizes emotions from the customer's voice and facial expressions and sends that information to the server.

[1390] Input: Customer voice and facial expression data

[1391] Data processing: Emotion recognition, creating data of the recognition results

[1392] Output: Recognized emotion information (e.g., "The customer is tired")

[1393] Step 9:

[1394] Based on the emotional information, the server makes personalized recipe suggestions or purchase promotions according to the emotions.

[1395] Input: Recognized emotion information

[1396] Data processing: Recipe suggestions readjustment, purchase promotion information creation

[1397] Output: Personalized recipe suggestions and promotional notifications (e.g., "Easy recipes perfect for a tiring day")

[1398] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1399] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1400] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1401] [Fourth embodiment]

[1402] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1403] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1404] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1405] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1406] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1407] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1408] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1409] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1410] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1411] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1412] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1413] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1414] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1415] This invention is a system that efficiently manages ingredients in a refrigerator and suggests suitable recipes. This system consists of a camera for monitoring the status of the refrigerator, a server for processing image data, and a terminal that notifies the user and suggests recipes. Each component and its operation are described in detail below.

[1416] System configuration

[1417] 1. Camera Means

[1418] A camera installed inside the refrigerator periodically takes pictures of the inside of the refrigerator. The camera has high resolution and can clearly capture each food item inside the refrigerator. The captured image data is sent to a server.

[1419] 2. Image data processing by the server

[1420] The server receives the image data sent from the refrigerator and uses an AI model to identify the type of food. The identified food is then stored in a database. The server also estimates the expiration date for each food item and records this information in the database.

[1421] 3. Best before date management

[1422] The server periodically checks the expiration dates of ingredients stored in the database, lists ingredients that are approaching their expiration date, and prepares to notify the user of the listed ingredients.

[1423] 4. Means of notification

[1424] The server sends information about food items approaching their expiration date to the user's device, which can be a smartphone or tablet, and can send push notifications or email notifications to the user.

[1425] 5. Recipe generation and suggestions

[1426] The server generates recipes based on ingredients that are approaching their expiration date. The recipe generation uses a database built in the past and information entered by the user (preferences, allergies, etc.). The generated recipes are presented to the user as multiple options.

[1427] 6. Personalization with user input

[1428] Users input their eating habits, preferences, allergy information, etc. through an application or web interface, and the server uses this information to suggest individually optimized recipes for the user.

[1429] Example of program processing

[1430] 1. Recognizing ingredients

[1431] The user places a new ingredient (e.g., cabbage) in the refrigerator.

[1432] The server analyzes the image sent from the camera inside the refrigerator and recognizes the cabbage. The recognition result and the expiration date of the cabbage are recorded in a database.

[1433] 2. Best before date management

[1434] The server periodically checks the database and finds that the cabbage's expiration date is two days away.

[1435] The server sends a notification to the terminal such as "The cabbage's expiration date is in two days."

[1436] 3. Recipe suggestions

[1437] The server generates recipes that use cabbage (e.g., cabbage rolls, cabbage salad).

[1438] The device suggests these recipes to the user.

[1439] When a user selects a recipe for cabbage rolls, detailed instructions and ingredients are displayed on the device.

[1440] In this way, the system of the present invention efficiently manages ingredients and supports the user's eating habits through ingredient recognition, expiration date management, notifications, recipe suggestions, and personalization based on user-entered information.

[1441] The processing flow will be explained below.

[1442] Step 1:

[1443] The user places a new ingredient (e.g., cabbage) in the refrigerator.

[1444] The refrigerator periodically takes pictures of the interior using a built-in camera.

[1445] Step 2:

[1446] Image data captured by the camera means is transmitted to the server.

[1447] The server receives the image data.

[1448] Step 3:

[1449] The server processes the image data and uses an AI model to recognize the type of food ingredient (e.g., cabbage).

[1450] The recognition results are recorded in a database.

[1451] Step 4:

[1452] The server automatically estimates the expiration date of the recognized ingredients and stores it in a database.

[1453] The server periodically checks the database to identify ingredients that are approaching their expiration date.

[1454] Step 5:

[1455] The server prepares a notification to the user for ingredients that are nearing their expiration date.

[1456] A notification such as "The cabbage's expiration date is in two days" is sent to the device.

[1457] Step 6:

[1458] The device displays a push or email notification to the user.

[1459] The user checks the notification and becomes aware of the presence of food items (e.g., cabbage) that are nearing their expiration date.

[1460] Step 7:

[1461] The server runs an algorithm to suggest recipes using cabbage (e.g., cabbage rolls, cabbage salad).

[1462] A proposal list is created and sent to the terminal.

[1463] Step 8:

[1464] The terminal displays a list of suggestions to the user.

[1465] The user selects the cabbage rolls recipe from the list of suggestions.

[1466] Step 9:

[1467] The server sends detailed recipe information (steps and ingredients) for the cabbage rolls to the terminal.

[1468] The terminal displays the detailed information to the user.

[1469] Step 10:

[1470] Users enter their eating habits, preferences, allergy information, etc. through the application.

[1471] The server stores this input information in a database and reflects it in future recipe suggestions.

[1472] Through this series of processing steps, users can efficiently manage and use ingredients, and improve the quality of their diet by utilizing optimal recipes.

[1473] Example 1

[1474] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1475] There is a need for efficient management of food in refrigerators, reducing food waste, and providing users with appropriate and timely recipe suggestions. However, existing refrigerator management systems have issues with food recognition accuracy, expiration date management, user notifications, and recipe suggestions, making it difficult to perform multiple different functions in an integrated manner.

[1476] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1477] In this invention, the server includes an image acquisition means, a means for processing image data captured using the image acquisition means and recognizing the type of food from the image data, a means for managing the expiration dates of the recognized foods, a means for periodically checking the database to confirm the expiration dates of the foods and notifying the user when the expiration date is approaching, a means for generating cooking methods based on the notified foods with an approaching expiration date and providing them to the user, and a means for making personalized suggestions based on information input by the user. This allows multiple different functions to be executed in an integrated manner as a series of processes, making it possible to improve the efficiency of food management, reduce waste, and even suggest recipes optimized for the user.

[1478] The "image acquisition means" is a device such as a camera or sensor that takes a picture of the food in the refrigerator and acquires the image data.

[1479] "Means for recognizing food types" refers to algorithms or AI models that analyze acquired image data and identify the food in the image.

[1480] A "best-before date management tool" is software or algorithms that record the best-before dates of recognized food products in a database and periodically check those dates.

[1481] "Means of checking the database to verify expiration dates" refers to the process or method of periodically checking food information in the database to identify foods that are approaching their expiration date.

[1482] The "notification means" refers to a push notification, email notification, or other notification method for informing the user about food products that are approaching their expiration date.

[1483] "Method for generating cooking instructions" refers to an algorithm or AI model that automatically generates recipes and cooking instructions based on specific foods.

[1484] The "means for providing to the user" refers to a terminal or application interface for displaying the generated recipes and cooking methods to the user.

[1485] "Means for personalized suggestions" refers to customization features and algorithms that provide optimized recipes and cooking methods based on the user's eating habits and preferences.

[1486] This invention is a system that efficiently manages ingredients in a refrigerator and suggests suitable recipes. The system consists of an image acquisition unit for monitoring the status of the refrigerator, a server for processing image data, and a terminal that notifies the user and suggests recipes.

[1487] Image Acquisition Method

[1488] A camera is installed inside the refrigerator and periodically takes pictures of the inside of the refrigerator. The camera has high resolution and can clearly capture each ingredient. The captured image data is sent to a server via wireless or wired communication.

[1489] Image data processing by the server

[1490] The server receives the image data sent from the refrigerator and uses an AI model (e.g., a model trained with TensorFlow or PyTorch) to recognize the type of food. The recognized food information is stored in a database. The server also estimates the expiration date of each food item and records this information in the database.

[1491] Best before date management

[1492] The server periodically checks the expiration dates of ingredients stored in the database. It lists ingredients whose expiration dates are approaching and prepares to notify users based on that information. The expiration date management algorithm estimates expiration dates based on, for example, the type of ingredient and the date of purchase.

[1493] Notification means

[1494] The server sends information about food items approaching their expiration date to a device (e.g., a smartphone or tablet). The device then sends a push notification or email notification to the user based on the received information.

[1495] Recipe generation and suggestions

[1496] The server generates recipes based on ingredients that are approaching their expiration date. The recipe generation uses a database built in the past and information entered by the user (e.g., preferences and allergies). The generated recipes are sent to the terminal and provided to the user as multiple options.

[1497] Personalization based on user input

[1498] Users input their eating habits, preferences, and allergies through an application or web interface, and the server uses this information to generate and suggest individually optimized recipes to the user.

[1499] Specific examples of operation

[1500] When a user places a new ingredient (e.g., cabbage) in the refrigerator, the camera takes a picture of it and sends it to the server. The server analyzes the image to recognize the cabbage and records its information and expiration date in a database. When the expiration date approaches, the server sends a notification to the device stating, "The cabbage will expire in two days." The device receives this notification and notifies the user as a push notification. The server then generates recipes using cabbage (e.g., cabbage rolls, cabbage salad) and sends them to the device. The device presents these recipes to the user, and if the user selects a cabbage roll recipe, it displays detailed instructions and the necessary ingredients.

[1501] Example prompts for generative AI models

[1502] "I'm storing some cabbage and it will expire in two days. Can you suggest some recipes using cabbage? Please take into consideration recipes that users have liked in the past."

[1503] In this way, the system of the present invention can effectively manage ingredients and support the user's eating habits through ingredient recognition, expiration date management, user notifications, recipe suggestions, and personalization based on user-entered information.

[1504] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1505] Step 1:

[1506] The camera periodically takes high-resolution images of the inside of the refrigerator. The input is an image of the food inside the refrigerator, and the output is the captured image data. The captured image data is sent to a server via a wireless or wired network. Specifically, the camera automatically releases the shutter and transmits the resulting image as digital data.

[1507] Step 2:

[1508] The server processes the image data received from the camera. The input is the image data of the food sent from the camera, and the output is the recognized food information (e.g., food name, quantity, etc.). Specifically, the server analyzes the image using an AI model (e.g., TensorFlow or PyTorch) and extracts the identified ingredient information. This analysis uses image processing algorithms and machine learning models.

[1509] Step 3:

[1510] The server estimates the expiration date based on the recognized food ingredient information. The input is the recognized food information, and the output is the estimated expiration date for each food item. Specifically, the algorithm calculates the expiration date based on the type of food ingredient, storage conditions, and past data. As a result, the expiration date information is recorded in a database.

[1511] Step 4:

[1512] The server periodically checks the database to identify ingredients that are approaching their expiration date. The input is all food information in the database, and the output is a list of ingredients that are approaching their expiration date. Specifically, a scheduling program traverses the database and lists ingredients that are within three days of their expiration date.

[1513] Step 5:

[1514] The server sends information about ingredients that are approaching their expiration date to the device. The input is a list of ingredients that are approaching their expiration date, and the output is a notification message to be sent to the user. Specifically, the server creates a push notification or email notification and sends it to the device. This notification contains a specific message, such as "The cabbage will expire in two days."

[1515] Step 6:

[1516] The device displays the notification message received from the server to the user. The input is the notification message sent from the server, and the output is the notification information displayed to the user. Specifically, the message is displayed on the device screen as a push notification or email notification.

[1517] Step 7:

[1518] The server generates recipes based on ingredients that are approaching their expiration date. The input is information about ingredients that are approaching their expiration date, and the output is a list of generated recipes. Specifically, the server generates multiple recipes based on a database of past recipes and user preferences. This process uses AI models and algorithms.

[1519] Step 8:

[1520] The server sends the generated recipes to the terminal. The input is a list of generated recipes, and the output is recipe information provided to the user. In concrete terms, the server sends recipe information to the terminal, and the terminal receives it.

[1521] Step 9:

[1522] The device presents the received recipe information to the user. The input is the recipe information sent from the server, and the output is a list of recipes displayed to the user. Specifically, the device displays multiple recipe options on the screen and displays detailed instructions and required ingredients for the recipe selected by the user.

[1523] Step 10:

[1524] A user provides input information, such as preferences and allergies, through an application or web interface. The input is the information entered by the user, and the output is the personalized information sent to the server. Specifically, the user enters information through a digital interface, and the information is sent to the server.

[1525] This series of processes realizes an overall flow from food recognition to expiration date management and recipe suggestions, effectively supporting the user's food management and eating habits.

[1526] (Application example 1)

[1527] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1528] Conventional refrigerator food management systems were limited to managing food ingredients within the home, and were unable to adequately manage ingredients or suggest recipes based on ingredients approaching their expiration date when purchasing at a physical store (such as a supermarket). Furthermore, it was not possible to track the history of ingredients purchased by the user and manage expiration dates or suggest recipes based on that information. This made food management cumbersome and led to the risk of food waste.

[1529] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1530] In this invention, the server includes a camera means for monitoring the status inside the refrigerator, a means for processing image data captured by the camera means and recognizing the type of ingredients from the image data, a means for managing the expiration dates of the recognized ingredients, a means for notifying the user of ingredients approaching their expiration dates, a means for generating recipes corresponding to the ingredients and suggesting them to the user, a means for making personalized suggestions based on information input by the user, and a means for managing ingredient purchase histories and presenting ingredients approaching their expiration dates in purchasing activities at physical stores. This enables ingredient expiration date management and recipe suggestions not only at home but also in purchasing activities at physical stores, enabling efficient ingredient management and reduction of food waste.

[1531] The "camera means" is a device for taking pictures of the inside of the refrigerator and acquiring the image data.

[1532] "Means for processing image data and recognizing the type of food ingredient from the image data" refers to technology for analyzing image data obtained from a camera means and identifying the type of food ingredient using an AI model, etc.

[1533] The "means for managing the expiration dates of recognized ingredients" is a system for storing the expiration dates of recognized ingredients in a database and periodically checking the expiration dates.

[1534] The "means for notifying users about food ingredients that are approaching their expiration date" is a system for notifying users about food ingredients that are approaching their expiration date, and is a mechanism for sending notifications to smartphones or tablets.

[1535] "Means for generating recipes corresponding to ingredients and suggesting them to users" refers to technology for generating appropriate recipes based on the types of ingredients and expiration dates and providing them to users.

[1536] "Means for making personalized suggestions based on user input" refers to a system that takes into account the user's preferences, eating habits, allergy information, etc., and suggests individually optimized recipes based on that information.

[1537] "Means for managing food purchase history and presenting food items approaching their expiration date during purchasing activities at physical stores" is a technology that stores the history of food items purchased at physical stores in a database and presents food items approaching their expiration date to the user.

[1538] The "means for periodically checking expiration dates" is a system that periodically checks the expiration dates of ingredients stored in a database.

[1539] The "means for generating a recipe including ingredients to be used based on a notification" is a technology that automatically generates a recipe including ingredients to be used based on ingredients whose expiration date is approaching.

[1540] The "terminal means for providing the generated recipe to the user" is a device for displaying the generated recipe to the user, such as a smartphone or tablet.

[1541] This system efficiently manages ingredients in a refrigerator and suggests suitable recipes, and also enables ingredient management and recipe suggestions for purchases in physical stores. This system consists of three main components: a camera device in the refrigerator, a server, and a user terminal.

[1542] System Components

[1543] 1. Camera Means

[1544] A camera installed inside the refrigerator periodically takes pictures of the interior. The high-resolution camera captures clear images of each ingredient. The captured image data is sent to a server.

[1545] 2. Image data processing by the server

[1546] The server receives the image data sent from the refrigerator and uses the generative AI model to recognize the type of food. The recognition results are recorded in a database. The server also estimates the expiration date, and this information is also stored in the database.

[1547] 3. Best before date management

[1548] The server periodically checks the expiration dates of ingredients stored in the database and lists ingredients that are approaching their expiration date, and prepares to notify the user of this information.

[1549] 4. Means of notification

[1550] The server sends notifications about food items approaching their expiration date to the user's device, such as a smartphone or tablet, via push notifications or email.

[1551] 5. Recipe generation and suggestions

[1552] The server generates recipes based on ingredients that are approaching their expiration date. The recipes are generated using a database built in the past and user input information (preferences, allergies, etc.). The generated recipes are provided to the user.

[1553] 6. Personalization with user input

[1554] Users input their eating habits, preferences, allergy information, etc. through the application, and the server uses this information to suggest individually optimized recipes.

[1555] 7. Managing food purchase history

[1556] The system also manages the history of food purchases at physical stores, and uses the purchase history database to suggest recipes based on ingredients that are close to their expiration date.

[1557] Hardware and software used

[1558] Hardware

[1559] Camera Method: High resolution camera installed inside the refrigerator.

[1560] User devices: smartphones, tablets.

[1561] Server: A cloud server such as AWS or GCP.

[1562] software

[1563] Image processing: Python, OpenCV, TensorFlow.

[1564] Server side: Django framework.

[1565] Database Management: Django ORM.

[1566] Communication method: Communication using REST API.

[1567] Specific examples of use

[1568] For example, if a user places "tomatoes" and "lettuce" in the refrigerator, the camera takes a picture of them and the server recognizes the type of food. The recognized food is stored in a database, and the user is notified when the expiration date approaches. At the same time, a recipe for "tomato and lettuce salad" is suggested. If the user purchases "chicken" at a physical store, the purchase history is stored in the database and used to suggest recipes for the next time.

[1569] Prompt Sentence Examples

[1570] "Please tell me some recipes that use the tomatoes, lettuce, and chicken I have in my fridge. The user prefers low-calorie recipes, and is allergic to nuts."

[1571] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1572] Step 1:

[1573] A camera means in the refrigerator periodically takes pictures of ingredients. The captured image data is sent to a server. The input is the image of the ingredients in the refrigerator, and the output is the image data sent to the server. The camera takes pictures when the camera button is pressed, or automatically at set intervals.

[1574] Step 2:

[1575] The server analyzes the received image data using a generative AI model to recognize the type of ingredient. The input is the image data sent in step 1, and the output is the recognized ingredient information. The analysis process is performed using Python and TensorFlow.

[1576] Step 3:

[1577] The server stores the information of the recognized ingredients in a database. The database records the type of ingredient, quantity, purchase date, and expiration date. The input is the ingredient information recognized in step 2, and the output is an organized ingredient database. It is saved to the database using Django ORM.

[1578] Step 4:

[1579] The server periodically checks the database and lists ingredients that are approaching their expiration date. The input is the ingredient information in the database, and the output is a list of ingredients that are approaching their expiration date. The checking process is performed by a Python script.

[1580] Step 5:

[1581] The server sends notifications to the user's device about ingredients that are approaching their expiration date. The input is the ingredient information listed in step 4, and the output is a notification sent to the user's smartphone or tablet. Notifications are sent via push notifications or email.

[1582] Step 6:

[1583] The server generates recipes based on ingredients that are approaching their expiration date. Multiple recipes are created using the generative AI model and the user's eating habits, preferences, and allergy information. The input is the ingredient information from step 4 and the user's personal information, and the output is the generated recipe. Prompt statements can be used to generate recipes.

[1584] Step 7:

[1585] The server provides the generated recipe to the user's device. The input is the recipe generated in step 6, and the output is the recipe displayed on the user's smartphone or tablet through the application interface.

[1586] Step 8:

[1587] When a user buys ingredients in a physical store, the purchase history is stored in a database. The input is the user's purchase information, and the output is an updated ingredient purchase history database. The database is managed using Django ORM.

[1588] Step 9:

[1589] The server uses the purchase history database to create a list of food items purchased in physical stores that are nearing their expiration date, and presents this to the user. The input is the updated purchase history database, and the output is a new notification sent to the user's device. The notification is displayed on a smartphone or tablet.

[1590] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1591] This invention is a system that efficiently manages ingredients in a refrigerator and suggests appropriate recipes, and by combining it with an emotion engine that recognizes the user's emotions, it provides more personalized dietary support. This system is composed of a camera for monitoring the status of the refrigerator, a server that processes image data, a terminal that notifies the user and suggests recipes, and an emotion engine that recognizes the user's emotions. Each component and its operation are described in detail below.

[1592] System configuration

[1593] 1. Camera Means

[1594] A camera installed inside the refrigerator periodically takes pictures of the inside of the refrigerator. The camera has high resolution and can clearly capture each food item inside the refrigerator. The captured image data is sent to a server.

[1595] 2. Image data processing by the server

[1596] The server receives the image data sent from the refrigerator and uses an AI model to identify the type of food. The identified food is then stored in a database. The server also estimates the expiration date for each food item and records this information in the database.

[1597] 3. Best before date management

[1598] The server periodically checks the expiration dates of ingredients stored in the database, lists ingredients that are approaching their expiration date, and prepares to notify the user of the listed ingredients.

[1599] 4. Means of notification

[1600] The server sends information about food items approaching their expiration date to the user's device, which can be a smartphone or tablet, and can send push notifications or email notifications to the user.

[1601] 5. Recipe generation and suggestions

[1602] The server generates recipes based on ingredients that are approaching their expiration date. The recipe generation uses a database built in the past and information entered by the user (preferences, allergies, etc.). The generated recipes are presented to the user as multiple options.

[1603] 6. Personalization with user input

[1604] Users input their eating habits, preferences, allergy information, etc. through an application or web interface, and the server uses this information to suggest individually optimized recipes for the user.

[1605] 7. Emotion Engine

[1606] The emotion engine recognizes emotions from the user's voice, facial expressions, text input, etc. The user's emotion information recognized by the emotion engine is also stored in the database.

[1607] Example of program processing

[1608] 1. Recognizing ingredients

[1609] The user places a new ingredient (e.g., cabbage) in the refrigerator.

[1610] The server analyzes the image sent from the camera inside the refrigerator and recognizes the cabbage. The recognition result and the expiration date of the cabbage are recorded in a database.

[1611] 2. Best before date management

[1612] The server periodically checks the database and finds that the cabbage's expiration date is two days away.

[1613] The server sends a notification to the terminal such as "The cabbage's expiration date is in two days."

[1614] 3. Recipe suggestions

[1615] The server generates recipes that use cabbage (e.g., cabbage rolls, cabbage salad).

[1616] The device suggests these recipes to the user.

[1617] When a user selects a recipe for cabbage rolls, detailed instructions and ingredients are displayed on the device.

[1618] 4. Personalization with user input

[1619] Users enter their eating habits, preferences, allergy information, etc. through the application.

[1620] The server stores this information in a database and reflects it in future recipe suggestions.

[1621] 5. Emotion recognition and suggestion optimization

[1622] The emotion engine recognizes emotions from the user's voice and facial expressions and sends this information to the server.

[1623] The server then tailors recipes and notifications to suit the user based on their emotional information, for example, suggesting easy-to-make recipes if the user is tired.

[1624] In this way, the system of the present invention provides optimal dietary support to users by combining ingredient recognition, expiration date management, notifications, recipe suggestions, personalization based on user-entered information, and emotion recognition using an emotion engine.

[1625] The processing flow will be explained below.

[1626] Step 1:

[1627] The user places a new ingredient (e.g., cabbage) in the refrigerator.

[1628] The refrigerator periodically takes pictures of the interior using a built-in camera.

[1629] Step 2:

[1630] Image data captured by the camera means is transmitted to the server.

[1631] The server receives the image data.

[1632] Step 3:

[1633] The server analyzes the image data and uses an AI model to recognize the type of food ingredient (e.g., cabbage).

[1634] The recognition results are recorded in a database.

[1635] Step 4:

[1636] The server automatically estimates the expiration date of the recognized ingredients and stores it in a database.

[1637] The server periodically checks the database to identify ingredients that are approaching their expiration date.

[1638] Step 5:

[1639] The server prepares a notification to the user for listed ingredients that are close to their expiration date.

[1640] A notification such as "The cabbage's expiration date is in two days" is sent to the device.

[1641] Step 6:

[1642] The device displays the push notification to the user.

[1643] The user checks the notification and recognizes ingredients (e.g., cabbage) that are nearing their expiration date.

[1644] Step 7:

[1645] The server requests data to check the user's emotions with an emotion engine.

[1646] The device collects the user's voice and facial expression data and sends it to the server.

[1647] Step 8:

[1648] The emotion engine analyzes the user's emotions (e.g., if the user is tired).

[1649] The server records the user's emotional information recognized by the emotion engine in a database and uses it to adjust the recipe.

[1650] Step 9:

[1651] The server executes a recipe algorithm based on the user's ingredient data and emotion data.

[1652] For example, if the cabbage is nearing its expiration date and the user is tired, the app will suggest an easy recipe (e.g., easy stir-fried cabbage).

[1653] Step 10:

[1654] The server generates recipe suggestions and sends them to the device.

[1655] The device displays recipe suggestions to the user.

[1656] When a user selects a recipe, detailed instructions and required ingredients are displayed.

[1657] Step 11:

[1658] Users enter their eating habits, preferences, allergy information, etc. through the application.

[1659] The server stores this information in a database and reflects it in future recipe suggestions.

[1660] In this way, the system provides optimal dietary support to users by recognizing ingredients, managing expiration dates, providing notifications, suggesting recipes, personalizing with user-entered information, and even recognizing emotions using an emotion engine.

[1661] Example 2

[1662] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1663] In modern households, managing food in the refrigerator is complicated, and food that has passed its expiration date is often wasted. Furthermore, when it comes to effectively utilizing ingredients and suggesting recipes, there is a lack of personalization that takes into account individual users' preferences and allergy information. Furthermore, there is no system that can make appropriate suggestions based on the user's emotional state.

[1664] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a camera means for monitoring the state inside the refrigerator, a means for processing image data captured by the camera means and recognizing the types of ingredients from the image data, a means for estimating and managing the expiration dates of the recognized ingredients, a means for notifying the user of ingredients whose expiration dates are approaching, a means for generating recipes based on the ingredients and suggesting them to the user, a means for making personalized suggestions based on information input by the user, and a means for recognizing the user's emotions from their voice, facial expressions, text input, etc., and adjusting the suggestions. This reduces ingredient waste, makes recipe suggestions that meet the user's individual preferences, and enables optimal suggestions based on the user's emotional state.

[1665] The "camera means" refers to a camera device installed to monitor the state inside the refrigerator, and has the function of taking pictures of ingredients.

[1666] "Image data" refers to visual information of the inside of a refrigerator photographed by a camera means, which is recorded and stored in digital format.

[1667] The "means for recognizing the type of food ingredient" is a function that analyzes the captured image data and identifies the name and attributes of the food ingredient using AI or machine learning models.

[1668] The "means for estimating and managing expiration dates" has the function of predicting and managing expiration dates for recognized food ingredients based on the purchase date and general storage period.

[1669] The "notification means" has a function for transmitting information about ingredients approaching their expiration date, recipe suggestions, etc. to the user's terminal.

[1670] The "means for generating recipes" has the function of automatically creating cooking instructions and steps based on the recognized ingredient information.

[1671] The "means for suggesting to the user" has a function for displaying the generated recipe and notification content to the user.

[1672] The "means for making personalized suggestions" has the function of generating recipes and notification content that take into account individual preferences and allergies based on the user's input information and past data.

[1673] The "means for recognizing emotions and adjusting suggested content" has the function of analyzing the user's current emotional state from their voice, facial expressions, text input, etc., and providing recipes and notification content accordingly.

[1674] This invention is a system that efficiently manages ingredients in the refrigerator and suggests suitable recipes, and by combining it with an emotion engine that recognizes the user's emotions, it provides more personalized dietary support. This system operates based on the following components.

[1675] 1. Camera Means

[1676] The camera installed inside the refrigerator takes high-resolution images of the interior of the refrigerator. The camera periodically (for example, every hour) takes a picture of the overall situation inside the refrigerator and sends this image data to a server. It is desirable to use a network camera with a resolution of 1080p or higher.

[1677] 2. Image data processing by the server

[1678] The server receives the image data sent from the camera and performs image analysis. This image analysis uses AI models such as TensorFlow and PyTorch. This automatically recognizes the types of ingredients in the refrigerator and stores the results in a database. Specifically, it uses the YOLOv3 model to perform high-speed, high-precision object recognition.

[1679] 3. Estimation and management of expiration dates

[1680] The server estimates the expiration date for the recognized ingredients. For example, if it recognizes cabbage, it calculates the expiration date based on the typical storage period for cabbage (about 5 days in the refrigerator) and records this information in the database. The database is a relational database such as MySQL or PostgreSQL.

[1681] 4. Sending notifications

[1682] The server periodically checks the information in the database and sends notifications to the user's device about ingredients that are approaching their expiration date. This notification is done via push notification or email. For example, if the expiration date of cabbage is approaching in two days, a message such as "The expiration date of the cabbage is in two days" is generated and sent to the user's smartphone.

[1683] 5. Recipe generation and suggestions

[1684] The server uses a database of past recipes and information entered by the user (such as preferences and allergies) to generate recipes based on ingredients approaching their expiration date. The generated recipes are presented to the user as multiple options. For example, recipes such as "stuffed cabbage" and "cabbage salad" using cabbage may be generated.

[1685] 6. Personalization with user input

[1686] Users input their eating habits, preferences, allergies, and other information through the application or web interface. The server stores this information in a database and uses it to suggest future recipes, allowing users to receive recipes tailored to their individual needs.

[1687] 7. Emotion Recognition with Emotion Engine

[1688] The emotion engine recognizes emotions from the user's voice, facial expressions, and text input. This emotion information is also sent to the server and stored in a database. The server uses this information to tailor recipes and notification content to suit the user. For example, if the emotion engine recognizes that the user is tired, it will prioritize suggesting easy recipes.

[1689] Examples and prompts

[1690] Example: When a user puts a new cabbage in the refrigerator, the camera recognizes this and sends data to the server to calculate the expiration date of the cabbage. When the expiration date approaches, the server sends a notification and generates several recipes using the cabbage and displays them on the device.

[1691] Example prompt: "Recognize the new ingredients in your refrigerator and suggest a recipe using them."

[1692] In this way, the system of the present invention comprehensively supports everything from managing ingredients in the refrigerator to suggesting recipes for each user, and even suggesting the best recipes based on the user's emotional state. Introducing this system will reduce food waste and lead to a richer and healthier diet for users.

[1693] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1694] Step 1:

[1695] Taking pictures with a camera

[1696] Input: Condition inside the refrigerator

[1697] Process: The camera takes high-resolution images of the inside of the refrigerator periodically (e.g., every hour). Use a network camera with a resolution of 1080p or higher.

[1698] Output: Image data of the inside of the refrigerator

[1699] Specific operation: The location and condition of all food items in the refrigerator are clearly photographed and data is generated.

[1700] Step 2:

[1701] Sending image data

[1702] Input: Image data of the inside of the refrigerator

[1703] Processing: Image data generated by the camera is sent to the server in real time via Wi-Fi.

[1704] Output: Image data sent to the server

[1705] Specific operation: The image file taken by the camera is compressed and a protocol (e.g. HTTP) is executed to send it to the server.

[1706] Step 3:

[1707] Receiving and analyzing image data

[1708] Input: Image data sent from the camera

[1709] Processing: The server receives the image data and uses an AI model (e.g., TensorFlow or PyTorch) to recognize the type of food. Specifically, the YOLOv3 model is used.

[1710] Output: Recognized food type and location information

[1711] Specific operation: The server analyzes the received image data, identifies the name and location of each ingredient, and records it in a database.

[1712] Step 4:

[1713] Estimating expiration dates and storing

[1714] Input: Recognized food type and location information

[1715] Processing: The server estimates the shelf life of each ingredient based on its type. For example, cabbage can be stored in the refrigerator for 5 days.

[1716] Output: Best before date data for each ingredient

[1717] What it does: Calculates the expiration date for each ingredient based on the purchase date and shelf life, and records that information in a database.

[1718] Step 5:

[1719] Regular check of expiration dates

[1720] Input: Best before date data stored in the database

[1721] Processing: The server periodically (e.g., every night at midnight) scans the database and lists ingredients that are nearing their expiration date.

[1722] Output: A list of ingredients that are nearing their expiration date

[1723] What it does: Query the database to find ingredients with a shelf life of less than 3 days.

[1724] Step 6:

[1725] Creating a notification message and preparing it for sending

[1726] Input: A list of ingredients that are nearing their expiration date

[1727] Processing: The server generates a notification message for the listed ingredients, for example, "The cabbage will expire in 2 days."

[1728] Output: The generated notification message

[1729] Specific operation: Automatically generate notification content using a message template and prepare to send it to the user's device.

[1730] Step 7:

[1731] Sending notifications

[1732] Input: The generated notification message

[1733] Process: The server sends a notification message to the user's device via push notification or email notification.

[1734] Output: Notification message received by the user

[1735] Specific operation: The server sends a message to the user's device via push notification or email system.

[1736] Step 8:

[1737] Receiving notifications

[1738] Input: Notification message sent by the server

[1739] Processing: The device receives the notification and displays it on the user's screen as a popup or email.

[1740] Output: The notification message displayed to the user

[1741] Specific behavior: The device's notification system receives the message from the server and immediately displays it to the user.

[1742] Step 9:

[1743] Recipe Generation

[1744] Input: List of ingredients approaching expiration date and user database

[1745] Processing: The server generates recipes based on ingredients that are nearing their expiration date, using a database of past data and user preferences and allergy information.

[1746] Output: A list of generated recipes

[1747] Specific operation: The system generates multiple appropriate recipes based on the ingredients data and the user's profile. For example, it suggests "cabbage rolls" and "cabbage salad" using cabbage.

[1748] Step 10:

[1749] Recipe Suggestions

[1750] Input: A list of generated recipes

[1751] What happens: The device presents the suggested recipes to the user. When the user selects a specific recipe, detailed instructions and required ingredients are displayed.

[1752] Output: Recipe details

[1753] Specific behavior: Displays a list of recipes on the device screen and provides detailed information about the recipe selected by the user.

[1754] Step 11:

[1755] Entering user information

[1756] Input: User-provided dietary habits, preferences, and allergy information

[1757] Processing: The user enters information through an application or web interface and sends it to the server.

[1758] Output: User information stored in the database

[1759] What happens: The user uses the interface to enter their information and presses the submit button. The server receives this and records it in the database.

[1760] Step 12:

[1761] Storage and Use of User Information

[1762] Input: User-submitted dietary habits, preferences, and allergy information

[1763] Processing: The server stores this information in a database and uses it to suggest recipes from the next time onwards.

[1764] Output: Updated user profile

[1765] Specific operation: The server uses the user's input information to personalize future recipe suggestions and notifications.

[1766] Step 13:

[1767] emotion recognition

[1768] Input: User voice, facial expressions, and text input

[1769] Processing: The emotion engine grasps the user's emotion and sends it to the server.

[1770] Output: Recognized user emotion information

[1771] Specific operation: The emotion engine analyzes the user's input data, grasps the user's emotional state, and sends that information to the server.

[1772] Step 14:

[1773] Use of emotional information

[1774] Input: Recognized user emotion information

[1775] Processing: The server records the emotion information in a database and adjusts the suggestions to the user based on this information.

[1776] Output: Adjusted proposal

[1777] Specific operation: The server refers to the emotional information and provides the user with the most appropriate content according to their emotions, such as suggesting simple recipes if the user is tired.

[1778] (Application example 2)

[1779] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1780] In recent years, refrigerator food management and food waste issues have been attracting attention, but traditional manual and simple digital management methods have their limitations. Furthermore, managing ingredients in brick-and-mortar stores and effectively suggesting recipes to customers requires a great deal of effort. Furthermore, while there is a demand for services that take customer emotions into account, no effective system exists to achieve this. To solve these issues, it is necessary to streamline refrigerator food management, provide appropriate information to store staff and customers, and provide personalized services that take customer emotions into account.

[1781] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1782] In this invention, the server includes means for processing image data captured by a camera means and recognizing the type of ingredient from the image data, means for managing the expiration dates of the recognized ingredients, means for notifying the user of ingredients approaching their expiration dates, means for generating recipes corresponding to the ingredients and suggesting them to the user, means for making personalized suggestions based on information input by the user, means for efficiently managing ingredients in a refrigerator and monitoring products stored in refrigerators in the store, means for notifying store staff of products approaching their expiration dates, and means for recognizing customer emotions and suggesting recipes or promoting purchases based on the emotions. This enables efficient management of ingredients in a refrigerator, appropriate management of ingredients in a physical store, personalized recipe suggestions to customers, and service provision based on customer emotions.

[1783] The "camera means" is a photographing device installed to monitor the state inside the refrigerator, and periodically takes pictures of the food ingredients.

[1784] "Image data" refers to video information of ingredients photographed using a camera, and is digital data that is analyzed by the server.

[1785] A "means for recognizing" is a method or device capable of processing image data and identifying and classifying objects in the image.

[1786] The "means for managing expiration dates" refers to a method or device for recording the expiration dates of recognized food ingredients and for notifying users when the expiration date is approaching.

[1787] "Notification means" refers to a method or device used to notify a user or store staff of specific information.

[1788] The "means for generating and suggesting recipes" refers to a method or device that creates a cooking method based on recognized ingredients and provides it to the user.

[1789] The "means for making personalized suggestions" refers to a method or device that makes individually optimized suggestions based on information input by the user, past data, and the like.

[1790] "Means for monitoring products" refers to a method or device for monitoring the condition of products stored in refrigerators in a store and understanding product trends and inventory status.

[1791] A "means for recognizing customer emotions" is a method or device that determines a customer's emotions from voice, facial expressions, text input, etc., and utilizes that information.

[1792] "Means for promoting purchases" are methods and devices for making individually optimized product suggestions based on customer emotions and behavioral data, thereby increasing purchasing motivation.

[1793] The present invention provides a system for improving the efficiency of food ingredient management in a refrigerator and product management in a physical store, and for suggesting personalized recipes to customers and promoting purchases. The following describes in detail an embodiment of the present invention.

[1794] First, regarding the hardware configuration of the entire system, a high-resolution camera is installed inside the refrigerator. This camera periodically takes pictures of the ingredients and products inside the store refrigerator and sends the image data to a server. A configuration using a virtual server on AWS EC2 is suitable for this server.

[1795] The server receives image data sent from the refrigerator and uses an image recognition AI model using TensorFlow to identify ingredients and product types. The identified ingredients are then stored in a database (AWS RDS).

[1796] The server then periodically checks the database and lists ingredients and products that are approaching their expiration date. Store staff are notified of these ingredients and products via push notifications sent via smartphones or tablets. The server also generates recipes based on the products that are approaching their expiration date.

[1797] Recipe generation utilizes previously constructed databases and information entered by users (such as preferences and allergies). The generated recipes are provided to customers via a sales promotion touch panel display and a customer application.

[1798] Furthermore, an emotion engine will be used to recognize emotions from the customer's voice and facial expressions. This emotion information will also be sent to the server, and personalized recipe suggestions and purchase promotions will be made based on the emotion. Microsoft Azure Cognitive Services could be used for the emotion engine.

[1799] The system includes the following specific processing steps:

[1800] 1. The camera means takes an image of the food in the refrigerator.

[1801] 2. The captured image data is sent to the server.

[1802] 3. The server analyzes the image using a TensorFlow model and recognizes the ingredients.

[1803] 4. The recognition results and expiration date information are stored in the database.

[1804] 5. The server periodically checks the database and notifies store staff of any food items that are nearing their expiration date.

[1805] 6. The server generates recipes for customers using ingredients that are close to their expiration date and provides them via a touch panel display or app.

[1806] 7. The emotion recognition engine recognizes the customer's emotions and sends the emotional information to the server.

[1807] 8. Based on the emotional information, the server will suggest recipes and promote purchases according to the emotions.

[1808] For example, consider the following prompt:

[1809] "A new cabbage is placed in the refrigerator. Use an image recognition AI model to recognize the cabbage and add it to the expiration date management system. When the cabbage's expiration date approaches, notify the customer via the touch panel display with a recommended recipe. Also, recognize the customer's emotions based on their facial expressions and suggest recipes that match those emotions."

[1810] This will enable efficient management of food in the refrigerator, proper notification to store staff, and the provision of services that respond to customer emotions.

[1811] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1812] Step 1:

[1813] A camera means within the refrigerator takes images of the food items.

[1814] Input: Refrigerator status

[1815] Data processing: image data capture

[1816] Output: High-resolution food image data

[1817] Step 2:

[1818] The captured image data is sent to a server.

[1819] Input: High-resolution food image data

[1820] Data processing: Sending image data

[1821] Output: Image data received by the server

[1822] Step 3:

[1823] The server uses a TensorFlow model to analyze the image data and recognize the type of ingredient.

[1824] Input: Received food image data

[1825] Data processing: image recognition, food classification

[1826] Output: Recognized ingredients (e.g., tomato, cabbage)

[1827] Step 4:

[1828] The recognized ingredient information and expiration date information are stored in a database on the server.

[1829] Input: Recognized ingredient information

[1830] Data processing: Recording in database, estimating expiration date

[1831] Output: Stored ingredients and expiration date data

[1832] Step 5:

[1833] The server periodically checks the database and notifies store staff of food items that are nearing their expiration date.

[1834] Input: Stored ingredients and expiration date data

[1835] Data processing: Expiration date check, notification information creation

[1836] Output: Notification information (e.g. "The expiration date for the tomatoes is in 2 days")

[1837] Step 6:

[1838] The server generates recipes using ingredients that are close to their expiration date and transmits them to a terminal for serving to the customer.

[1839] Input: Information about ingredients that are close to expiry date, as well as user preferences and allergy information

[1840] Data processing: Recipe generation, recipe data creation

[1841] Output: Recipe suggestions (e.g., tomato salad, tomato pasta)

[1842] Step 7:

[1843] The generated recipe information is provided to the customer via a touch panel display or a customer application.

[1844] Input: Recipe to suggest

[1845] Data processing: Preparing to display recipe information

[1846] Output: Display of recipe (e.g., recipe displayed on a touch panel or smartphone)

[1847] Step 8:

[1848] The emotion engine recognizes emotions from the customer's voice and facial expressions and sends that information to the server.

[1849] Input: Customer voice and facial expression data

[1850] Data processing: Emotion recognition, creating data of the recognition results

[1851] Output: Recognized emotion information (e.g., "The customer is tired")

[1852] Step 9:

[1853] Based on the emotional information, the server makes personalized recipe suggestions or purchase promotions according to the emotions.

[1854] Input: Recognized emotion information

[1855] Data processing: Recipe suggestions readjustment, purchase promotion information creation

[1856] Output: Personalized recipe suggestions and promotional notifications (e.g., "Easy recipes perfect for a tiring day")

[1857] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1858] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1859] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1860] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1861] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1862] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1863] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1864] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1865] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1866] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1867] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1868] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1869] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1870] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1871] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1872] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1873] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1874] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1875] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1876] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1877] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1878] The following is further disclosed regarding the above embodiment.

[1879] (Claim 1)

[1880] a camera means for monitoring the state inside the refrigerator;

[1881] a means for processing image data captured by the camera means and recognizing the type of food material from the image data;

[1882] a means of controlling the expiration dates of recognized ingredients;

[1883] means for notifying a user of food items approaching their expiration date;

[1884] A means for generating recipes corresponding to ingredients and suggesting them to a user;

[1885] means for providing personalized suggestions based on user input;

[1886] A system including:

[1887] (Claim 2)

[1888] A means for periodically acquiring image data of food items in a refrigerator;

[1889] means for transmitting the acquired image data to a server;

[1890] A means for the server to recognize ingredients from image data and store them in a database;

[1891] means for notifying a user when the expiration date is approaching;

[1892] A means of suggesting recipes based on ingredients,

[1893] A means for suggesting recipes according to the user's eating habits and preferences;

[1894] 10. The system of claim 1, comprising:

[1895] (Claim 3)

[1896] Estimating the expiration date of the food material recognized by the camera means;

[1897] Regularly check expiration dates and

[1898] means for sending a notification to the user when the expiration date is approaching;

[1899] means for generating a recipe including ingredients to be used based on the notification;

[1900] a terminal means for providing the generated recipe to a user;

[1901] 10. The system of claim 1, comprising:

[1902] "Example 1"

[1903] (Claim 1)

[1904] an image acquisition means for monitoring the state inside the refrigerator;

[1905] a means for processing image data captured by the image capturing means and recognizing the type of food from the image data;

[1906] A means of controlling the expiration date of recognized foods;

[1907] means for notifying a user of food products approaching their expiration date;

[1908] A means for generating cooking methods based on foods and suggesting them to a user;

[1909] means for providing personalized suggestions based on user input;

[1910] a means for periodically checking the database to check the expiration dates of food products and notifying the user when the expiration date is approaching;

[1911] a means for generating a cooking method based on the notified food whose expiration date is approaching and providing the cooking method to the user;

[1912] A system including:

[1913] (Claim 2)

[1914] A means for periodically acquiring image data of food in a refrigerator;

[1915] means for transmitting the acquired image data to a server;

[1916] A means for the server to recognize food from image data and store the data in a database;

[1917] means for notifying a user when the expiration date is approaching;

[1918] a means for suggesting cooking methods based on food;

[1919] A means for suggesting cooking methods according to the user's eating habits and preferences;

[1920] 10. The system of claim 1, comprising:

[1921] (Claim 3)

[1922] Estimating the expiration date of the food recognized by the image acquisition means;

[1923] Regularly check expiration dates and

[1924] means for sending a notification to the user when the expiration date is approaching;

[1925] means for generating a cooking recipe including the food to be used based on the notification;

[1926] a terminal means for providing the generated recipe to a user;

[1927] 10. The system of claim 1, comprising:

[1928] "Application Example 1"

[1929] (Claim 1)

[1930] a camera means for monitoring the state inside the refrigerator;

[1931] a means for processing image data captured by the camera means and recognizing the type of food material from the image data;

[1932] a means of controlling the expiration dates of recognized ingredients;

[1933] means for notifying a user of food items approaching their expiration date;

[1934] A means for generating recipes corresponding to ingredients and suggesting them to a user;

[1935] means for providing personalized suggestions based on user input;

[1936] A means of managing food purchase history and presenting food items that are approaching their expiration date during purchasing activities at physical stores;

[1937] A system including:

[1938] (Claim 2)

[1939] A means for periodically acquiring image data of food items in a refrigerator;

[1940] means for transmitting the acquired image data to a server;

[1941] A means for the server to recognize ingredients from image data and store them in a database;

[1942] means for notifying a user when the expiration date is approaching;

[1943] A means of suggesting recipes based on ingredients,

[1944] A means for suggesting recipes according to the user's eating habits and preferences;

[1945] A means for users to check expiration date information for ingredients when purchasing in a physical store;

[1946] 10. The system of claim 1, comprising:

[1947] (Claim 3)

[1948] Estimating the expiration date of the food material recognized by the camera means;

[1949] Regularly check expiration dates and

[1950] means for sending a notification to the user when the expiration date is approaching;

[1951] means for generating a recipe including ingredients to be used based on the notification;

[1952] a terminal means for providing the generated recipe to a user;

[1953] A method to manage ingredients based on purchase history at physical stores and suggest recipes based on expiration dates.

[1954] 10. The system of claim 1, comprising:

[1955] "Example 2: Combining Emotion Engines"

[1956] (Claim 1)

[1957] a camera means for monitoring the state inside the refrigerator;

[1958] a means for processing image data captured by the camera means and recognizing the type of food material from the image data;

[1959] a means of estimating and controlling the shelf life of recognized food ingredients;

[1960] means for notifying the user about food items approaching their expiration date;

[1961] A means for generating recipes based on ingredients and suggesting them to a user;

[1962] means for providing personalized suggestions based on user input;

[1963] A means of recognizing emotions from the user's voice, facial expressions, text input, etc., and adjusting the content of suggestions;

[1964] A system including:

[1965] (Claim 2)

[1966] A means for periodically acquiring image data of food items in a refrigerator;

[1967] means for transmitting the acquired image data to a server;

[1968] A means for the server to recognize ingredients from image data and store them in a database;

[1969] means for sending a notification to the user when the expiration date is approaching;

[1970] A means for generating and suggesting recipes based on ingredients;

[1971] A means for providing personalized recipe suggestions based on the user's eating habits and preferences;

[1972] a means for recognizing a user's emotions using an emotion engine and adjusting the content of the suggestions;

[1973] 10. The system of claim 1, comprising:

[1974] (Claim 3)

[1975] means for estimating and periodically checking the expiration dates of the foodstuffs recognized by the camera means;

[1976] means for sending a notification to the user when the expiration date is approaching;

[1977] means for generating a recipe including ingredients to be used based on the notification;

[1978] a terminal means for providing the generated recipe to a user;

[1979] a means for recognizing a user's emotions using an emotion engine and adjusting the content of the suggestions;

[1980] 10. The system of claim 1, comprising:

[1981] "Application example 2 when combining emotion engines"

[1982] (Claim 1)

[1983] a camera means for monitoring the state inside the refrigerator;

[1984] a means for processing image data captured by the camera means and recognizing the typ...

Claims

1. a camera means for monitoring the state inside the refrigerator; a means for processing image data captured by the camera means and recognizing the type of food material from the image data; a means of controlling the expiration dates of recognized ingredients; means for notifying a user of food items approaching their expiration date; A means for generating recipes corresponding to ingredients and suggesting them to a user; means for providing personalized suggestions based on user input; A system including:

2. A means for periodically acquiring image data of food items in a refrigerator; means for transmitting the acquired image data to a server; A means for the server to recognize ingredients from image data and store them in a database; means for notifying a user when the expiration date is approaching; A means of suggesting recipes based on ingredients, A means for suggesting recipes according to the user's eating habits and preferences; The system of claim 1 , comprising:

3. Estimating the expiration date of the food material recognized by the camera means; Regularly check expiration dates and means for sending a notification to the user when the expiration date is approaching; means for generating a recipe including ingredients to be used based on the notification; a terminal means for providing the generated recipe to a user; The system of claim 1 , comprising:

Images