System

Abstract

A system is provided.SOLUTION: A system comprising: a camera that captures an image of a food material in a refrigerator; an identification unit that identifies the food material by analyzing image data captured by the camera; a proposal unit that proposes a menu that can be created based on food material information identified by the identification unit; and a purchase proposal unit that determines a missing food material and proposes purchase based on the menu proposed by the proposal unit.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In modern society, households that cook on a daily basis are required to efficiently utilize the ingredients in their refrigerators. However, finding suitable recipes can be difficult, especially when recipes contain ingredients or seasonings that are not normally used. Furthermore, rising prices have made saving on food costs an unavoidable challenge. Furthermore, there is the hassle of having to go to the supermarket to purchase ingredients that are running low. To solve these issues, there is a need for the development of a system that can automatically identify ingredients in the refrigerator, suggest appropriate menus, and suggest the purchase of ingredients that are running low. [Means for solving the problem]

[0005] The present invention is a system including a camera that photographs ingredients in a refrigerator, an identification means that analyzes image data photographed by the camera to identify ingredients, a suggestion means that suggests menus that can be made based on information about ingredients identified by the identification means, and a purchase suggestion means that identifies ingredients that are in short supply based on the menus suggested by the suggestion means and suggests their purchase. This system further includes a configuration in which the suggestion means suggests menus based on cooking time and user preferences, and the camera periodically photographs the refrigerator door after it is closed, thereby improving home cooking efficiency, saving on food costs, and reducing the hassle of shopping.

[0006] A "refrigerator camera" is a camera that can take pictures of food inside a refrigerator.

[0007] "Image data" refers to data of a digital image captured by a camera.

[0008] The "identification means" is an algorithm or software that analyzes image data to identify ingredients in the refrigerator.

[0009] "Ingredient information" refers to information such as the type and quantity of ingredients recognized using the identification means.

[0010] The "suggestion means" is an algorithm or software for suggesting menus that can be created based on ingredient information.

[0011] A "menu" is a recipe for a dish that can be made using specific ingredients.

[0012] The "purchase suggestion means" is a means for determining the ingredients that are in short supply based on the proposed menu and encouraging the purchase of those ingredients.

[0013] "Cooking time" refers to the time it takes for the user to cook.

[0014] "User preferences" refer to individual preferences such as the user's taste and food preferences.

[0015] A "database" is a data storage device or system for storing identified ingredient information. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] The present invention is a system for efficiently utilizing food ingredients in a refrigerator, saving household food costs and reducing the effort required for cooking. The system includes the following main elements:

[0038] System configuration

[0039] 1. Fridge Camera

[0040] This camera is installed inside the refrigerator and takes pictures of food ingredients. The camera takes pictures periodically after the refrigerator door is closed.

[0041] 2. Server

[0042] It includes means for receiving image data transmitted from the camera and executing an image recognition algorithm to identify ingredients from the data.

[0043] 3. Terminal

[0044] A device used by a user, such as a smartphone or tablet, on which the application is installed.

[0045] Program processing

[0046] 1. Photographing food in the refrigerator

[0047] How the refrigerator camera works

[0048] The refrigerator camera periodically photographs the food inside the refrigerator after the door is closed.

[0049] 2. Analysis of image data

[0050] Server Operation

[0051] The server receives the image data sent from the refrigerator camera.

[0052] An image recognition algorithm is applied to the received image data to identify the ingredients in the refrigerator.

[0053] For example, the server identifies "tomato," "cheese," "lettuce," "bread," etc.

[0054] The identified ingredient information is recorded in a database.

[0055] 3. Create a composable menu

[0056] Server Operation

[0057] The server generates menu options that can be created using a generative AI model based on the identified ingredient information.

[0058] For example, using "tomato," "cheese," "lettuce," and "bread," it suggests "tomato and cheese sandwich" and "lettuce and tomato salad."

[0059] The generated menu candidates are sent to the terminal.

[0060] 4. Menu suggestions

[0061] Device behavior

[0062] The application on the device receives and displays the menu information sent from the server.

[0063] The user selects the desired menu from the proposed menus.

[0064] 5. Check for missing ingredients and suggest purchasing them

[0065] Server Operation

[0066] The server generates a list of ingredients required based on the menu selected by the user.

[0067] It compares the information on food inventory in the refrigerator and extracts any ingredients that are missing.

[0068] For example, if you select "tomato and cheese sandwich," it will identify that "tomato," "cheese," and "bread" are present in the refrigerator, but "mayonnaise" is missing.

[0069] A list of missing ingredients and a purchasing suggestion including a link to the online supermarket's purchasing page are sent to the device.

[0070] 6. Displaying and executing purchase offers

[0071] Device behavior

[0072] The app on the device receives and displays the information about the missing ingredients and purchasing suggestions sent from the server.

[0073] User Actions

[0074] The user checks the list of ingredients they are short of and clicks on a link to an online supermarket to purchase the ingredients they are short of.

[0075] Specific examples

[0076] Using this system, let's consider a case where a refrigerator contains "tomatoes," "cheese," "lettuce," and "bread." The camera takes a picture of these ingredients, and the server analyzes the image data to identify them. Based on the identified ingredients, the server generates menu items such as "tomato and cheese sandwich" and "lettuce and tomato salad," and sends this to the device. The user opens the app, checks the suggested menu items, and selects the desired "tomato and cheese sandwich." Based on this menu, the server identifies the missing "mayonnaise" and suggests a link to an online supermarket to the user. The user can click the link to purchase "mayonnaise."

[0077] In this way, the present invention makes efficient use of ingredients in the refrigerator and suggests suitable menus, thereby saving household food costs and reducing the effort required for cooking.

[0078] The processing flow will be explained below.

[0079] Step 1:

[0080] The refrigerator camera takes pictures of the inside of the refrigerator at regular intervals or after the refrigerator door is closed. The camera generates high-resolution image data and transmits the data to a server.

[0081] Step 2:

[0082] The server receives the image data sent from the refrigerator camera and stores it in a temporary storage area for saving the image data.

[0083] Step 3:

[0084] The server applies image recognition algorithms to identify ingredients in the image. For example, the server uses image recognition software to identify ingredients such as "tomato," "cheese," "lettuce," and "bread."

[0085] Step 4:

[0086] The server records the identified ingredient information in a database, which includes detailed information such as the type of ingredient, the quantity, and the date and time of identification.

[0087] Step 5:

[0088] The server uses a generative AI model to generate possible menu items based on the ingredient information stored in the database. For example, using "tomato," "cheese," "lettuce," and "bread," it can suggest "tomato and cheese sandwich" or "lettuce and tomato salad."

[0089] Step 6:

[0090] The server sends the generated menu options and necessary ingredient information to the terminal, which receives this information and prepares to display it to the user.

[0091] Step 7:

[0092] The app on the device receives the menu information sent from the server and displays it to the user. The user opens the app and checks the proposed menu.

[0093] Step 8:

[0094] The user selects the desired menu from the proposed menus, and the user's selection information is sent from the terminal to the server.

[0095] Step 9:

[0096] The server generates a list of ingredients required for the selected menu based on the user's selection, and compares it with ingredient inventory information to extract any ingredients that are in short supply.

[0097] Step 10:

[0098] The server compiles a list of ingredients that are missing, generates a link to the online supermarket's purchasing page, and sends this information to the device.

[0099] Step 11:

[0100] The app on the device receives the information about the missing ingredients and the purchase suggestions sent from the server and displays them to the user.

[0101] Step 12:

[0102] The user checks the list of ingredients they are short of and clicks on a link to an online supermarket, where they can purchase the ingredients they are short of.

[0103] Example 1

[0104] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0105] Modern households are seeking to effectively utilize the ingredients in their refrigerators to save on food costs and reduce the effort required for cooking. However, current systems lack the functionality to automatically recognize ingredients in the refrigerator and suggest menus based on that information, forcing users to manually manage ingredients. Furthermore, their functionality to support the purchase of ingredients that are in short supply is limited. To address these issues, the present invention aims to provide a series of processes for automatically identifying ingredients in the refrigerator, generating menus based on that information, and purchasing ingredients that are in short supply.

[0106] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0107] In this invention, the server includes a camera for taking pictures of ingredients in the refrigerator, a means for receiving and saving image data taken by the camera, a means for identifying ingredients by applying an image recognition algorithm to the received and saved image data, a means for recording information on ingredients identified by the camera in a database, a means for generating menu candidates that can be created using a generative AI model based on the identified ingredient information, a means for sending the generated menu candidates to a user terminal, a means for displaying the generated menu candidates on the user terminal, a means for generating a list of necessary ingredients based on the menu selected by the user, a means for comparing the list of necessary ingredients with information on ingredient inventory in the refrigerator and extracting missing ingredients, and a means for sending the list of missing ingredients and a link to a purchasing page to the user terminal. This allows the system to automatically manage ingredients in the refrigerator, suggest suitable menus to the user, and quickly and easily purchase missing ingredients.

[0108] The "photography device" is a device for photographing the environment inside the refrigerator, and mainly includes a camera.

[0109] "Image data" is digital data obtained by photographing the food items in the refrigerator.

[0110] The "receiving and saving means" refers to a means for receiving image data transmitted from the imaging device and saving it temporarily or permanently, and includes server storage and the like.

[0111] An "image recognition algorithm" is a computational method for analyzing image data and identifying specific objects or features, including approaches that utilize artificial intelligence techniques.

[0112] The "identification means" is a means for applying an image recognition algorithm to identify ingredients in the refrigerator and obtain information about them.

[0113] A "database" is a system for systematically storing and managing information on identified ingredients, etc., and includes SQL databases, NoSQL databases, etc.

[0114] A "generative AI model" is an artificial intelligence model that generates new information and suggestions based on input data, and is based on machine learning and deep learning.

[0115] "Menu candidates" are options for dishes and recipes suggested by the generative AI model based on identified ingredient information.

[0116] A "user terminal" is a device used by a user, including a smartphone, tablet, computer, etc.

[0117] The "list of necessary ingredients" is a list of all ingredients required to prepare the menu selected by the user.

[0118] "Inventory information" refers to information about all ingredients currently in the refrigerator, including details such as the quantity and type.

[0119] "Missing ingredients" refers to ingredients that are needed to prepare the selected menu but are not currently present in the refrigerator.

[0120] The "Purchase Page Link" is a hyperlink to a web page that allows the user to purchase the ingredients they are running low on online.

[0121] This invention is a system for efficiently utilizing ingredients in a refrigerator, saving household food costs and reducing the effort required for cooking. The system includes a camera for photographing ingredients in the refrigerator, a server for receiving and storing image data, a means for identifying ingredients using an image recognition algorithm based on the image data, a database for recording information on identified ingredients, a means for generating menu options using a generative AI model, and a means for extracting ingredients that are in short supply and suggesting their purchase.

[0122] First, a refrigerator camera is used to take pictures of the food inside the refrigerator. This camera automatically starts up after the refrigerator door is closed and periodically takes pictures of the food inside. For example, it takes a total of three images three seconds after the refrigerator door is closed. The captured image data is sent to a server via Wi-Fi.

[0123] Next, the server receives this image data and temporarily stores it in a database. Image recognition algorithms using TensorFlow, OpenCV, etc. are applied to the stored image data. This algorithm identifies the ingredients in the refrigerator, and ingredient information such as "tomato," "cheese," "lettuce," and "bread" is recorded in the database.

[0124] The server then uses this identified ingredient information to generate menu suggestions that can be created using a generative AI model (e.g., GPT-3). An example prompt used here is:

[0125] "There are tomatoes, cheese, lettuce, and bread in the refrigerator. Can you suggest a recipe using these?"

[0126] This generates menu options such as "tomato and cheese sandwich" and "lettuce and tomato salad," which are then sent to the user's device.

[0127] An application installed on the user's device receives this menu information and displays it on the screen. The user can then select the desired menu from the suggested menu. For example, if the user selects a "tomato and cheese sandwich," the server generates a list of ingredients based on this menu and compares it with the inventory information of ingredients in the refrigerator.

[0128] At this stage, the missing ingredients are identified, and it is determined that "mayonnaise," for example, is in short supply. The server calls the online supermarket's API, obtains a purchase link for "mayonnaise," and sends it to the user's device. The user can click this link to easily purchase the missing ingredients.

[0129] As described above, the present invention can improve user convenience by automatically managing ingredients in the refrigerator, suggesting suitable menus, and suggesting the purchase of ingredients that are in short supply, thereby saving food costs and reducing the effort required for cooking.

[0130] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0131] Step 1:

[0132] How the refrigerator camera works

[0133] After the refrigerator door is closed, the refrigerator camera automatically activates and takes a picture of the food inside.

[0134] input

[0135] A signal that the refrigerator door is closing.

[0136] output

[0137] High resolution image data of the inside of a refrigerator.

[0138] Specific actions

[0139] The camera takes three images three seconds after the door is closed and saves them as digital image files in JPEG format.

[0140] Step 2:

[0141] Receiving image data from the server

[0142] The image data captured by the refrigerator camera is sent via Wi-Fi to a server, which receives and stores it.

[0143] input

[0144] Image data sent from the refrigerator camera.

[0145] output

[0146] Saving received image data.

[0147] Specific actions

[0148] The server receives image files from the camera via the FTP server function and stores them in a specific directory.

[0149] Step 3:

[0150] Server image analysis

[0151] The server applies an image recognition algorithm (e.g., TensorFlow's Inception model) to the received image data to identify the ingredients.

[0152] input

[0153] Saved image data.

[0154] output

[0155] Identified ingredient information.

[0156] Specific actions

[0157] The server uses the TensorFlow library to perform object detection on the stored image data, identifying ingredients such as "tomato," "cheese," "lettuce," and "bread." The results are stored in a database in JSON format.

[0158] Step 4:

[0159] Server menu generation

[0160] Based on the identified ingredient information, the server generates menu options that can be created using a generative AI model (e.g., GPT-3).

[0161] input

[0162] Identified ingredient information.

[0163] output

[0164] The generated menu suggestions.

[0165] Specific actions

[0166] The server inputs the following prompt into the generative AI model:

[0167] "There are tomatoes, cheese, lettuce, and bread in the refrigerator. Can you suggest a recipe using these?"

[0168] The generated results are "tomato and cheese sandwich" and "lettuce and tomato salad", which are saved in JSON format.

[0169] Step 5:

[0170] Sending and displaying menu information

[0171] The server sends the generated menu suggestions to the user's terminal, which displays them.

[0172] input

[0173] The generated menu suggestions.

[0174] output

[0175] Displaying menu information on the user's terminal.

[0176] Specific actions

[0177] The server pushes the JSON data of the menu suggestions to the user's device, which receives and displays it in the device's application. The user can then select the desired menu from the suggested menus.

[0178] Step 6:

[0179] Identifying missing ingredients

[0180] Based on the menu selected by the user, the server generates a list of ingredients needed, compares it with the inventory information in the refrigerator, and extracts any ingredients that are missing.

[0181] input

[0182] The menu selected by the user.

[0183] output

[0184] Missing ingredient information.

[0185] Specific actions

[0186] The server does the following:

[0187] 1. Refer to the recipe database for "tomato and cheese sandwich" and generate a list of ingredients needed.

[0188] 2. Compare with the food information in the refrigerator and determine that there is a shortage of "mayonnaise."

[0189] Step 7:

[0190] Submit a Purchase Offer

[0191] The server sends a list of missing ingredients and a link to the purchasing page to the user's terminal.

[0192] input

[0193] Missing ingredient information.

[0194] output

[0195] Sending purchase suggestion information to the user terminal.

[0196] Specific actions

[0197] The server calls the online supermarket's API, obtains a purchase link for "mayonnaise," and sends this along with a list of ingredients that are in short supply to the user's terminal.

[0198] Step 8:

[0199] View purchase suggestions

[0200] The device's app receives the information about the missing ingredients and the purchase suggestions sent from the server and displays them on the screen.

[0201] input

[0202] Purchase offer information sent from the server.

[0203] output

[0204] Display of purchase suggestion information on user terminal.

[0205] Specific actions

[0206] The app displays a list of missing ingredients in a pop-up window. For example, "You're running low on mayonnaise. Here's a link to buy it." The user can click the link to buy the missing ingredients.

[0207] (Application example 1)

[0208] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0209] In modern households, it is important to efficiently manage food in the refrigerator and optimize consumption. However, currently, it is time-consuming to identify the presence of ingredients and identify what is missing, and it is difficult to suggest appropriate menus or replenish missing ingredients. Furthermore, particularly in the field of food delivery, there is no technology that automatically replenishes or suggests ingredients when a specific ingredient is missing. Therefore, there is a need for a system that can efficiently manage food in the refrigerator and even automatically replenish missing ingredients.

[0210] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0211] In this invention, the server includes an imaging device that photographs ingredients in the refrigerator, identification means that analyzes image data photographed by the imaging device to identify ingredients, suggestion means that proposes menus that can be created based on ingredient information identified by the identification means, purchase suggestion means that identifies ingredients that are lacking based on the menus proposed by the suggestion means and suggests their purchase, an auxiliary device that is installed in an automatic delivery machine associated with the refrigerator and that scans ingredients in the refrigerator with an imaging device, and replenishment means that the automatic delivery machine automatically replenishes ingredients that are lacking. This makes it possible to efficiently manage ingredients in the refrigerator and automatically replenish ingredients that are lacking.

[0212] An "imaging device that photographs food ingredients in a refrigerator" is a device that is installed inside a refrigerator and that captures image data of food ingredients in the refrigerator.

[0213] The "identification means" is a device or program that has the function of analyzing image data captured by the imaging device and identifying each food ingredient in the refrigerator.

[0214] The "suggestion means" is a device or program that has the function of generating and suggesting menus that can be made using ingredients based on the ingredient information identified by the identification means.

[0215] The "purchase suggestion means" is a device or program having the function of identifying ingredients that are in short supply from the ingredient information in the refrigerator based on the menu generated by the suggestion means and suggesting to the user that they be purchased.

[0216] A "delivery automaton" is a mechanical device, such as an automatically mobile robot, used to assist in the management of refrigerators.

[0217] The "auxiliary device" is an imaging device that is installed in the automated delivery machine and is used to scan ingredients in the refrigerator.

[0218] The "replenishment means" is a device or program that has the function of identifying a shortage of ingredients and automatically replenishing the ingredients when necessary.

[0219] The present invention is a system for efficiently managing ingredients in a refrigerator and automatically replenishing ingredients when they are in short supply. Detailed embodiments of the present invention will be described below.

[0220] The present invention includes an imaging device that photographs ingredients in a refrigerator, a server, a user terminal, and an automated delivery machine. The imaging device is installed inside the refrigerator and periodically photographs ingredients and sends the image data to the server. The server analyzes the received image data and has an identification means for identifying the ingredients.

[0221] The server generates a menu that can be prepared using the suggestion means based on the identified ingredient information and transmits it to the user's terminal. The user can check and select the suggested menu through the terminal. Based on the selected menu, the server's purchase suggestion means identifies ingredients that are in short supply and suggests their purchase to the user.

[0222] Furthermore, the delivery machine is equipped with an auxiliary device that can scan the ingredients in the refrigerator. As a means of replenishment, the delivery machine can automatically replenish any ingredients that are missing.

[0223] The system operates as follows.

[0224] 1. Scan food in the refrigerator

[0225] An image capture device installed in the delivery machine periodically scans the food items in the refrigerator and captures image data, for example, after the refrigerator door is closed.

[0226] 2. Analysis of image data

[0227] The server receives the image data sent from the imaging device and identifies the ingredients using image analysis software (e.g., TensorFlow or OpenCV). The identified ingredient information is recorded in a database.

[0228] 3. Creating a Menu

[0229] The server uses a generative AI model to generate menu options based on the identified ingredients. For example, if "tomato," "cheese," "lettuce," and "bread" are identified, the server will suggest options such as "tomato and cheese sandwich" or "lettuce and tomato salad."

[0230] 4. Identifying missing ingredients and suggesting purchases

[0231] The server generates a list of ingredients based on the user's menu selection and compares it with the refrigerator's inventory data. If any ingredients are missing, a purchase suggestion is sent to the user's device.

[0232] 5. Automatic replenishment function

[0233] The automated delivery machine follows instructions from the server, automatically orders missing ingredients from online supermarkets, etc., and replenishes the refrigerator if necessary.

[0234] As a concrete example, consider a situation where there are "tomatoes," "cheese," "lettuce," and "bread" in a refrigerator. The image capture device of the automated delivery machine takes a picture of these items, and the server analyzes and identifies the image data. Based on the identified ingredient information, the server generates and suggests menu items such as "tomato and cheese sandwich" or "lettuce and tomato salad." If the user selects "tomato and cheese sandwich," the server identifies the missing "mayonnaise" and suggests a link to purchase it from an online supermarket to the user. Furthermore, the automated delivery machine automatically purchases mayonnaise and replenishes the refrigerator.

[0235] Example prompt for generative AI model:

[0236] We will build an AI system that recognizes ingredients in the user's refrigerator and suggests ingredients that are missing. Please identify ingredients from the input image, compare them with the list of necessary ingredients, extract the missing ingredients, and generate a purchase link.

[0237] As described above, the present invention can improve the efficiency of food ingredient management in a refrigerator and realize automatic replenishment of ingredients that are running low.

[0238] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0239] Step 1:

[0240] Scan food in the refrigerator

[0241] Input: Image data from an imaging device (camera) installed inside the refrigerator

[0242] Specific operation: After the refrigerator door is closed, the image capture device takes an image of the inside of the refrigerator. The captured image data is sent to the server.

[0243] Output: Image data of ingredients in the refrigerator

[0244] Step 2:

[0245] Receiving and analyzing image data

[0246] Input: Image data of the inside of the refrigerator sent from the imaging device

[0247] Specific operation: The server receives the image data and identifies the ingredients using image analysis software (e.g., TensorFlow or OpenCV). The identified ingredient information is then recorded in a database.

[0248] Output: Identification information of ingredients (e.g., "tomato," "cheese," "lettuce," "bread")

[0249] Step 3:

[0250] Generate Menu

[0251] Input: Identification information of ingredients recorded on the server

[0252] Specific operation: The server uses the generative AI model to generate menu options based on the identified ingredients. The generated menu options are sent to the user's device.

[0253] Output: Possible menu items (e.g., "Tomato and cheese sandwich," "Lettuce and tomato salad")

[0254] Step 4:

[0255] Menu suggestions and selections

[0256] Input: Menu suggestions sent from the server to the user's device

[0257] Specific operation: The user's device displays the menu options received from the server, and the user selects the desired menu.

[0258] Output: The menu selected by the user

[0259] Step 5:

[0260] Identifying missing ingredients and suggesting purchases

[0261] Input: Menu selected by the user and ingredient information in the database

[0262] Specific operation: Based on the selected menu, the server generates a list of necessary ingredients and checks it against the refrigerator inventory in the database. If any ingredients are missing, it sends a purchase suggestion to the user's device.

[0263] Output: List of ingredients that are in short supply and purchase suggestions (e.g., "mayonnaise" purchase link)

[0264] Step 6:

[0265] Purchasing and Auto-Refill

[0266] Input: Purchase offers and delivery machine replenishment orders sent from the server to the user

[0267] Specific operation: When the user checks the purchase suggestion and clicks on the link to the online supermarket to purchase the missing ingredients, the delivery machine automatically replenishes the missing ingredients based on instructions from the server. The delivery machine is equipped with an auxiliary device that places the purchased ingredients in the refrigerator.

[0268] Output: Automatically replenished food in the refrigerator

[0269] Example prompt sentence:

[0270] We will build an AI system that recognizes ingredients in the user's refrigerator and suggests ingredients that are missing. Please identify ingredients from the input image, compare them with the list of necessary ingredients, extract the missing ingredients, and generate a purchase link.

[0271] This allows users to efficiently manage ingredients in their refrigerators and automatically replenish ingredients when they run out.

[0272] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0273] The present invention combines a system for efficiently utilizing ingredients in the refrigerator, saving household food costs and reducing cooking time, with an emotion engine that recognizes the user's emotions. The system includes the following main components:

[0274] System configuration

[0275] 1. Fridge Camera

[0276] This camera is installed inside the refrigerator and takes pictures of food ingredients. The camera takes pictures periodically after the refrigerator door is closed.

[0277] 2. Server

[0278] It includes means for receiving image data transmitted from the camera and executing an image recognition algorithm to identify ingredients from the data.

[0279] It is equipped with an emotion engine that recognizes the user's emotions and suggests appropriate menus based on those emotions.

[0280] 3. Terminal

[0281] A device used by a user, such as a smartphone or tablet, on which the application is installed.

[0282] Program processing

[0283] 1. Photographing food in the refrigerator

[0284] How the refrigerator camera works

[0285] The refrigerator camera periodically photographs the food inside the refrigerator after the door is closed.

[0286] 2. Analysis of image data

[0287] Server Operation

[0288] The server receives the image data sent from the refrigerator camera.

[0289] An image recognition algorithm is applied to the received image data to identify the ingredients in the refrigerator.

[0290] For example, the server identifies "tomato," "cheese," "lettuce," "bread," etc.

[0291] The identified ingredient information is recorded in a database.

[0292] 3. Create a composable menu

[0293] Server Operation

[0294] The server generates menu options that can be created using a generative AI model based on the identified ingredient information.

[0295] For example, using "tomato," "cheese," "lettuce," and "bread," it suggests "tomato and cheese sandwich" and "lettuce and tomato salad."

[0296] 4. Emotion Recognition by Emotion Engine

[0297] Server Operation

[0298] The server uses an emotion engine to recognize the user's emotions by analyzing the user's facial expressions and tone of voice.

[0299] The emotion engine also learns the user's past selection history and behavioral patterns to predict emotions.

[0300] 5. Menu suggestions based on emotions

[0301] Server Operation

[0302] The server proposes a menu according to the user's emotions based on the user's emotion information obtained from the emotion engine.

[0303] For example, if the server detects that the user is tired, it will suggest a menu that is simple and quick to prepare.

[0304] 6. Menu suggestions

[0305] Device behavior

[0306] The application on the device receives the menu information sent from the server and displays it to the user.

[0307] The user opens the app and sees the suggested menu.

[0308] 7. Check for missing ingredients and suggest purchasing them

[0309] Server Operation

[0310] The server generates a list of ingredients required based on the menu selected by the user.

[0311] The system compares the information with food stock information and extracts any ingredients that are in short supply.

[0312] 8. Displaying and executing purchase offers

[0313] Device behavior

[0314] The app on the device receives the information about the missing ingredients and the purchase suggestions sent from the server and displays them to the user.

[0315] User Actions

[0316] The user checks the list of ingredients they are short of and clicks on a link to an online supermarket to purchase the ingredients they are short of.

[0317] Specific examples

[0318] Using this system, let's assume that there are "tomatoes," "cheese," "lettuce," and "bread" in the refrigerator. The camera takes a picture of these ingredients, and the server analyzes the image data to identify them. Based on the identified ingredients, the server generates a menu such as "tomato and cheese sandwich" or "lettuce and tomato salad" and sends it to the terminal.

[0319] When a user opens the app and checks the suggested menu, the emotion engine analyzes the user's facial expressions and tone of voice. If it determines that the user is tired, it suggests a "tomato and cheese sandwich," which is easy and quick to make. When the user selects this suggestion, the server checks the ingredients needed for the "tomato and cheese sandwich" and identifies the missing "mayonnaise." The server then suggests the user purchase "mayonnaise" from an online supermarket, and the user can click a link to purchase it.

[0320] In this way, by combining emotion engines, it becomes possible to suggest menus that correspond to the user's emotions, enabling more individually tailored dish suggestions to be realized.

[0321] The processing flow will be explained below.

[0322] Step 1:

[0323] The refrigerator camera takes pictures of the inside of the refrigerator at regular intervals or after the refrigerator door is closed. The camera generates high-resolution image data and transmits the data to a server.

[0324] Step 2:

[0325] The server receives the image data sent from the refrigerator camera and stores it in a temporary storage area for saving the image data.

[0326] Step 3:

[0327] The server applies image recognition algorithms to identify ingredients in the image. For example, the server uses image recognition software to identify ingredients such as "tomato," "cheese," "lettuce," and "bread."

[0328] Step 4:

[0329] The server records the identified ingredient information in a database, which includes detailed information such as the type of ingredient, the quantity, and the date and time of identification.

[0330] Step 5:

[0331] The server uses a generative AI model to generate possible menu items based on the ingredient information stored in the database. For example, using "tomato," "cheese," "lettuce," and "bread," it can suggest "tomato and cheese sandwich" or "lettuce and tomato salad."

[0332] Step 6:

[0333] The server sends the generated menu options and necessary ingredient information to the terminal, which receives this information and prepares to display it to the user.

[0334] Step 7:

[0335] The app on the device receives the menu information sent from the server and displays it to the user. The user opens the app and checks the proposed menu.

[0336] Step 8:

[0337] The app on the device recognizes the user's facial expressions and tone of voice and sends that data to the emotion engine, which then analyzes facial expressions and voice to recognize the user's emotional state.

[0338] Step 9:

[0339] The emotion engine on the server recognizes the user's emotions based on the analysis results and adjusts the menu suggestions based on the user's emotional state. For example, if the system recognizes that the user is tired, it will prioritize suggestions of simple and quick meals.

[0340] Step 10:

[0341] The user selects the desired menu from the proposed menus, and the user's selection information is sent from the terminal to the server.

[0342] Step 11:

[0343] The server generates a list of ingredients required for the selected menu based on the user's selection, and compares it with ingredient inventory information to extract any ingredients that are in short supply.

[0344] Step 12:

[0345] The server compiles a list of ingredients that are missing, generates a link to the online supermarket's purchasing page, and sends this information to the device.

[0346] Step 13:

[0347] The app on the device receives the information about the missing ingredients and the purchase suggestions sent from the server and displays them to the user.

[0348] Step 14:

[0349] The user checks the list of ingredients they are short of and clicks on a link to an online supermarket, where they can purchase the ingredients they are short of.

[0350] Example 2

[0351] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0352] In modern households, efficiently utilizing the ingredients in the refrigerator to reduce food costs and cooking time is an important issue. However, it is difficult for users to properly manage the ingredients in their refrigerator, resulting in frequent waste and unused ingredients. In addition, menu suggestions are not tailored to the user's condition or mood, making it difficult for users to maintain their motivation to cook. Furthermore, if necessary ingredients are in short supply, users have to go shopping separately, which is a hassle.

[0353] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0354] In this invention, the server includes a camera that takes pictures of ingredients in the refrigerator, an image analysis unit that analyzes image data taken by the camera and identifies ingredients, a suggestion unit that suggests menus that can be made based on the ingredient information identified by the image analysis unit, a purchase suggestion unit that identifies ingredients that are lacking based on the menus suggested by the suggestion unit and suggests the purchase of those ingredients, and an emotion recognition unit that recognizes the user's emotions and suggests appropriate menus accordingly. This makes it possible to efficiently use ingredients in the refrigerator and reduce household food costs, and furthermore, by suggesting menus that correspond to the user's emotions, it is possible to improve motivation to cook and reduce the effort required.

[0355] "Photography device" refers to a device installed to photograph food items in a refrigerator.

[0356] "Image analysis means" refers to algorithms or software that analyzes image data acquired by the imaging device and identifies ingredients.

[0357] The "suggestion means" refers to a system for suggesting menus that can be prepared based on the ingredient information identified by the image analysis means.

[0358] The "purchase suggestion means" refers to a system for determining ingredients that are in short supply based on the menu suggested by the suggestion means and suggesting to the user to purchase those ingredients.

[0359] "Emotion recognition means" refers to algorithms or devices that analyze a user's facial expressions and tone of voice to recognize their emotions.

[0360] A "refrigerator" refers to a cooling device used in homes and businesses to preserve food.

[0361] "Ingredients" refers to food stored in the refrigerator for use in cooking.

[0362] "Image data" refers to digital data of photographs and videos captured by a photographing device.

[0363] "Ingredient information" refers to data relating to the type and quantity of ingredients identified by the image analysis means.

[0364] "Menu" refers to a list of dishes that can be made using ingredients.

[0365] The present invention combines a system for efficiently utilizing ingredients in a refrigerator, saving household food costs and reducing the effort required for cooking, with an emotion engine that recognizes the user's emotions. This system includes a photographing device for photographing ingredients in the refrigerator, image analysis means for analyzing image data and identifying ingredients, suggestion means for suggesting menus based on identified ingredient information, purchase suggestion means for identifying ingredients that are in short supply and suggesting their purchase, and emotion recognition means for recognizing the user's emotions and suggesting appropriate menus.

[0366] Specific hardware and software:

[0367] 1. Camera: A camera installed inside the refrigerator takes pictures periodically after the refrigerator door is closed. For example, a commercially available webcam or a special-purpose camera can be used.

[0368] 2. Image analysis means: Uses image recognition algorithms installed on the server. Specifically, open source image processing libraries and machine learning models (e.g., TensorFlow and OpenCV) can be used.

[0369] 3. Proposal method: Menu suggestions are made using a generative AI model. Specifically, a generative AI model that performs natural language processing (e.g., GPT-3) is used to generate a menu based on ingredient information.

[0370] 4. Purchase suggestion means: Using a database management system installed on the server, the missing ingredients are identified based on the identified ingredient information and the proposed menu, and a purchase suggestion is made.

[0371] 5. Emotion Recognition: An emotion engine is used to analyze the user's facial expressions and tone of voice. For example, a deep learning model (e.g., FaceNet) can be used for facial expression analysis, and a speech recognition engine (e.g., Google Cloud Speech-to-Text) can be used for voice analysis.

[0372] Operational flow and data calculation:

[0373] The server receives image data sent from the refrigerator camera and performs image analysis to identify ingredients from the data. The analyzed ingredient information is recorded in a database within the server. The server then uses a generative AI model based on this ingredient information to generate possible menu options. When the user uses the application to check the proposed menu, the emotion engine analyzes the user's facial expressions and tone of voice and presents a menu that corresponds to the user's emotions. Furthermore, based on the proposed menu, the server determines which ingredients are in short supply and suggests that the user purchase them.

[0374] Example 1:

[0375] If there are "tomatoes," "cheese," "lettuce," and "bread" in the refrigerator, the camera takes a picture of these ingredients, and the server analyzes the image data to identify them. Based on the identified ingredient information, the server generates menu items such as "tomato and cheese sandwich" and "lettuce and tomato salad," and sends them to the device. When the user opens the app and checks the suggested menu items, the emotion engine analyzes the user's facial expressions and tone of voice, and if it determines that the user is tired, it suggests "tomato and cheese sandwich," which is easy to make and can be made quickly. If the user selects this suggestion, the server checks the ingredients needed for the "tomato and cheese sandwich" and identifies the missing "mayonnaise." The server then suggests the user purchase "mayonnaise" from an online supermarket, and the user can click a link to purchase it.

[0376] Example 2:

[0377] Suggest a menu that is easy and quick to make using the "chicken," "carrots," and "onions" in the refrigerator. Also, if the user is tired, prioritize items that are easy and nutritious. An example of a prompt sentence is, "Suggest a menu that is easy and quick to make using the "chicken," "carrots," and "onions" in the refrigerator. Also, if the user is tired, prioritize items that are easy and nutritious."

[0378] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0379] Step 1:

[0380] Photographing food in the refrigerator

[0381] How the refrigerator camera works

[0382] The refrigerator camera periodically takes pictures of the inside of the refrigerator after the door is closed.

[0383] For example, a refrigerator is set to take a photo every hour.

[0384] Input: Image of the inside of the refrigerator

[0385] Output: Captured image data

[0386] Step 2:

[0387] Image data analysis

[0388] Server Operation

[0389] The server receives the image data sent from the refrigerator camera.

[0390] An image recognition algorithm is applied to the received image data to identify the ingredients inside.

[0391] For example, TensorFlow or OpenCV is used as an image recognition algorithm.

[0392] Input: Photographed image data

[0393] Data processing and calculation: Apply image recognition algorithms to identify the type of food

[0394] Output: Identified ingredients (e.g., "tomato," "cheese," "lettuce," "bread," etc.)

[0395] Step 3:

[0396] Generate composable menus

[0397] Server Operation

[0398] The server generates menu options that can be created using a generative AI model based on the identified ingredient information.

[0399] Specifically, we use a generative AI model (e.g., GPT-3) that performs natural language processing.

[0400] Input: Identified ingredient information

[0401] Data processing and data calculation: Generate menu suggestions using generative AI models

[0402] Output: Generated menu options (e.g., "tomato and cheese sandwich," "lettuce and tomato salad," etc.)

[0403] Step 4:

[0404] Emotion recognition by emotion recognition means

[0405] Server Operation

[0406] The server uses an emotion recognition means to recognize the user's emotion.

[0407] A deep learning model (e.g., FaceNet) is used for facial expression analysis, and a speech recognition engine (e.g., Google Cloud Speech-to-Text) is used for voice analysis.

[0408] Input: User's facial image and voice data

[0409] Data processing and data calculation: Applying emotion recognition algorithms to analyze user emotions

[0410] Output: Analyzed user's emotional information (e.g., "Tired," "Relaxed," etc.)

[0411] Step 5:

[0412] Menu suggestions based on emotions

[0413] Server Operation

[0414] The server proposes a menu according to the user's emotion based on the emotion information obtained from the emotion recognition means.

[0415] For example, if the user is tired, the system will prioritize suggesting menus that are simple and can be prepared in a short time.

[0416] Input: Analyzed user emotion information and generated menu suggestions

[0417] Data processing and data calculation: determining menu priorities based on emotional information

[0418] Output: Menu suggestions based on emotions (e.g., "Tomato and cheese sandwich")

[0419] Step 6:

[0420] Menu suggestions

[0421] Device behavior

[0422] The application on the device receives the menu information sent from the server and displays it to the user.

[0423] The user opens the app and sees the suggested menu.

[0424] For example, a smartphone or tablet app might display menu items such as "Tomato and Cheese Sandwich" or "Lettuce and Tomato Salad."

[0425] Input: Emotion-based menu suggestions

[0426] Output: Displayed menu information

[0427] Step 7:

[0428] Check for missing ingredients and suggest purchasing

[0429] Server Operation

[0430] The server generates a list of ingredients required based on the menu selected by the user.

[0431] The generated ingredient list is compared with the inventory information in the refrigerator to extract any ingredients that are missing.

[0432] For example, it may identify that a tomato and cheese sandwich requires mayonnaise, but that it is not in stock in the refrigerator.

[0433] Input: Selected menu, refrigerator inventory information

[0434] Data processing and calculation: Generate a list of necessary ingredients and extract missing ingredients

[0435] Output: Missing ingredient information (e.g., "mayonnaise")

[0436] Step 8:

[0437] View and execute purchase offers

[0438] Device behavior

[0439] The app on the device receives the information about the missing ingredients and the purchase suggestions sent from the server and displays them to the user.

[0440] For example, a message may appear saying, "We're low on mayonnaise. Would you like to purchase some from the online supermarket?"

[0441] Input: Missing ingredient information

[0442] Output: Displayed purchase suggestions

[0443] User Actions

[0444] The user checks the list of ingredients they are short of and clicks on a link to an online supermarket to purchase the ingredients they are short of.

[0445] For example, when you click, you are redirected to an online supermarket website, where the missing ingredients are automatically added to your cart.

[0446] Input: Purchase suggestion for ingredients that are in short supply

[0447] Output: The purchase is completed and the required ingredients are added back to inventory.

[0448] (Application example 2)

[0449] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0450] Conventional refrigerator management systems have technologies that identify ingredients in a refrigerator and suggest menus based on that information, but they are unable to suggest optimal menus because they are unable to make suggestions based on the user's emotions or physical condition. In particular, in busy daily lives and under stressful circumstances, menu suggestions based on the user's emotions are more desirable. Therefore, the present invention aims to provide a system that recognizes the user's emotions and suggests optimal menus based on emotional information.

[0451] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0452] In this invention, the server includes a photographing device that photographs ingredients in the refrigerator, an identification means that analyzes image data photographed by the photographing device to identify ingredients, a suggestion means that suggests menus that can be made based on the ingredient information identified by the identification means, a purchase suggestion means that identifies ingredients that are in short supply based on the menu suggested by the suggestion means and suggests their purchase, an emotion recognition means that recognizes the user's emotion, and an emotion-based suggestion means that suggests menus based on the emotion information recognized by the emotion recognition means. This makes it possible to suggest optimal menus according to the user's emotions.

[0453] The "photography device" is a device used to take pictures of ingredients in the refrigerator.

[0454] The "identification means" is a technology that analyzes image data acquired by the imaging device and identifies ingredients.

[0455] The "proposing means" is a mechanism for providing the user with menus that can be prepared based on the ingredient information identified by the identifying means.

[0456] The "purchase suggestion means" is a mechanism for checking ingredients required for the menu suggested by the suggestion means and prompting the user to purchase any ingredients that are missing.

[0457] "Emotion recognition means" is a technology that analyzes the user's facial expressions, tone of voice, etc. to identify the user's emotions.

[0458] The "emotion-based suggestion means" is a mechanism for providing the user with an optimal menu based on the user's emotional information identified by the emotion recognition means.

[0459] The system embodying this invention efficiently utilizes ingredients in the refrigerator, proposes menus based on the user's emotions, and assists in purchasing ingredients. The components that make up this system and their specific operations are described below.

[0460] Main components of the system

[0461] 1. Imaging equipment

[0462] The camera is installed inside the refrigerator and takes pictures of the food periodically after the refrigerator door is closed. The camera can also be a smartphone camera.

[0463] 2. Identification Method

[0464] The server receives the image data sent from the camera and identifies the ingredients using an image recognition algorithm (e.g., TensorFlow). The identified ingredient information is recorded in a database (e.g., Firebase Firestore).

[0465] 3. Proposal method

[0466] The server generates a menu that can be created using a generative AI model (e.g., GPT-3) based on the identified ingredients. For example, if the identified ingredients are "tomato," "cheese," "lettuce," and "bread," the following prompt sentence can be used:

[0467] Please suggest a menu that can be made if tomatoes, cheese, lettuce, and bread are available.

[0468] The generated menu options are sent to the device (smartphone or tablet).

[0469] 4. Emotion recognition means

[0470] The server uses the smartphone's camera and microphone to record the user's facial expressions and tone of voice, and recognizes the user's emotions using an emotion recognition engine (e.g., Affectiva SDK). The results of emotion recognition are fed back to the proposal method.

[0471] 5. Emotion-based suggestion tools

[0472] Based on the emotion information, the server adjusts the menu candidates generated by the suggestion tool and generates a new menu that suits the user's state. The server also generates prompts using an AI model:

[0473] If you have tomato, cheese, lettuce, and bread, suggest a menu recommendation for a situation where the user is tired.

[0474] For example, if the system detects that the user is tired, it will suggest a quick and easy recipe such as a tomato and cheese sandwich.

[0475] 6. Purchase suggestion method

[0476] Based on the menu proposed by the suggestion means, the server generates a list of necessary ingredients. The identification means compares this with ingredient inventory information and extracts any missing ingredients. The missing ingredients are then suggested to the user so that they can be purchased through a delivery service (e.g., Uber Eats API).

[0477] Specific examples

[0478] As a concrete example of using this system, let's consider the case where a housewife takes a picture of the inside of her refrigerator using her smartphone. The camera identifies "tomato," "cheese," "lettuce," and "bread," and the server uses this information to suggest a menu. When the user confirms the suggested menu, the emotion recognition means analyzes the user's facial expression and tone of voice and determines that the user is tired. In this case, the server generates and suggests a menu that is easy to prepare in a short time.

[0479] For example, if a user selects a "tomato and cheese sandwich," the server double-checks the ingredients for that menu and finds that "mayonnaise" is missing. The server then provides a link to purchase "mayonnaise" from a delivery service.

[0480] This makes it possible to suggest appropriate menus based on the user's emotions and to suggest purchasing ingredients that are in short supply, thereby reducing household food costs and the effort required for cooking.

[0481] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0482] Step 1:

[0483] Photographing ingredients using a photography device

[0484] When the user closes the refrigerator door, the camera (smartphone camera) automatically takes a picture of the inside of the refrigerator.

[0485] Input: Image of the inside of the refrigerator

[0486] Output: Photographed food image data

[0487] Step 2:

[0488] Image data analysis and food ingredient identification

[0489] The server receives the image data sent from the camera and identifies the ingredients using an image recognition algorithm (e.g., TensorFlow).

[0490] Input: Photographed food image data

[0491] Data processing / calculation: Identifying ingredients using image recognition models

[0492] Output: List of identified ingredients

[0493] Step 3:

[0494] Storage of identified ingredient information

[0495] The server stores the identified ingredient information in a database (e.g., Firebase Firestore).

[0496] Input: Identified ingredient list

[0497] Data processing / calculation: Writing data to the database

[0498] Output: Saved ingredient information

[0499] Step 4:

[0500] User Emotion Recognition

[0501] The device's camera and microphone are used to capture the user's facial expressions and voice, and the server uses an emotion recognition engine (e.g., Affectiva SDK) to analyze the user's emotions.

[0502] Input: User facial and voice data

[0503] Data processing / calculation: Emotion analysis using emotion recognition models

[0504] Output: User's emotional information

[0505] Step 5:

[0506] Generate menu suggestions

[0507] Based on the identified ingredient information and emotional information, the server uses a generative AI model (e.g., GPT-3) to generate prompts and suggest appropriate menu items.

[0508] Input: Identified ingredients list, user's emotional information

[0509] Data processing / calculation: Menu generation using generative AI models

[0510] Output: Suggested menu options

[0511] Step 6:

[0512] View Menu Suggestions

[0513] The application on the terminal receives the menu information sent from the server and displays it to the user.

[0514] Input: Suggested menu options

[0515] Data processing / calculation: Displaying menu information to the user

[0516] Output: Menu suggestions displayed on the device

[0517] Step 7:

[0518] Check for missing ingredients

[0519] Based on the menu selected by the user, the server generates a list of ingredients needed and compares it with the ingredient inventory information in the database. If there are any ingredients that are missing, it extracts them.

[0520] Input: Selected menu, identified ingredient list

[0521] Data processing / calculation: Creating a list of necessary ingredients, extracting ingredients that are lacking

[0522] Output: List of missing ingredients

[0523] Step 8:

[0524] Generate and display purchase offers

[0525] The server creates a purchasing suggestion based on the missing ingredients and displays it to the user through the device application. The user can then click a link on the device to purchase the missing ingredients through a delivery service (e.g., Uber Eats API).

[0526] Input: List of ingredients in short supply

[0527] Data processing / calculation: generating purchase proposals

[0528] Output: Purchase suggestions displayed on the device, links to delivery services

[0529] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0530] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0531] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0532] [Second embodiment]

[0533] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0534] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0535] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0536] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0537] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0538] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0539] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0540] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0541] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0542] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0543] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0544] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0545] The present invention is a system for efficiently utilizing food ingredients in a refrigerator, saving household food costs and reducing the effort required for cooking. The system includes the following main elements:

[0546] System configuration

[0547] 1. Fridge Camera

[0548] This camera is installed inside the refrigerator and takes pictures of food ingredients. The camera takes pictures periodically after the refrigerator door is closed.

[0549] 2. Server

[0550] It includes means for receiving image data transmitted from the camera and executing an image recognition algorithm to identify ingredients from the data.

[0551] 3. Terminal

[0552] A device used by a user, such as a smartphone or tablet, on which the application is installed.

[0553] Program processing

[0554] 1. Photographing food in the refrigerator

[0555] How the refrigerator camera works

[0556] The refrigerator camera periodically photographs the food inside the refrigerator after the door is closed.

[0557] 2. Analysis of image data

[0558] Server Operation

[0559] The server receives the image data sent from the refrigerator camera.

[0560] An image recognition algorithm is applied to the received image data to identify the ingredients in the refrigerator.

[0561] For example, the server identifies "tomato," "cheese," "lettuce," "bread," etc.

[0562] The identified ingredient information is recorded in a database.

[0563] 3. Create a composable menu

[0564] Server Operation

[0565] The server generates menu options that can be created using a generative AI model based on the identified ingredient information.

[0566] For example, using "tomato," "cheese," "lettuce," and "bread," it suggests "tomato and cheese sandwich" and "lettuce and tomato salad."

[0567] The generated menu candidates are sent to the terminal.

[0568] 4. Menu suggestions

[0569] Device behavior

[0570] The application on the device receives and displays the menu information sent from the server.

[0571] The user selects the desired menu from the proposed menus.

[0572] 5. Check for missing ingredients and suggest purchasing them

[0573] Server Operation

[0574] The server generates a list of ingredients required based on the menu selected by the user.

[0575] It compares the information on food inventory in the refrigerator and extracts any ingredients that are missing.

[0576] For example, if you select "tomato and cheese sandwich," it will identify that "tomato," "cheese," and "bread" are present in the refrigerator, but "mayonnaise" is missing.

[0577] A list of missing ingredients and a purchasing suggestion including a link to the online supermarket's purchasing page are sent to the device.

[0578] 6. Displaying and executing purchase offers

[0579] Device behavior

[0580] The app on the device receives and displays the information about the missing ingredients and purchasing suggestions sent from the server.

[0581] User Actions

[0582] The user checks the list of ingredients they are short of and clicks on a link to an online supermarket to purchase the ingredients they are short of.

[0583] Specific examples

[0584] Using this system, let's consider a case where a refrigerator contains "tomatoes," "cheese," "lettuce," and "bread." The camera takes a picture of these ingredients, and the server analyzes the image data to identify them. Based on the identified ingredients, the server generates menu items such as "tomato and cheese sandwich" and "lettuce and tomato salad," and sends this to the device. The user opens the app, checks the suggested menu items, and selects the desired "tomato and cheese sandwich." Based on this menu, the server identifies the missing "mayonnaise" and suggests a link to an online supermarket to the user. The user can click the link to purchase "mayonnaise."

[0585] In this way, the present invention makes efficient use of ingredients in the refrigerator and suggests suitable menus, thereby saving household food costs and reducing the effort required for cooking.

[0586] The processing flow will be explained below.

[0587] Step 1:

[0588] The refrigerator camera takes pictures of the inside of the refrigerator at regular intervals or after the refrigerator door is closed. The camera generates high-resolution image data and transmits the data to a server.

[0589] Step 2:

[0590] The server receives the image data sent from the refrigerator camera and stores it in a temporary storage area for saving the image data.

[0591] Step 3:

[0592] The server applies image recognition algorithms to identify ingredients in the image. For example, the server uses image recognition software to identify ingredients such as "tomato," "cheese," "lettuce," and "bread."

[0593] Step 4:

[0594] The server records the identified ingredient information in a database, which includes detailed information such as the type of ingredient, the quantity, and the date and time of identification.

[0595] Step 5:

[0596] The server uses a generative AI model to generate possible menu items based on the ingredient information stored in the database. For example, using "tomato," "cheese," "lettuce," and "bread," it can suggest "tomato and cheese sandwich" or "lettuce and tomato salad."

[0597] Step 6:

[0598] The server sends the generated menu options and necessary ingredient information to the terminal, which receives this information and prepares to display it to the user.

[0599] Step 7:

[0600] The app on the device receives the menu information sent from the server and displays it to the user. The user opens the app and checks the proposed menu.

[0601] Step 8:

[0602] The user selects the desired menu from the proposed menus, and the user's selection information is sent from the terminal to the server.

[0603] Step 9:

[0604] The server generates a list of ingredients required for the selected menu based on the user's selection, and compares it with ingredient inventory information to extract any ingredients that are in short supply.

[0605] Step 10:

[0606] The server compiles a list of ingredients that are missing, generates a link to the online supermarket's purchasing page, and sends this information to the device.

[0607] Step 11:

[0608] The app on the device receives the information about the missing ingredients and the purchase suggestions sent from the server and displays them to the user.

[0609] Step 12:

[0610] The user checks the list of ingredients they are short of and clicks on a link to an online supermarket, where they can purchase the ingredients they are short of.

[0611] Example 1

[0612] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0613] Modern households are seeking to effectively utilize the ingredients in their refrigerators to save on food costs and reduce the effort required for cooking. However, current systems lack the functionality to automatically recognize ingredients in the refrigerator and suggest menus based on that information, forcing users to manually manage ingredients. Furthermore, their functionality to support the purchase of ingredients that are in short supply is limited. To address these issues, the present invention aims to provide a series of processes for automatically identifying ingredients in the refrigerator, generating menus based on that information, and purchasing ingredients that are in short supply.

[0614] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0615] In this invention, the server includes a camera for taking pictures of ingredients in the refrigerator, a means for receiving and saving image data taken by the camera, a means for identifying ingredients by applying an image recognition algorithm to the received and saved image data, a means for recording information on ingredients identified by the camera in a database, a means for generating menu candidates that can be created using a generative AI model based on the identified ingredient information, a means for sending the generated menu candidates to a user terminal, a means for displaying the generated menu candidates on the user terminal, a means for generating a list of necessary ingredients based on the menu selected by the user, a means for comparing the list of necessary ingredients with information on ingredient inventory in the refrigerator and extracting missing ingredients, and a means for sending the list of missing ingredients and a link to a purchasing page to the user terminal. This allows the system to automatically manage ingredients in the refrigerator, suggest suitable menus to the user, and quickly and easily purchase missing ingredients.

[0616] The "photography device" is a device for photographing the environment inside the refrigerator, and mainly includes a camera.

[0617] "Image data" is digital data obtained by photographing the food items in the refrigerator.

[0618] The "receiving and saving means" refers to a means for receiving image data transmitted from the imaging device and saving it temporarily or permanently, and includes server storage and the like.

[0619] An "image recognition algorithm" is a computational method for analyzing image data and identifying specific objects or features, including approaches that utilize artificial intelligence techniques.

[0620] The "identification means" is a means for applying an image recognition algorithm to identify ingredients in the refrigerator and obtain information about them.

[0621] A "database" is a system for systematically storing and managing information on identified ingredients, etc., and includes SQL databases, NoSQL databases, etc.

[0622] A "generative AI model" is an artificial intelligence model that generates new information and suggestions based on input data, and is based on machine learning and deep learning.

[0623] "Menu candidates" are options for dishes and recipes suggested by the generative AI model based on identified ingredient information.

[0624] A "user terminal" is a device used by a user, including a smartphone, tablet, computer, etc.

[0625] The "list of necessary ingredients" is a list of all ingredients required to prepare the menu selected by the user.

[0626] "Inventory information" refers to information about all ingredients currently in the refrigerator, including details such as the quantity and type.

[0627] "Missing ingredients" refers to ingredients that are needed to prepare the selected menu but are not currently present in the refrigerator.

[0628] The "Purchase Page Link" is a hyperlink to a web page that allows the user to purchase the ingredients they are running low on online.

[0629] This invention is a system for efficiently utilizing ingredients in a refrigerator, saving household food costs and reducing the effort required for cooking. The system includes a camera for photographing ingredients in the refrigerator, a server for receiving and storing image data, a means for identifying ingredients using an image recognition algorithm based on the image data, a database for recording information on identified ingredients, a means for generating menu options using a generative AI model, and a means for extracting ingredients that are in short supply and suggesting their purchase.

[0630] First, a refrigerator camera is used to take pictures of the food inside the refrigerator. This camera automatically starts up after the refrigerator door is closed and periodically takes pictures of the food inside. For example, it takes a total of three images three seconds after the refrigerator door is closed. The captured image data is sent to a server via Wi-Fi.

[0631] Next, the server receives this image data and temporarily stores it in a database. Image recognition algorithms using TensorFlow, OpenCV, etc. are applied to the stored image data. This algorithm identifies the ingredients in the refrigerator, and ingredient information such as "tomato," "cheese," "lettuce," and "bread" is recorded in the database.

[0632] The server then uses this identified ingredient information to generate menu suggestions that can be created using a generative AI model (e.g., GPT-3). An example prompt used here is:

[0633] "There are tomatoes, cheese, lettuce, and bread in the refrigerator. Can you suggest a recipe using these?"

[0634] This generates menu options such as "tomato and cheese sandwich" and "lettuce and tomato salad," which are then sent to the user's device.

[0635] An application installed on the user's device receives this menu information and displays it on the screen. The user can then select the desired menu from the suggested menu. For example, if the user selects a "tomato and cheese sandwich," the server generates a list of ingredients based on this menu and compares it with the inventory information of ingredients in the refrigerator.

[0636] At this stage, the missing ingredients are identified, and it is determined that "mayonnaise," for example, is in short supply. The server calls the online supermarket's API, obtains a purchase link for "mayonnaise," and sends it to the user's device. The user can click this link to easily purchase the missing ingredients.

[0637] As described above, the present invention can improve user convenience by automatically managing ingredients in the refrigerator, suggesting suitable menus, and suggesting the purchase of ingredients that are in short supply, thereby saving food costs and reducing the effort required for cooking.

[0638] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0639] Step 1:

[0640] How the refrigerator camera works

[0641] After the refrigerator door is closed, the refrigerator camera automatically activates and takes a picture of the food inside.

[0642] input

[0643] A signal that the refrigerator door is closing.

[0644] output

[0645] High resolution image data of the inside of a refrigerator.

[0646] Specific actions

[0647] The camera takes three images three seconds after the door is closed and saves them as digital image files in JPEG format.

[0648] Step 2:

[0649] Receiving image data from the server

[0650] The image data captured by the refrigerator camera is sent via Wi-Fi to a server, which receives and stores it.

[0651] input

[0652] Image data sent from the refrigerator camera.

[0653] output

[0654] Saving received image data.

[0655] Specific actions

[0656] The server receives image files from the camera via the FTP server function and stores them in a specific directory.

[0657] Step 3:

[0658] Server image analysis

[0659] The server applies an image recognition algorithm (e.g., TensorFlow's Inception model) to the received image data to identify the ingredients.

[0660] input

[0661] Saved image data.

[0662] output

[0663] Identified ingredient information.

[0664] Specific actions

[0665] The server uses the TensorFlow library to perform object detection on the stored image data, identifying ingredients such as "tomato," "cheese," "lettuce," and "bread." The results are stored in a database in JSON format.

[0666] Step 4:

[0667] Server menu generation

[0668] Based on the identified ingredient information, the server generates menu options that can be created using a generative AI model (e.g., GPT-3).

[0669] input

[0670] Identified ingredient information.

[0671] output

[0672] The generated menu suggestions.

[0673] Specific actions

[0674] The server inputs the following prompt into the generative AI model:

[0675] "There are tomatoes, cheese, lettuce, and bread in the refrigerator. Can you suggest a recipe using these?"

[0676] The generated results are "tomato and cheese sandwich" and "lettuce and tomato salad", which are saved in JSON format.

[0677] Step 5:

[0678] Sending and displaying menu information

[0679] The server sends the generated menu suggestions to the user's terminal, which displays them.

[0680] input

[0681] The generated menu suggestions.

[0682] output

[0683] Displaying menu information on the user's terminal.

[0684] Specific actions

[0685] The server pushes the JSON data of the menu suggestions to the user's device, which receives and displays it in the device's application. The user can then select the desired menu from the suggested menus.

[0686] Step 6:

[0687] Identifying missing ingredients

[0688] Based on the menu selected by the user, the server generates a list of ingredients needed, compares it with the inventory information in the refrigerator, and extracts any ingredients that are missing.

[0689] input

[0690] The menu selected by the user.

[0691] output

[0692] Missing ingredient information.

[0693] Specific actions

[0694] The server does the following:

[0695] 1. Refer to the recipe database for "tomato and cheese sandwich" and generate a list of ingredients needed.

[0696] 2. Compare with the food information in the refrigerator and determine that there is a shortage of "mayonnaise."

[0697] Step 7:

[0698] Submit a Purchase Offer

[0699] The server sends a list of missing ingredients and a link to the purchasing page to the user's terminal.

[0700] input

[0701] Missing ingredient information.

[0702] output

[0703] Sending purchase suggestion information to the user terminal.

[0704] Specific actions

[0705] The server calls the online supermarket's API, obtains a purchase link for "mayonnaise," and sends this along with a list of ingredients that are in short supply to the user's terminal.

[0706] Step 8:

[0707] View purchase suggestions

[0708] The device's app receives the information about the missing ingredients and the purchase suggestions sent from the server and displays them on the screen.

[0709] input

[0710] Purchase offer information sent from the server.

[0711] output

[0712] Display of purchase suggestion information on user terminal.

[0713] Specific actions

[0714] The app displays a list of missing ingredients in a pop-up window. For example, "You're running low on mayonnaise. Here's a link to buy it." The user can click the link to buy the missing ingredients.

[0715] (Application example 1)

[0716] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0717] In modern households, it is important to efficiently manage food in the refrigerator and optimize consumption. However, currently, it is time-consuming to identify the presence of ingredients and identify what is missing, and it is difficult to suggest appropriate menus or replenish missing ingredients. Furthermore, particularly in the field of food delivery, there is no technology that automatically replenishes or suggests ingredients when a specific ingredient is missing. Therefore, there is a need for a system that can efficiently manage food in the refrigerator and even automatically replenish missing ingredients.

[0718] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0719] In this invention, the server includes an imaging device that photographs ingredients in the refrigerator, identification means that analyzes image data photographed by the imaging device to identify ingredients, suggestion means that proposes menus that can be created based on ingredient information identified by the identification means, purchase suggestion means that identifies ingredients that are lacking based on the menus proposed by the suggestion means and suggests their purchase, an auxiliary device that is installed in an automatic delivery machine associated with the refrigerator and that scans ingredients in the refrigerator with an imaging device, and replenishment means that the automatic delivery machine automatically replenishes ingredients that are lacking. This makes it possible to efficiently manage ingredients in the refrigerator and automatically replenish ingredients that are lacking.

[0720] An "imaging device that photographs food ingredients in a refrigerator" is a device that is installed inside a refrigerator and that captures image data of food ingredients in the refrigerator.

[0721] The "identification means" is a device or program that has the function of analyzing image data captured by the imaging device and identifying each food ingredient in the refrigerator.

[0722] The "suggestion means" is a device or program that has the function of generating and suggesting menus that can be made using ingredients based on the ingredient information identified by the identification means.

[0723] The "purchase suggestion means" is a device or program having the function of identifying ingredients that are in short supply from the ingredient information in the refrigerator based on the menu generated by the suggestion means and suggesting to the user that they be purchased.

[0724] A "delivery automaton" is a mechanical device, such as an automatically mobile robot, used to assist in the management of refrigerators.

[0725] The "auxiliary device" is an imaging device that is installed in the automated delivery machine and is used to scan ingredients in the refrigerator.

[0726] The "replenishment means" is a device or program that has the function of identifying a shortage of ingredients and automatically replenishing the ingredients when necessary.

[0727] The present invention is a system for efficiently managing ingredients in a refrigerator and automatically replenishing ingredients when they are in short supply. Detailed embodiments of the present invention will be described below.

[0728] The present invention includes an imaging device that photographs ingredients in a refrigerator, a server, a user terminal, and an automated delivery machine. The imaging device is installed inside the refrigerator and periodically photographs ingredients and sends the image data to the server. The server analyzes the received image data and has an identification means for identifying the ingredients.

[0729] The server generates a menu that can be prepared using the suggestion means based on the identified ingredient information and transmits it to the user's terminal. The user can check and select the suggested menu through the terminal. Based on the selected menu, the server's purchase suggestion means identifies ingredients that are in short supply and suggests their purchase to the user.

[0730] Furthermore, the delivery machine is equipped with an auxiliary device that can scan the ingredients in the refrigerator. As a means of replenishment, the delivery machine can automatically replenish any ingredients that are missing.

[0731] The system operates as follows.

[0732] 1. Scan food in the refrigerator

[0733] An image capture device installed in the delivery machine periodically scans the food items in the refrigerator and captures image data, for example, after the refrigerator door is closed.

[0734] 2. Analysis of image data

[0735] The server receives the image data sent from the imaging device and identifies the ingredients using image analysis software (e.g., TensorFlow or OpenCV). The identified ingredient information is recorded in a database.

[0736] 3. Creating a Menu

[0737] The server uses a generative AI model to generate menu options based on the identified ingredients. For example, if "tomato," "cheese," "lettuce," and "bread" are identified, the server will suggest options such as "tomato and cheese sandwich" or "lettuce and tomato salad."

[0738] 4. Identifying missing ingredients and suggesting purchases

[0739] The server generates a list of ingredients based on the user's menu selection and compares it with the refrigerator's inventory data. If any ingredients are missing, a purchase suggestion is sent to the user's device.

[0740] 5. Automatic replenishment function

[0741] The automated delivery machine follows instructions from the server, automatically orders missing ingredients from online supermarkets, etc., and replenishes the refrigerator if necessary.

[0742] As a concrete example, consider a situation where there are "tomatoes," "cheese," "lettuce," and "bread" in a refrigerator. The image capture device of the automated delivery machine takes a picture of these items, and the server analyzes and identifies the image data. Based on the identified ingredient information, the server generates and suggests menu items such as "tomato and cheese sandwich" or "lettuce and tomato salad." If the user selects "tomato and cheese sandwich," the server identifies the missing "mayonnaise" and suggests a link to purchase it from an online supermarket to the user. Furthermore, the automated delivery machine automatically purchases mayonnaise and replenishes the refrigerator.

[0743] Example prompt for generative AI model:

[0744] We will build an AI system that recognizes ingredients in the user's refrigerator and suggests ingredients that are missing. Please identify ingredients from the input image, compare them with the list of necessary ingredients, extract the missing ingredients, and generate a purchase link.

[0745] As described above, the present invention can improve the efficiency of food ingredient management in a refrigerator and realize automatic replenishment of ingredients that are running low.

[0746] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0747] Step 1:

[0748] Scan food in the refrigerator

[0749] Input: Image data from an imaging device (camera) installed inside the refrigerator

[0750] Specific operation: After the refrigerator door is closed, the image capture device takes an image of the inside of the refrigerator. The captured image data is sent to the server.

[0751] Output: Image data of ingredients in the refrigerator

[0752] Step 2:

[0753] Receiving and analyzing image data

[0754] Input: Image data of the inside of the refrigerator sent from the imaging device

[0755] Specific operation: The server receives the image data and identifies the ingredients using image analysis software (e.g., TensorFlow or OpenCV). The identified ingredient information is then recorded in a database.

[0756] Output: Identification information of ingredients (e.g., "tomato," "cheese," "lettuce," "bread")

[0757] Step 3:

[0758] Generate Menu

[0759] Input: Identification information of ingredients recorded on the server

[0760] Specific operation: The server uses the generative AI model to generate menu options based on the identified ingredients. The generated menu options are sent to the user's device.

[0761] Output: Possible menu items (e.g., "Tomato and cheese sandwich," "Lettuce and tomato salad")

[0762] Step 4:

[0763] Menu suggestions and selections

[0764] Input: Menu suggestions sent from the server to the user's device

[0765] Specific operation: The user's device displays the menu options received from the server, and the user selects the desired menu.

[0766] Output: The menu selected by the user

[0767] Step 5:

[0768] Identifying missing ingredients and suggesting purchases

[0769] Input: Menu selected by the user and ingredient information in the database

[0770] Specific operation: Based on the selected menu, the server generates a list of necessary ingredients and checks it against the refrigerator inventory in the database. If any ingredients are missing, it sends a purchase suggestion to the user's device.

[0771] Output: List of ingredients that are in short supply and purchase suggestions (e.g., "mayonnaise" purchase link)

[0772] Step 6:

[0773] Purchasing and Auto-Refill

[0774] Input: Purchase offers and delivery machine replenishment orders sent from the server to the user

[0775] Specific operation: When the user checks the purchase suggestion and clicks on the link to the online supermarket to purchase the missing ingredients, the delivery machine automatically replenishes the missing ingredients based on instructions from the server. The delivery machine is equipped with an auxiliary device that places the purchased ingredients in the refrigerator.

[0776] Output: Automatically replenished food in the refrigerator

[0777] Example prompt sentence:

[0778] We will build an AI system that recognizes ingredients in the user's refrigerator and suggests ingredients that are missing. Please identify ingredients from the input image, compare them with the list of necessary ingredients, extract the missing ingredients, and generate a purchase link.

[0779] This allows users to efficiently manage ingredients in their refrigerators and automatically replenish ingredients when they run out.

[0780] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0781] The present invention combines a system for efficiently utilizing ingredients in the refrigerator, saving household food costs and reducing cooking time, with an emotion engine that recognizes the user's emotions. The system includes the following main components:

[0782] System configuration

[0783] 1. Fridge Camera

[0784] This camera is installed inside the refrigerator and takes pictures of food ingredients. The camera takes pictures periodically after the refrigerator door is closed.

[0785] 2. Server

[0786] It includes means for receiving image data transmitted from the camera and executing an image recognition algorithm to identify ingredients from the data.

[0787] It is equipped with an emotion engine that recognizes the user's emotions and suggests appropriate menus based on those emotions.

[0788] 3. Terminal

[0789] A device used by a user, such as a smartphone or tablet, on which the application is installed.

[0790] Program processing

[0791] 1. Photographing food in the refrigerator

[0792] How the refrigerator camera works

[0793] The refrigerator camera periodically photographs the food inside the refrigerator after the door is closed.

[0794] 2. Analysis of image data

[0795] Server Operation

[0796] The server receives the image data sent from the refrigerator camera.

[0797] An image recognition algorithm is applied to the received image data to identify the ingredients in the refrigerator.

[0798] For example, the server identifies "tomato," "cheese," "lettuce," "bread," etc.

[0799] The identified ingredient information is recorded in a database.

[0800] 3. Create a composable menu

[0801] Server Operation

[0802] The server generates menu options that can be created using a generative AI model based on the identified ingredient information.

[0803] For example, using "tomato," "cheese," "lettuce," and "bread," it suggests "tomato and cheese sandwich" and "lettuce and tomato salad."

[0804] 4. Emotion Recognition by Emotion Engine

[0805] Server Operation

[0806] The server uses an emotion engine to recognize the user's emotions by analyzing the user's facial expressions and tone of voice.

[0807] The emotion engine also learns the user's past selection history and behavioral patterns to predict emotions.

[0808] 5. Menu suggestions based on emotions

[0809] Server Operation

[0810] The server proposes a menu according to the user's emotions based on the user's emotion information obtained from the emotion engine.

[0811] For example, if the server detects that the user is tired, it will suggest a menu that is simple and quick to prepare.

[0812] 6. Menu suggestions

[0813] Device behavior

[0814] The application on the device receives the menu information sent from the server and displays it to the user.

[0815] The user opens the app and sees the suggested menu.

[0816] 7. Check for missing ingredients and suggest purchasing them

[0817] Server Operation

[0818] The server generates a list of ingredients required based on the menu selected by the user.

[0819] The system compares the information with food stock information and extracts any ingredients that are in short supply.

[0820] 8. Displaying and executing purchase offers

[0821] Device behavior

[0822] The app on the device receives the information about the missing ingredients and the purchase suggestions sent from the server and displays them to the user.

[0823] User Actions

[0824] The user checks the list of ingredients they are short of and clicks on a link to an online supermarket to purchase the ingredients they are short of.

[0825] Specific examples

[0826] Using this system, let's assume that there are "tomatoes," "cheese," "lettuce," and "bread" in the refrigerator. The camera takes a picture of these ingredients, and the server analyzes the image data to identify them. Based on the identified ingredients, the server generates a menu such as "tomato and cheese sandwich" or "lettuce and tomato salad" and sends it to the terminal.

[0827] When a user opens the app and checks the suggested menu, the emotion engine analyzes the user's facial expressions and tone of voice. If it determines that the user is tired, it suggests a "tomato and cheese sandwich," which is easy and quick to make. When the user selects this suggestion, the server checks the ingredients needed for the "tomato and cheese sandwich" and identifies the missing "mayonnaise." The server then suggests the user purchase "mayonnaise" from an online supermarket, and the user can click a link to purchase it.

[0828] In this way, by combining emotion engines, it becomes possible to suggest menus that correspond to the user's emotions, enabling more individually tailored dish suggestions to be realized.

[0829] The processing flow will be explained below.

[0830] Step 1:

[0831] The refrigerator camera takes pictures of the inside of the refrigerator at regular intervals or after the refrigerator door is closed. The camera generates high-resolution image data and transmits the data to a server.

[0832] Step 2:

[0833] The server receives the image data sent from the refrigerator camera and stores it in a temporary storage area for saving the image data.

[0834] Step 3:

[0835] The server applies image recognition algorithms to identify ingredients in the image. For example, the server uses image recognition software to identify ingredients such as "tomato," "cheese," "lettuce," and "bread."

[0836] Step 4:

[0837] The server records the identified ingredient information in a database, which includes detailed information such as the type of ingredient, the quantity, and the date and time of identification.

[0838] Step 5:

[0839] The server uses a generative AI model to generate possible menu items based on the ingredient information stored in the database. For example, using "tomato," "cheese," "lettuce," and "bread," it can suggest "tomato and cheese sandwich" or "lettuce and tomato salad."

[0840] Step 6:

[0841] The server sends the generated menu options and necessary ingredient information to the terminal, which receives this information and prepares to display it to the user.

[0842] Step 7:

[0843] The app on the device receives the menu information sent from the server and displays it to the user. The user opens the app and checks the proposed menu.

[0844] Step 8:

[0845] The app on the device recognizes the user's facial expressions and tone of voice and sends that data to the emotion engine, which then analyzes facial expressions and voice to recognize the user's emotional state.

[0846] Step 9:

[0847] The emotion engine on the server recognizes the user's emotions based on the analysis results and adjusts the menu suggestions based on the user's emotional state. For example, if the system recognizes that the user is tired, it will prioritize suggestions of simple and quick meals.

[0848] Step 10:

[0849] The user selects the desired menu from the proposed menus, and the user's selection information is sent from the terminal to the server.

[0850] Step 11:

[0851] The server generates a list of ingredients required for the selected menu based on the user's selection, and compares it with ingredient inventory information to extract any ingredients that are in short supply.

[0852] Step 12:

[0853] The server compiles a list of ingredients that are missing, generates a link to the online supermarket's purchasing page, and sends this information to the device.

[0854] Step 13:

[0855] The app on the device receives the information about the missing ingredients and the purchase suggestions sent from the server and displays them to the user.

[0856] Step 14:

[0857] The user checks the list of ingredients they are short of and clicks on a link to an online supermarket, where they can purchase the ingredients they are short of.

[0858] Example 2

[0859] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0860] In modern households, efficiently utilizing the ingredients in the refrigerator to reduce food costs and cooking time is an important issue. However, it is difficult for users to properly manage the ingredients in their refrigerator, resulting in frequent waste and unused ingredients. In addition, menu suggestions are not tailored to the user's condition or mood, making it difficult for users to maintain their motivation to cook. Furthermore, if necessary ingredients are in short supply, users have to go shopping separately, which is a hassle.

[0861] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0862] In this invention, the server includes a camera that takes pictures of ingredients in the refrigerator, an image analysis unit that analyzes image data taken by the camera and identifies ingredients, a suggestion unit that suggests menus that can be made based on the ingredient information identified by the image analysis unit, a purchase suggestion unit that identifies ingredients that are lacking based on the menus suggested by the suggestion unit and suggests the purchase of those ingredients, and an emotion recognition unit that recognizes the user's emotions and suggests appropriate menus accordingly. This makes it possible to efficiently use ingredients in the refrigerator and reduce household food costs, and furthermore, by suggesting menus that correspond to the user's emotions, it is possible to improve motivation to cook and reduce the effort required.

[0863] "Photography device" refers to a device installed to photograph food items in a refrigerator.

[0864] "Image analysis means" refers to algorithms or software that analyzes image data acquired by the imaging device and identifies ingredients.

[0865] The "suggestion means" refers to a system for suggesting menus that can be prepared based on the ingredient information identified by the image analysis means.

[0866] The "purchase suggestion means" refers to a system for determining ingredients that are in short supply based on the menu suggested by the suggestion means and suggesting to the user to purchase those ingredients.

[0867] "Emotion recognition means" refers to algorithms or devices that analyze a user's facial expressions and tone of voice to recognize their emotions.

[0868] A "refrigerator" refers to a cooling device used in homes and businesses to preserve food.

[0869] "Ingredients" refers to food stored in the refrigerator for use in cooking.

[0870] "Image data" refers to digital data of photographs and videos captured by a photographing device.

[0871] "Ingredient information" refers to data relating to the type and quantity of ingredients identified by the image analysis means.

[0872] "Menu" refers to a list of dishes that can be made using ingredients.

[0873] The present invention combines a system for efficiently utilizing ingredients in a refrigerator, saving household food costs and reducing the effort required for cooking, with an emotion engine that recognizes the user's emotions. This system includes a photographing device for photographing ingredients in the refrigerator, image analysis means for analyzing image data and identifying ingredients, suggestion means for suggesting menus based on identified ingredient information, purchase suggestion means for identifying ingredients that are in short supply and suggesting their purchase, and emotion recognition means for recognizing the user's emotions and suggesting appropriate menus.

[0874] Specific hardware and software:

[0875] 1. Camera: A camera installed inside the refrigerator takes pictures periodically after the refrigerator door is closed. For example, a commercially available webcam or a special-purpose camera can be used.

[0876] 2. Image analysis means: Uses image recognition algorithms installed on the server. Specifically, open source image processing libraries and machine learning models (e.g., TensorFlow and OpenCV) can be used.

[0877] 3. Proposal method: Menu suggestions are made using a generative AI model. Specifically, a generative AI model that performs natural language processing (e.g., GPT-3) is used to generate a menu based on ingredient information.

[0878] 4. Purchase suggestion means: Using a database management system installed on the server, the missing ingredients are identified based on the identified ingredient information and the proposed menu, and a purchase suggestion is made.

[0879] 5. Emotion Recognition: An emotion engine is used to analyze the user's facial expressions and tone of voice. For example, a deep learning model (e.g., FaceNet) can be used for facial expression analysis, and a speech recognition engine (e.g., Google Cloud Speech-to-Text) can be used for voice analysis.

[0880] Operational flow and data calculation:

[0881] The server receives image data sent from the refrigerator camera and performs image analysis to identify ingredients from the data. The analyzed ingredient information is recorded in a database within the server. The server then uses a generative AI model based on this ingredient information to generate possible menu options. When the user uses the application to check the proposed menu, the emotion engine analyzes the user's facial expressions and tone of voice and presents a menu that corresponds to the user's emotions. Furthermore, based on the proposed menu, the server determines which ingredients are in short supply and suggests that the user purchase them.

[0882] Example 1:

[0883] If there are "tomatoes," "cheese," "lettuce," and "bread" in the refrigerator, the camera takes a picture of these ingredients, and the server analyzes the image data to identify them. Based on the identified ingredient information, the server generates menu items such as "tomato and cheese sandwich" and "lettuce and tomato salad," and sends them to the device. When the user opens the app and checks the suggested menu items, the emotion engine analyzes the user's facial expressions and tone of voice, and if it determines that the user is tired, it suggests "tomato and cheese sandwich," which is easy to make and can be made quickly. If the user selects this suggestion, the server checks the ingredients needed for the "tomato and cheese sandwich" and identifies the missing "mayonnaise." The server then suggests the user purchase "mayonnaise" from an online supermarket, and the user can click a link to purchase it.

[0884] Example 2:

[0885] Suggest a menu that is easy and quick to make using the "chicken," "carrots," and "onions" in the refrigerator. Also, if the user is tired, prioritize items that are easy and nutritious. An example of a prompt sentence is, "Suggest a menu that is easy and quick to make using the "chicken," "carrots," and "onions" in the refrigerator. Also, if the user is tired, prioritize items that are easy and nutritious."

[0886] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0887] Step 1:

[0888] Photographing food in the refrigerator

[0889] How the refrigerator camera works

[0890] The refrigerator camera periodically takes pictures of the inside of the refrigerator after the door is closed.

[0891] For example, a refrigerator is set to take a photo every hour.

[0892] Input: Image of the inside of the refrigerator

[0893] Output: Captured image data

[0894] Step 2:

[0895] Image data analysis

[0896] Server Operation

[0897] The server receives the image data sent from the refrigerator camera.

[0898] An image recognition algorithm is applied to the received image data to identify the ingredients inside.

[0899] For example, TensorFlow or OpenCV is used as an image recognition algorithm.

[0900] Input: Photographed image data

[0901] Data processing and calculation: Apply image recognition algorithms to identify the type of food

[0902] Output: Identified ingredients (e.g., "tomato," "cheese," "lettuce," "bread," etc.)

[0903] Step 3:

[0904] Generate composable menus

[0905] Server Operation

[0906] The server generates menu options that can be created using a generative AI model based on the identified ingredient information.

[0907] Specifically, we use a generative AI model (e.g., GPT-3) that performs natural language processing.

[0908] Input: Identified ingredient information

[0909] Data processing and data calculation: Generate menu suggestions using generative AI models

[0910] Output: Generated menu options (e.g., "tomato and cheese sandwich," "lettuce and tomato salad," etc.)

[0911] Step 4:

[0912] Emotion recognition by emotion recognition means

[0913] Server Operation

[0914] The server uses an emotion recognition means to recognize the user's emotion.

[0915] A deep learning model (e.g., FaceNet) is used for facial expression analysis, and a speech recognition engine (e.g., Google Cloud Speech-to-Text) is used for voice analysis.

[0916] Input: User's facial image and voice data

[0917] Data processing and data calculation: Applying emotion recognition algorithms to analyze user emotions

[0918] Output: Analyzed user's emotional information (e.g., "Tired," "Relaxed," etc.)

[0919] Step 5:

[0920] Menu suggestions based on emotions

[0921] Server Operation

[0922] The server proposes a menu according to the user's emotion based on the emotion information obtained from the emotion recognition means.

[0923] For example, if the user is tired, the system will prioritize suggesting menus that are simple and can be prepared in a short time.

[0924] Input: Analyzed user emotion information and generated menu suggestions

[0925] Data processing and data calculation: determining menu priorities based on emotional information

[0926] Output: Menu suggestions based on emotions (e.g., "Tomato and cheese sandwich")

[0927] Step 6:

[0928] Menu suggestions

[0929] Device behavior

[0930] The application on the device receives the menu information sent from the server and displays it to the user.

[0931] The user opens the app and sees the suggested menu.

[0932] For example, a smartphone or tablet app might display menu items such as "Tomato and Cheese Sandwich" or "Lettuce and Tomato Salad."

[0933] Input: Emotion-based menu suggestions

[0934] Output: Displayed menu information

[0935] Step 7:

[0936] Check for missing ingredients and suggest purchasing

[0937] Server Operation

[0938] The server generates a list of ingredients required based on the menu selected by the user.

[0939] The generated ingredient list is compared with the inventory information in the refrigerator to extract any ingredients that are missing.

[0940] For example, it may identify that a tomato and cheese sandwich requires mayonnaise, but that it is not in stock in the refrigerator.

[0941] Input: Selected menu, refrigerator inventory information

[0942] Data processing and calculation: Generate a list of necessary ingredients and extract missing ingredients

[0943] Output: Missing ingredient information (e.g., "mayonnaise")

[0944] Step 8:

[0945] View and execute purchase offers

[0946] Device behavior

[0947] The app on the device receives the information about the missing ingredients and the purchase suggestions sent from the server and displays them to the user.

[0948] For example, a message may appear saying, "We're low on mayonnaise. Would you like to purchase some from the online supermarket?"

[0949] Input: Missing ingredient information

[0950] Output: Displayed purchase suggestions

[0951] User Actions

[0952] The user checks the list of ingredients they are short of and clicks on a link to an online supermarket to purchase the ingredients they are short of.

[0953] For example, when you click, you are redirected to an online supermarket website, where the missing ingredients are automatically added to your cart.

[0954] Input: Purchase suggestion for ingredients that are in short supply

[0955] Output: The purchase is completed and the required ingredients are added back to inventory.

[0956] (Application example 2)

[0957] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0958] Conventional refrigerator management systems have technologies that identify ingredients in a refrigerator and suggest menus based on that information, but they are unable to suggest optimal menus because they are unable to make suggestions based on the user's emotions or physical condition. In particular, in busy daily lives and under stressful circumstances, menu suggestions based on the user's emotions are more desirable. Therefore, the present invention aims to provide a system that recognizes the user's emotions and suggests optimal menus based on emotional information.

[0959] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0960] In this invention, the server includes a photographing device that photographs ingredients in the refrigerator, an identification means that analyzes image data photographed by the photographing device to identify ingredients, a suggestion means that suggests menus that can be made based on the ingredient information identified by the identification means, a purchase suggestion means that identifies ingredients that are in short supply based on the menu suggested by the suggestion means and suggests their purchase, an emotion recognition means that recognizes the user's emotion, and an emotion-based suggestion means that suggests menus based on the emotion information recognized by the emotion recognition means. This makes it possible to suggest optimal menus according to the user's emotions.

[0961] The "photography device" is a device used to take pictures of ingredients in the refrigerator.

[0962] The "identification means" is a technology that analyzes image data acquired by the imaging device and identifies ingredients.

[0963] The "proposing means" is a mechanism for providing the user with menus that can be prepared based on the ingredient information identified by the identifying means.

[0964] The "purchase suggestion means" is a mechanism for checking ingredients required for the menu suggested by the suggestion means and prompting the user to purchase any ingredients that are missing.

[0965] "Emotion recognition means" is a technology that analyzes the user's facial expressions, tone of voice, etc. to identify the user's emotions.

[0966] The "emotion-based suggestion means" is a mechanism for providing the user with an optimal menu based on the user's emotional information identified by the emotion recognition means.

[0967] The system embodying this invention efficiently utilizes ingredients in the refrigerator, proposes menus based on the user's emotions, and assists in purchasing ingredients. The components that make up this system and their specific operations are described below.

[0968] Main components of the system

[0969] 1. Imaging equipment

[0970] The camera is installed inside the refrigerator and takes pictures of the food periodically after the refrigerator door is closed. The camera can also be a smartphone camera.

[0971] 2. Identification Method

[0972] The server receives the image data sent from the camera and identifies the ingredients using an image recognition algorithm (e.g., TensorFlow). The identified ingredient information is recorded in a database (e.g., Firebase Firestore).

[0973] 3. Proposal method

[0974] The server generates a menu that can be created using a generative AI model (e.g., GPT-3) based on the identified ingredients. For example, if the identified ingredients are "tomato," "cheese," "lettuce," and "bread," the following prompt sentence can be used:

[0975] Please suggest a menu that can be made if tomatoes, cheese, lettuce, and bread are available.

[0976] The generated menu options are sent to the device (smartphone or tablet).

[0977] 4. Emotion recognition means

[0978] The server uses the smartphone's camera and microphone to record the user's facial expressions and tone of voice, and recognizes the user's emotions using an emotion recognition engine (e.g., Affectiva SDK). The results of emotion recognition are fed back to the proposal method.

[0979] 5. Emotion-based suggestion tools

[0980] Based on the emotion information, the server adjusts the menu candidates generated by the suggestion tool and generates a new menu that suits the user's state. The server also generates prompts using an AI model:

[0981] If you have tomato, cheese, lettuce, and bread, suggest a menu recommendation for a situation where the user is tired.

[0982] For example, if the system detects that the user is tired, it will suggest a quick and easy recipe such as a tomato and cheese sandwich.

[0983] 6. Purchase suggestion method

[0984] Based on the menu proposed by the suggestion means, the server generates a list of necessary ingredients. The identification means compares this with ingredient inventory information and extracts any missing ingredients. The missing ingredients are then suggested to the user so that they can be purchased through a delivery service (e.g., Uber Eats API).

[0985] Specific examples

[0986] As a concrete example of using this system, let's consider the case where a housewife takes a picture of the inside of her refrigerator using her smartphone. The camera identifies "tomato," "cheese," "lettuce," and "bread," and the server uses this information to suggest a menu. When the user confirms the suggested menu, the emotion recognition means analyzes the user's facial expression and tone of voice and determines that the user is tired. In this case, the server generates and suggests a menu that is easy to prepare in a short time.

[0987] For example, if a user selects a "tomato and cheese sandwich," the server double-checks the ingredients for that menu and finds that "mayonnaise" is missing. The server then provides a link to purchase "mayonnaise" from a delivery service.

[0988] This makes it possible to suggest appropriate menus based on the user's emotions and to suggest purchasing ingredients that are in short supply, thereby reducing household food costs and the effort required for cooking.

[0989] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0990] Step 1:

[0991] Photographing ingredients using a photography device

[0992] When the user closes the refrigerator door, the camera (smartphone camera) automatically takes a picture of the inside of the refrigerator.

[0993] Input: Image of the inside of the refrigerator

[0994] Output: Photographed food image data

[0995] Step 2:

[0996] Image data analysis and food ingredient identification

[0997] The server receives the image data sent from the camera and identifies the ingredients using an image recognition algorithm (e.g., TensorFlow).

[0998] Input: Photographed food image data

[0999] Data processing / calculation: Identifying ingredients using image recognition models

[1000] Output: List of identified ingredients

[1001] Step 3:

[1002] Storage of identified ingredient information

[1003] The server stores the identified ingredient information in a database (e.g., Firebase Firestore).

[1004] Input: Identified ingredient list

[1005] Data processing / calculation: Writing data to the database

[1006] Output: Saved ingredient information

[1007] Step 4:

[1008] User Emotion Recognition

[1009] The device's camera and microphone are used to capture the user's facial expressions and voice, and the server uses an emotion recognition engine (e.g., Affectiva SDK) to analyze the user's emotions.

[1010] Input: User facial and voice data

[1011] Data processing / calculation: Emotion analysis using emotion recognition models

[1012] Output: User's emotional information

[1013] Step 5:

[1014] Generate menu suggestions

[1015] Based on the identified ingredient information and emotional information, the server uses a generative AI model (e.g., GPT-3) to generate prompts and suggest appropriate menu items.

[1016] Input: Identified ingredients list, user's emotional information

[1017] Data processing / calculation: Menu generation using generative AI models

[1018] Output: Suggested menu options

[1019] Step 6:

[1020] View Menu Suggestions

[1021] The application on the terminal receives the menu information sent from the server and displays it to the user.

[1022] Input: Suggested menu options

[1023] Data processing / calculation: Displaying menu information to the user

[1024] Output: Menu suggestions displayed on the device

[1025] Step 7:

[1026] Check for missing ingredients

[1027] Based on the menu selected by the user, the server generates a list of ingredients needed and compares it with the ingredient inventory information in the database. If there are any ingredients that are missing, it extracts them.

[1028] Input: Selected menu, identified ingredient list

[1029] Data processing / calculation: Creating a list of necessary ingredients, extracting ingredients that are lacking

[1030] Output: List of missing ingredients

[1031] Step 8:

[1032] Generate and display purchase offers

[1033] The server creates a purchasing suggestion based on the missing ingredients and displays it to the user through the device application. The user can then click a link on the device to purchase the missing ingredients through a delivery service (e.g., Uber Eats API).

[1034] Input: List of ingredients in short supply

[1035] Data processing / calculation: generating purchase proposals

[1036] Output: Purchase suggestions displayed on the device, links to delivery services

[1037] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1038] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1039] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1040] [Third embodiment]

[1041] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1042] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1043] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1044] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1045] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1046] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1047] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1048] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1049] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1050] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1051] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1052] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1053] The present invention is a system for efficiently utilizing food ingredients in a refrigerator, saving household food costs and reducing the effort required for cooking. The system includes the following main elements:

[1054] System configuration

[1055] 1. Fridge Camera

[1056] This camera is installed inside the refrigerator and takes pictures of food ingredients. The camera takes pictures periodically after the refrigerator door is closed.

[1057] 2. Server

[1058] It includes means for receiving image data transmitted from the camera and executing an image recognition algorithm to identify ingredients from the data.

[1059] 3. Terminal

[1060] A device used by a user, such as a smartphone or tablet, on which the application is installed.

[1061] Program processing

[1062] 1. Photographing food in the refrigerator

[1063] How the refrigerator camera works

[1064] The refrigerator camera periodically photographs the food inside the refrigerator after the door is closed.

[1065] 2. Analysis of image data

[1066] Server Operation

[1067] The server receives the image data sent from the refrigerator camera.

[1068] An image recognition algorithm is applied to the received image data to identify the ingredients in the refrigerator.

[1069] For example, the server identifies "tomato," "cheese," "lettuce," "bread," etc.

[1070] The identified ingredient information is recorded in a database.

[1071] 3. Create a composable menu

[1072] Server Operation

[1073] The server generates menu options that can be created using a generative AI model based on the identified ingredient information.

[1074] For example, using "tomato," "cheese," "lettuce," and "bread," it suggests "tomato and cheese sandwich" and "lettuce and tomato salad."

[1075] The generated menu candidates are sent to the terminal.

[1076] 4. Menu suggestions

[1077] Device behavior

[1078] The application on the device receives and displays the menu information sent from the server.

[1079] The user selects the desired menu from the proposed menus.

[1080] 5. Check for missing ingredients and suggest purchasing them

[1081] Server Operation

[1082] The server generates a list of ingredients required based on the menu selected by the user.

[1083] It compares the information on food inventory in the refrigerator and extracts any ingredients that are missing.

[1084] For example, if you select "tomato and cheese sandwich," it will identify that "tomato," "cheese," and "bread" are present in the refrigerator, but "mayonnaise" is missing.

[1085] A list of missing ingredients and a purchasing suggestion including a link to the online supermarket's purchasing page are sent to the device.

[1086] 6. Displaying and executing purchase offers

[1087] Device behavior

[1088] The app on the device receives and displays the information about the missing ingredients and purchasing suggestions sent from the server.

[1089] User Actions

[1090] The user checks the list of ingredients they are short of and clicks on a link to an online supermarket to purchase the ingredients they are short of.

[1091] Specific examples

[1092] Using this system, let's consider a case where a refrigerator contains "tomatoes," "cheese," "lettuce," and "bread." The camera takes a picture of these ingredients, and the server analyzes the image data to identify them. Based on the identified ingredients, the server generates menu items such as "tomato and cheese sandwich" and "lettuce and tomato salad," and sends this to the device. The user opens the app, checks the suggested menu items, and selects the desired "tomato and cheese sandwich." Based on this menu, the server identifies the missing "mayonnaise" and suggests a link to an online supermarket to the user. The user can click the link to purchase "mayonnaise."

[1093] In this way, the present invention makes efficient use of ingredients in the refrigerator and suggests suitable menus, thereby saving household food costs and reducing the effort required for cooking.

[1094] The processing flow will be explained below.

[1095] Step 1:

[1096] The refrigerator camera takes pictures of the inside of the refrigerator at regular intervals or after the refrigerator door is closed. The camera generates high-resolution image data and transmits the data to a server.

[1097] Step 2:

[1098] The server receives the image data sent from the refrigerator camera and stores it in a temporary storage area for saving the image data.

[1099] Step 3:

[1100] The server applies image recognition algorithms to identify ingredients in the image. For example, the server uses image recognition software to identify ingredients such as "tomato," "cheese," "lettuce," and "bread."

[1101] Step 4:

[1102] The server records the identified ingredient information in a database, which includes detailed information such as the type of ingredient, the quantity, and the date and time of identification.

[1103] Step 5:

[1104] The server uses a generative AI model to generate possible menu items based on the ingredient information stored in the database. For example, using "tomato," "cheese," "lettuce," and "bread," it can suggest "tomato and cheese sandwich" or "lettuce and tomato salad."

[1105] Step 6:

[1106] The server sends the generated menu options and necessary ingredient information to the terminal, which receives this information and prepares to display it to the user.

[1107] Step 7:

[1108] The app on the device receives the menu information sent from the server and displays it to the user. The user opens the app and checks the proposed menu.

[1109] Step 8:

[1110] The user selects the desired menu from the proposed menus, and the user's selection information is sent from the terminal to the server.

[1111] Step 9:

[1112] The server generates a list of ingredients required for the selected menu based on the user's selection, and compares it with ingredient inventory information to extract any ingredients that are in short supply.

[1113] Step 10:

[1114] The server compiles a list of ingredients that are missing, generates a link to the online supermarket's purchasing page, and sends this information to the device.

[1115] Step 11:

[1116] The app on the device receives the information about the missing ingredients and the purchase suggestions sent from the server and displays them to the user.

[1117] Step 12:

[1118] The user checks the list of ingredients they are short of and clicks on a link to an online supermarket, where they can purchase the ingredients they are short of.

[1119] Example 1

[1120] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1121] Modern households are seeking to effectively utilize the ingredients in their refrigerators to save on food costs and reduce the effort required for cooking. However, current systems lack the functionality to automatically recognize ingredients in the refrigerator and suggest menus based on that information, forcing users to manually manage ingredients. Furthermore, their functionality to support the purchase of ingredients that are in short supply is limited. To address these issues, the present invention aims to provide a series of processes for automatically identifying ingredients in the refrigerator, generating menus based on that information, and purchasing ingredients that are in short supply.

[1122] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1123] In this invention, the server includes a camera for taking pictures of ingredients in the refrigerator, a means for receiving and saving image data taken by the camera, a means for identifying ingredients by applying an image recognition algorithm to the received and saved image data, a means for recording information on ingredients identified by the camera in a database, a means for generating menu candidates that can be created using a generative AI model based on the identified ingredient information, a means for sending the generated menu candidates to a user terminal, a means for displaying the generated menu candidates on the user terminal, a means for generating a list of necessary ingredients based on the menu selected by the user, a means for comparing the list of necessary ingredients with information on ingredient inventory in the refrigerator and extracting missing ingredients, and a means for sending the list of missing ingredients and a link to a purchasing page to the user terminal. This allows the system to automatically manage ingredients in the refrigerator, suggest suitable menus to the user, and quickly and easily purchase missing ingredients.

[1124] The "photography device" is a device for photographing the environment inside the refrigerator, and mainly includes a camera.

[1125] "Image data" is digital data obtained by photographing the food items in the refrigerator.

[1126] The "receiving and saving means" refers to a means for receiving image data transmitted from the imaging device and saving it temporarily or permanently, and includes server storage and the like.

[1127] An "image recognition algorithm" is a computational method for analyzing image data and identifying specific objects or features, including approaches that utilize artificial intelligence techniques.

[1128] The "identification means" is a means for applying an image recognition algorithm to identify ingredients in the refrigerator and obtain information about them.

[1129] A "database" is a system for systematically storing and managing information on identified ingredients, etc., and includes SQL databases, NoSQL databases, etc.

[1130] A "generative AI model" is an artificial intelligence model that generates new information and suggestions based on input data, and is based on machine learning and deep learning.

[1131] "Menu candidates" are options for dishes and recipes suggested by the generative AI model based on identified ingredient information.

[1132] A "user terminal" is a device used by a user, including a smartphone, tablet, computer, etc.

[1133] The "list of necessary ingredients" is a list of all ingredients required to prepare the menu selected by the user.

[1134] "Inventory information" refers to information about all ingredients currently in the refrigerator, including details such as the quantity and type.

[1135] "Missing ingredients" refers to ingredients that are needed to prepare the selected menu but are not currently present in the refrigerator.

[1136] The "Purchase Page Link" is a hyperlink to a web page that allows the user to purchase the ingredients they are running low on online.

[1137] This invention is a system for efficiently utilizing ingredients in a refrigerator, saving household food costs and reducing the effort required for cooking. The system includes a camera for photographing ingredients in the refrigerator, a server for receiving and storing image data, a means for identifying ingredients using an image recognition algorithm based on the image data, a database for recording information on identified ingredients, a means for generating menu options using a generative AI model, and a means for extracting ingredients that are in short supply and suggesting their purchase.

[1138] First, a refrigerator camera is used to take pictures of the food inside the refrigerator. This camera automatically starts up after the refrigerator door is closed and periodically takes pictures of the food inside. For example, it takes a total of three images three seconds after the refrigerator door is closed. The captured image data is sent to a server via Wi-Fi.

[1139] Next, the server receives this image data and temporarily stores it in a database. Image recognition algorithms using TensorFlow, OpenCV, etc. are applied to the stored image data. This algorithm identifies the ingredients in the refrigerator, and ingredient information such as "tomato," "cheese," "lettuce," and "bread" is recorded in the database.

[1140] The server then uses this identified ingredient information to generate menu suggestions that can be created using a generative AI model (e.g., GPT-3). An example prompt used here is:

[1141] "There are tomatoes, cheese, lettuce, and bread in the refrigerator. Can you suggest a recipe using these?"

[1142] This generates menu options such as "tomato and cheese sandwich" and "lettuce and tomato salad," which are then sent to the user's device.

[1143] An application installed on the user's device receives this menu information and displays it on the screen. The user can then select the desired menu from the suggested menu. For example, if the user selects a "tomato and cheese sandwich," the server generates a list of ingredients based on this menu and compares it with the inventory information of ingredients in the refrigerator.

[1144] At this stage, the missing ingredients are identified, and it is determined that "mayonnaise," for example, is in short supply. The server calls the online supermarket's API, obtains a purchase link for "mayonnaise," and sends it to the user's device. The user can click this link to easily purchase the missing ingredients.

[1145] As described above, the present invention can improve user convenience by automatically managing ingredients in the refrigerator, suggesting suitable menus, and suggesting the purchase of ingredients that are in short supply, thereby saving food costs and reducing the effort required for cooking.

[1146] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1147] Step 1:

[1148] How the refrigerator camera works

[1149] After the refrigerator door is closed, the refrigerator camera automatically activates and takes a picture of the food inside.

[1150] input

[1151] A signal that the refrigerator door is closing.

[1152] output

[1153] High resolution image data of the inside of a refrigerator.

[1154] Specific actions

[1155] The camera takes three images three seconds after the door is closed and saves them as digital image files in JPEG format.

[1156] Step 2:

[1157] Receiving image data from the server

[1158] The image data captured by the refrigerator camera is sent via Wi-Fi to a server, which receives and stores it.

[1159] input

[1160] Image data sent from the refrigerator camera.

[1161] output

[1162] Saving received image data.

[1163] Specific actions

[1164] The server receives image files from the camera via the FTP server function and stores them in a specific directory.

[1165] Step 3:

[1166] Server image analysis

[1167] The server applies an image recognition algorithm (e.g., TensorFlow's Inception model) to the received image data to identify the ingredients.

[1168] input

[1169] Saved image data.

[1170] output

[1171] Identified ingredient information.

[1172] Specific actions

[1173] The server uses the TensorFlow library to perform object detection on the stored image data, identifying ingredients such as "tomato," "cheese," "lettuce," and "bread." The results are stored in a database in JSON format.

[1174] Step 4:

[1175] Server menu generation

[1176] Based on the identified ingredient information, the server generates menu options that can be created using a generative AI model (e.g., GPT-3).

[1177] input

[1178] Identified ingredient information.

[1179] output

[1180] The generated menu suggestions.

[1181] Specific actions

[1182] The server inputs the following prompt into the generative AI model:

[1183] "There are tomatoes, cheese, lettuce, and bread in the refrigerator. Can you suggest a recipe using these?"

[1184] The generated results are "tomato and cheese sandwich" and "lettuce and tomato salad", which are saved in JSON format.

[1185] Step 5:

[1186] Sending and displaying menu information

[1187] The server sends the generated menu suggestions to the user's terminal, which displays them.

[1188] input

[1189] The generated menu suggestions.

[1190] output

[1191] Displaying menu information on the user's terminal.

[1192] Specific actions

[1193] The server pushes the JSON data of the menu suggestions to the user's device, which receives and displays it in the device's application. The user can then select the desired menu from the suggested menus.

[1194] Step 6:

[1195] Identifying missing ingredients

[1196] Based on the menu selected by the user, the server generates a list of ingredients needed, compares it with the inventory information in the refrigerator, and extracts any ingredients that are missing.

[1197] input

[1198] The menu selected by the user.

[1199] output

[1200] Missing ingredient information.

[1201] Specific actions

[1202] The server does the following:

[1203] 1. Refer to the recipe database for "tomato and cheese sandwich" and generate a list of ingredients needed.

[1204] 2. Compare with the food information in the refrigerator and determine that there is a shortage of "mayonnaise."

[1205] Step 7:

[1206] Submit a Purchase Offer

[1207] The server sends a list of missing ingredients and a link to the purchasing page to the user's terminal.

[1208] input

[1209] Missing ingredient information.

[1210] output

[1211] Sending purchase suggestion information to the user terminal.

[1212] Specific actions

[1213] The server calls the online supermarket's API, obtains a purchase link for "mayonnaise," and sends this along with a list of ingredients that are in short supply to the user's terminal.

[1214] Step 8:

[1215] View purchase suggestions

[1216] The device's app receives the information about the missing ingredients and the purchase suggestions sent from the server and displays them on the screen.

[1217] input

[1218] Purchase offer information sent from the server.

[1219] output

[1220] Display of purchase suggestion information on user terminal.

[1221] Specific actions

[1222] The app displays a list of missing ingredients in a pop-up window. For example, "You're running low on mayonnaise. Here's a link to buy it." The user can click the link to buy the missing ingredients.

[1223] (Application example 1)

[1224] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1225] In modern households, it is important to efficiently manage food in the refrigerator and optimize consumption. However, currently, it is time-consuming to identify the presence of ingredients and identify what is missing, and it is difficult to suggest appropriate menus or replenish missing ingredients. Furthermore, particularly in the field of food delivery, there is no technology that automatically replenishes or suggests ingredients when a specific ingredient is missing. Therefore, there is a need for a system that can efficiently manage food in the refrigerator and even automatically replenish missing ingredients.

[1226] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1227] In this invention, the server includes an imaging device that photographs ingredients in the refrigerator, identification means that analyzes image data photographed by the imaging device to identify ingredients, suggestion means that proposes menus that can be created based on ingredient information identified by the identification means, purchase suggestion means that identifies ingredients that are lacking based on the menus proposed by the suggestion means and suggests their purchase, an auxiliary device that is installed in an automatic delivery machine associated with the refrigerator and that scans ingredients in the refrigerator with an imaging device, and replenishment means that the automatic delivery machine automatically replenishes ingredients that are lacking. This makes it possible to efficiently manage ingredients in the refrigerator and automatically replenish ingredients that are lacking.

[1228] An "imaging device that photographs food ingredients in a refrigerator" is a device that is installed inside a refrigerator and that captures image data of food ingredients in the refrigerator.

[1229] The "identification means" is a device or program that has the function of analyzing image data captured by the imaging device and identifying each food ingredient in the refrigerator.

[1230] The "suggestion means" is a device or program that has the function of generating and suggesting menus that can be made using ingredients based on the ingredient information identified by the identification means.

[1231] The "purchase suggestion means" is a device or program having the function of identifying ingredients that are in short supply from the ingredient information in the refrigerator based on the menu generated by the suggestion means and suggesting to the user that they be purchased.

[1232] A "delivery automaton" is a mechanical device, such as an automatically mobile robot, used to assist in the management of refrigerators.

[1233] The "auxiliary device" is an imaging device that is installed in the automated delivery machine and is used to scan ingredients in the refrigerator.

[1234] The "replenishment means" is a device or program that has the function of identifying a shortage of ingredients and automatically replenishing the ingredients when necessary.

[1235] The present invention is a system for efficiently managing ingredients in a refrigerator and automatically replenishing ingredients when they are in short supply. Detailed embodiments of the present invention will be described below.

[1236] The present invention includes an imaging device that photographs ingredients in a refrigerator, a server, a user terminal, and an automated delivery machine. The imaging device is installed inside the refrigerator and periodically photographs ingredients and sends the image data to the server. The server analyzes the received image data and has an identification means for identifying the ingredients.

[1237] The server generates a menu that can be prepared using the suggestion means based on the identified ingredient information and transmits it to the user's terminal. The user can check and select the suggested menu through the terminal. Based on the selected menu, the server's purchase suggestion means identifies ingredients that are in short supply and suggests their purchase to the user.

[1238] Furthermore, the delivery machine is equipped with an auxiliary device that can scan the ingredients in the refrigerator. As a means of replenishment, the delivery machine can automatically replenish any ingredients that are missing.

[1239] The system operates as follows.

[1240] 1. Scan food in the refrigerator

[1241] An image capture device installed in the delivery machine periodically scans the food items in the refrigerator and captures image data, for example, after the refrigerator door is closed.

[1242] 2. Analysis of image data

[1243] The server receives the image data sent from the imaging device and identifies the ingredients using image analysis software (e.g., TensorFlow or OpenCV). The identified ingredient information is recorded in a database.

[1244] 3. Creating a Menu

[1245] The server uses a generative AI model to generate menu options based on the identified ingredients. For example, if "tomato," "cheese," "lettuce," and "bread" are identified, the server will suggest options such as "tomato and cheese sandwich" or "lettuce and tomato salad."

[1246] 4. Identifying missing ingredients and suggesting purchases

[1247] The server generates a list of ingredients based on the user's menu selection and compares it with the refrigerator's inventory data. If any ingredients are missing, a purchase suggestion is sent to the user's device.

[1248] 5. Automatic replenishment function

[1249] The automated delivery machine follows instructions from the server, automatically orders missing ingredients from online supermarkets, etc., and replenishes the refrigerator if necessary.

[1250] As a concrete example, consider a situation where there are "tomatoes," "cheese," "lettuce," and "bread" in a refrigerator. The image capture device of the automated delivery machine takes a picture of these items, and the server analyzes and identifies the image data. Based on the identified ingredient information, the server generates and suggests menu items such as "tomato and cheese sandwich" or "lettuce and tomato salad." If the user selects "tomato and cheese sandwich," the server identifies the missing "mayonnaise" and suggests a link to purchase it from an online supermarket to the user. Furthermore, the automated delivery machine automatically purchases mayonnaise and replenishes the refrigerator.

[1251] Example prompt for generative AI model:

[1252] We will build an AI system that recognizes ingredients in the user's refrigerator and suggests ingredients that are missing. Please identify ingredients from the input image, compare them with the list of necessary ingredients, extract the missing ingredients, and generate a purchase link.

[1253] As described above, the present invention can improve the efficiency of food ingredient management in a refrigerator and realize automatic replenishment of ingredients that are running low.

[1254] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1255] Step 1:

[1256] Scan food in the refrigerator

[1257] Input: Image data from an imaging device (camera) installed inside the refrigerator

[1258] Specific operation: After the refrigerator door is closed, the image capture device takes an image of the inside of the refrigerator. The captured image data is sent to the server.

[1259] Output: Image data of ingredients in the refrigerator

[1260] Step 2:

[1261] Receiving and analyzing image data

[1262] Input: Image data of the inside of the refrigerator sent from the imaging device

[1263] Specific operation: The server receives the image data and identifies the ingredients using image analysis software (e.g., TensorFlow or OpenCV). The identified ingredient information is then recorded in a database.

[1264] Output: Identification information of ingredients (e.g., "tomato," "cheese," "lettuce," "bread")

[1265] Step 3:

[1266] Generate Menu

[1267] Input: Identification information of ingredients recorded on the server

[1268] Specific operation: The server uses the generative AI model to generate menu options based on the identified ingredients. The generated menu options are sent to the user's device.

[1269] Output: Possible menu items (e.g., "Tomato and cheese sandwich," "Lettuce and tomato salad")

[1270] Step 4:

[1271] Menu suggestions and selections

[1272] Input: Menu suggestions sent from the server to the user's device

[1273] Specific operation: The user's device displays the menu options received from the server, and the user selects the desired menu.

[1274] Output: The menu selected by the user

[1275] Step 5:

[1276] Identifying missing ingredients and suggesting purchases

[1277] Input: Menu selected by the user and ingredient information in the database

[1278] Specific operation: Based on the selected menu, the server generates a list of necessary ingredients and checks it against the refrigerator inventory in the database. If any ingredients are missing, it sends a purchase suggestion to the user's device.

[1279] Output: List of ingredients that are in short supply and purchase suggestions (e.g., "mayonnaise" purchase link)

[1280] Step 6:

[1281] Purchasing and Auto-Refill

[1282] Input: Purchase offers and delivery machine replenishment orders sent from the server to the user

[1283] Specific operation: When the user checks the purchase suggestion and clicks on the link to the online supermarket to purchase the missing ingredients, the delivery machine automatically replenishes the missing ingredients based on instructions from the server. The delivery machine is equipped with an auxiliary device that places the purchased ingredients in the refrigerator.

[1284] Output: Automatically replenished food in the refrigerator

[1285] Example prompt sentence:

[1286] We will build an AI system that recognizes ingredients in the user's refrigerator and suggests ingredients that are missing. Please identify ingredients from the input image, compare them with the list of necessary ingredients, extract the missing ingredients, and generate a purchase link.

[1287] This allows users to efficiently manage ingredients in their refrigerators and automatically replenish ingredients when they run out.

[1288] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1289] The present invention combines a system for efficiently utilizing ingredients in the refrigerator, saving household food costs and reducing cooking time, with an emotion engine that recognizes the user's emotions. The system includes the following main components:

[1290] System configuration

[1291] 1. Fridge Camera

[1292] This camera is installed inside the refrigerator and takes pictures of food ingredients. The camera takes pictures periodically after the refrigerator door is closed.

[1293] 2. Server

[1294] It includes means for receiving image data transmitted from the camera and executing an image recognition algorithm to identify ingredients from the data.

[1295] It is equipped with an emotion engine that recognizes the user's emotions and suggests appropriate menus based on those emotions.

[1296] 3. Terminal

[1297] A device used by a user, such as a smartphone or tablet, on which the application is installed.

[1298] Program processing

[1299] 1. Photographing food in the refrigerator

[1300] How the refrigerator camera works

[1301] The refrigerator camera periodically photographs the food inside the refrigerator after the door is closed.

[1302] 2. Analysis of image data

[1303] Server Operation

[1304] The server receives the image data sent from the refrigerator camera.

[1305] An image recognition algorithm is applied to the received image data to identify the ingredients in the refrigerator.

[1306] For example, the server identifies "tomato," "cheese," "lettuce," "bread," etc.

[1307] The identified ingredient information is recorded in a database.

[1308] 3. Create a composable menu

[1309] Server Operation

[1310] The server generates menu options that can be created using a generative AI model based on the identified ingredient information.

[1311] For example, using "tomato," "cheese," "lettuce," and "bread," it suggests "tomato and cheese sandwich" and "lettuce and tomato salad."

[1312] 4. Emotion Recognition by Emotion Engine

[1313] Server Operation

[1314] The server uses an emotion engine to recognize the user's emotions by analyzing the user's facial expressions and tone of voice.

[1315] The emotion engine also learns the user's past selection history and behavioral patterns to predict emotions.

[1316] 5. Menu suggestions based on emotions

[1317] Server Operation

[1318] The server proposes a menu according to the user's emotions based on the user's emotion information obtained from the emotion engine.

[1319] For example, if the server detects that the user is tired, it will suggest a menu that is simple and quick to prepare.

[1320] 6. Menu suggestions

[1321] Device behavior

[1322] The application on the device receives the menu information sent from the server and displays it to the user.

[1323] The user opens the app and sees the suggested menu.

[1324] 7. Check for missing ingredients and suggest purchasing them

[1325] Server Operation

[1326] The server generates a list of ingredients required based on the menu selected by the user.

[1327] The system compares the information with food stock information and extracts any ingredients that are in short supply.

[1328] 8. Displaying and executing purchase offers

[1329] Device behavior

[1330] The app on the device receives the information about the missing ingredients and the purchase suggestions sent from the server and displays them to the user.

[1331] User Actions

[1332] The user checks the list of ingredients they are short of and clicks on a link to an online supermarket to purchase the ingredients they are short of.

[1333] Specific examples

[1334] Using this system, let's assume that there are "tomatoes," "cheese," "lettuce," and "bread" in the refrigerator. The camera takes a picture of these ingredients, and the server analyzes the image data to identify them. Based on the identified ingredients, the server generates a menu such as "tomato and cheese sandwich" or "lettuce and tomato salad" and sends it to the terminal.

[1335] When a user opens the app and checks the suggested menu, the emotion engine analyzes the user's facial expressions and tone of voice. If it determines that the user is tired, it suggests a "tomato and cheese sandwich," which is easy and quick to make. When the user selects this suggestion, the server checks the ingredients needed for the "tomato and cheese sandwich" and identifies the missing "mayonnaise." The server then suggests the user purchase "mayonnaise" from an online supermarket, and the user can click a link to purchase it.

[1336] In this way, by combining emotion engines, it becomes possible to suggest menus that correspond to the user's emotions, enabling more individually tailored dish suggestions to be realized.

[1337] The processing flow will be explained below.

[1338] Step 1:

[1339] The refrigerator camera takes pictures of the inside of the refrigerator at regular intervals or after the refrigerator door is closed. The camera generates high-resolution image data and transmits the data to a server.

[1340] Step 2:

[1341] The server receives the image data sent from the refrigerator camera and stores it in a temporary storage area for saving the image data.

[1342] Step 3:

[1343] The server applies image recognition algorithms to identify ingredients in the image. For example, the server uses image recognition software to identify ingredients such as "tomato," "cheese," "lettuce," and "bread."

[1344] Step 4:

[1345] The server records the identified ingredient information in a database, which includes detailed information such as the type of ingredient, the quantity, and the date and time of identification.

[1346] Step 5:

[1347] The server uses a generative AI model to generate possible menu items based on the ingredient information stored in the database. For example, using "tomato," "cheese," "lettuce," and "bread," it can suggest "tomato and cheese sandwich" or "lettuce and tomato salad."

[1348] Step 6:

[1349] The server sends the generated menu options and necessary ingredient information to the terminal, which receives this information and prepares to display it to the user.

[1350] Step 7:

[1351] The app on the device receives the menu information sent from the server and displays it to the user. The user opens the app and checks the proposed menu.

[1352] Step 8:

[1353] The app on the device recognizes the user's facial expressions and tone of voice and sends that data to the emotion engine, which then analyzes facial expressions and voice to recognize the user's emotional state.

[1354] Step 9:

[1355] The emotion engine on the server recognizes the user's emotions based on the analysis results and adjusts the menu suggestions based on the user's emotional state. For example, if the system recognizes that the user is tired, it will prioritize suggestions of simple and quick meals.

[1356] Step 10:

[1357] The user selects the desired menu from the proposed menus, and the user's selection information is sent from the terminal to the server.

[1358] Step 11:

[1359] The server generates a list of ingredients required for the selected menu based on the user's selection, and compares it with ingredient inventory information to extract any ingredients that are in short supply.

[1360] Step 12:

[1361] The server compiles a list of ingredients that are missing, generates a link to the online supermarket's purchasing page, and sends this information to the device.

[1362] Step 13:

[1363] The app on the device receives the information about the missing ingredients and the purchase suggestions sent from the server and displays them to the user.

[1364] Step 14:

[1365] The user checks the list of ingredients they are short of and clicks on a link to an online supermarket, where they can purchase the ingredients they are short of.

[1366] Example 2

[1367] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1368] In modern households, efficiently utilizing the ingredients in the refrigerator to reduce food costs and cooking time is an important issue. However, it is difficult for users to properly manage the ingredients in their refrigerator, resulting in frequent waste and unused ingredients. In addition, menu suggestions are not tailored to the user's condition or mood, making it difficult for users to maintain their motivation to cook. Furthermore, if necessary ingredients are in short supply, users have to go shopping separately, which is a hassle.

[1369] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1370] In this invention, the server includes a camera that takes pictures of ingredients in the refrigerator, an image analysis unit that analyzes image data taken by the camera and identifies ingredients, a suggestion unit that suggests menus that can be made based on the ingredient information identified by the image analysis unit, a purchase suggestion unit that identifies ingredients that are lacking based on the menus suggested by the suggestion unit and suggests the purchase of those ingredients, and an emotion recognition unit that recognizes the user's emotions and suggests appropriate menus accordingly. This makes it possible to efficiently use ingredients in the refrigerator and reduce household food costs, and furthermore, by suggesting menus that correspond to the user's emotions, it is possible to improve motivation to cook and reduce the effort required.

[1371] "Photography device" refers to a device installed to photograph food items in a refrigerator.

[1372] "Image analysis means" refers to algorithms or software that analyzes image data acquired by the imaging device and identifies ingredients.

[1373] The "suggestion means" refers to a system for suggesting menus that can be prepared based on the ingredient information identified by the image analysis means.

[1374] The "purchase suggestion means" refers to a system for determining ingredients that are in short supply based on the menu suggested by the suggestion means and suggesting to the user to purchase those ingredients.

[1375] "Emotion recognition means" refers to algorithms or devices that analyze a user's facial expressions and tone of voice to recognize their emotions.

[1376] A "refrigerator" refers to a cooling device used in homes and businesses to preserve food.

[1377] "Ingredients" refers to food stored in the refrigerator for use in cooking.

[1378] "Image data" refers to digital data of photographs and videos captured by a photographing device.

[1379] "Ingredient information" refers to data relating to the type and quantity of ingredients identified by the image analysis means.

[1380] "Menu" refers to a list of dishes that can be made using ingredients.

[1381] The present invention combines a system for efficiently utilizing ingredients in a refrigerator, saving household food costs and reducing the effort required for cooking, with an emotion engine that recognizes the user's emotions. This system includes a photographing device for photographing ingredients in the refrigerator, image analysis means for analyzing image data and identifying ingredients, suggestion means for suggesting menus based on identified ingredient information, purchase suggestion means for identifying ingredients that are in short supply and suggesting their purchase, and emotion recognition means for recognizing the user's emotions and suggesting appropriate menus.

[1382] Specific hardware and software:

[1383] 1. Camera: A camera installed inside the refrigerator takes pictures periodically after the refrigerator door is closed. For example, a commercially available webcam or a special-purpose camera can be used.

[1384] 2. Image analysis means: Uses image recognition algorithms installed on the server. Specifically, open source image processing libraries and machine learning models (e.g., TensorFlow and OpenCV) can be used.

[1385] 3. Proposal method: Menu suggestions are made using a generative AI model. Specifically, a generative AI model that performs natural language processing (e.g., GPT-3) is used to generate a menu based on ingredient information.

[1386] 4. Purchase suggestion means: Using a database management system installed on the server, the missing ingredients are identified based on the identified ingredient information and the proposed menu, and a purchase suggestion is made.

[1387] 5. Emotion Recognition: An emotion engine is used to analyze the user's facial expressions and tone of voice. For example, a deep learning model (e.g., FaceNet) can be used for facial expression analysis, and a speech recognition engine (e.g., Google Cloud Speech-to-Text) can be used for voice analysis.

[1388] Operational flow and data calculation:

[1389] The server receives image data sent from the refrigerator camera and performs image analysis to identify ingredients from the data. The analyzed ingredient information is recorded in a database within the server. The server then uses a generative AI model based on this ingredient information to generate possible menu options. When the user uses the application to check the proposed menu, the emotion engine analyzes the user's facial expressions and tone of voice and presents a menu that corresponds to the user's emotions. Furthermore, based on the proposed menu, the server determines which ingredients are in short supply and suggests that the user purchase them.

[1390] Example 1:

[1391] If there are "tomatoes," "cheese," "lettuce," and "bread" in the refrigerator, the camera takes a picture of these ingredients, and the server analyzes the image data to identify them. Based on the identified ingredient information, the server generates menu items such as "tomato and cheese sandwich" and "lettuce and tomato salad," and sends them to the device. When the user opens the app and checks the suggested menu items, the emotion engine analyzes the user's facial expressions and tone of voice, and if it determines that the user is tired, it suggests "tomato and cheese sandwich," which is easy to make and can be made quickly. If the user selects this suggestion, the server checks the ingredients needed for the "tomato and cheese sandwich" and identifies the missing "mayonnaise." The server then suggests the user purchase "mayonnaise" from an online supermarket, and the user can click a link to purchase it.

[1392] Example 2:

[1393] Suggest a menu that is easy and quick to make using the "chicken," "carrots," and "onions" in the refrigerator. Also, if the user is tired, prioritize items that are easy and nutritious. An example of a prompt sentence is, "Suggest a menu that is easy and quick to make using the "chicken," "carrots," and "onions" in the refrigerator. Also, if the user is tired, prioritize items that are easy and nutritious."

[1394] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1395] Step 1:

[1396] Photographing food in the refrigerator

[1397] How the refrigerator camera works

[1398] The refrigerator camera periodically takes pictures of the inside of the refrigerator after the door is closed.

[1399] For example, a refrigerator is set to take a photo every hour.

[1400] Input: Image of the inside of the refrigerator

[1401] Output: Captured image data

[1402] Step 2:

[1403] Image data analysis

[1404] Server Operation

[1405] The server receives the image data sent from the refrigerator camera.

[1406] An image recognition algorithm is applied to the received image data to identify the ingredients inside.

[1407] For example, TensorFlow or OpenCV is used as an image recognition algorithm.

[1408] Input: Photographed image data

[1409] Data processing and calculation: Apply image recognition algorithms to identify the type of food

[1410] Output: Identified ingredients (e.g., "tomato," "cheese," "lettuce," "bread," etc.)

[1411] Step 3:

[1412] Generate composable menus

[1413] Server Operation

[1414] The server generates menu options that can be created using a generative AI model based on the identified ingredient information.

[1415] Specifically, we use a generative AI model (e.g., GPT-3) that performs natural language processing.

[1416] Input: Identified ingredient information

[1417] Data processing and data calculation: Generate menu suggestions using generative AI models

[1418] Output: Generated menu options (e.g., "tomato and cheese sandwich," "lettuce and tomato salad," etc.)

[1419] Step 4:

[1420] Emotion recognition by emotion recognition means

[1421] Server Operation

[1422] The server uses an emotion recognition means to recognize the user's emotion.

[1423] A deep learning model (e.g., FaceNet) is used for facial expression analysis, and a speech recognition engine (e.g., Google Cloud Speech-to-Text) is used for voice analysis.

[1424] Input: User's facial image and voice data

[1425] Data processing and data calculation: Applying emotion recognition algorithms to analyze user emotions

[1426] Output: Analyzed user's emotional information (e.g., "Tired," "Relaxed," etc.)

[1427] Step 5:

[1428] Menu suggestions based on emotions

[1429] Server Operation

[1430] The server proposes a menu according to the user's emotion based on the emotion information obtained from the emotion recognition means.

[1431] For example, if the user is tired, the system will prioritize suggesting menus that are simple and can be prepared in a short time.

[1432] Input: Analyzed user emotion information and generated menu suggestions

[1433] Data processing and data calculation: determining menu priorities based on emotional information

[1434] Output: Menu suggestions based on emotions (e.g., "Tomato and cheese sandwich")

[1435] Step 6:

[1436] Menu suggestions

[1437] Device behavior

[1438] The application on the device receives the menu information sent from the server and displays it to the user.

[1439] The user opens the app and sees the suggested menu.

[1440] For example, a smartphone or tablet app might display menu items such as "Tomato and Cheese Sandwich" or "Lettuce and Tomato Salad."

[1441] Input: Emotion-based menu suggestions

[1442] Output: Displayed menu information

[1443] Step 7:

[1444] Check for missing ingredients and suggest purchasing

[1445] Server Operation

[1446] The server generates a list of ingredients required based on the menu selected by the user.

[1447] The generated ingredient list is compared with the inventory information in the refrigerator to extract any ingredients that are missing.

[1448] For example, it may identify that a tomato and cheese sandwich requires mayonnaise, but that it is not in stock in the refrigerator.

[1449] Input: Selected menu, refrigerator inventory information

[1450] Data processing and calculation: Generate a list of necessary ingredients and extract missing ingredients

[1451] Output: Missing ingredient information (e.g., "mayonnaise")

[1452] Step 8:

[1453] View and execute purchase offers

[1454] Device behavior

[1455] The app on the device receives the information about the missing ingredients and the purchase suggestions sent from the server and displays them to the user.

[1456] For example, a message may appear saying, "We're low on mayonnaise. Would you like to purchase some from the online supermarket?"

[1457] Input: Missing ingredient information

[1458] Output: Displayed purchase suggestions

[1459] User Actions

[1460] The user checks the list of ingredients they are short of and clicks on a link to an online supermarket to purchase the ingredients they are short of.

[1461] For example, when you click, you are redirected to an online supermarket website, where the missing ingredients are automatically added to your cart.

[1462] Input: Purchase suggestion for ingredients that are in short supply

[1463] Output: The purchase is completed and the required ingredients are added back to inventory.

[1464] (Application example 2)

[1465] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1466] Conventional refrigerator management systems have technologies that identify ingredients in a refrigerator and suggest menus based on that information, but they are unable to suggest optimal menus because they are unable to make suggestions based on the user's emotions or physical condition. In particular, in busy daily lives and under stressful circumstances, menu suggestions based on the user's emotions are more desirable. Therefore, the present invention aims to provide a system that recognizes the user's emotions and suggests optimal menus based on emotional information.

[1467] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1468] In this invention, the server includes a photographing device that photographs ingredients in the refrigerator, an identification means that analyzes image data photographed by the photographing device to identify ingredients, a suggestion means that suggests menus that can be made based on the ingredient information identified by the identification means, a purchase suggestion means that identifies ingredients that are in short supply based on the menu suggested by the suggestion means and suggests their purchase, an emotion recognition means that recognizes the user's emotion, and an emotion-based suggestion means that suggests menus based on the emotion information recognized by the emotion recognition means. This makes it possible to suggest optimal menus according to the user's emotions.

[1469] The "photography device" is a device used to take pictures of ingredients in the refrigerator.

[1470] The "identification means" is a technology that analyzes image data acquired by the imaging device and identifies ingredients.

[1471] The "proposing means" is a mechanism for providing the user with menus that can be prepared based on the ingredient information identified by the identifying means.

[1472] The "purchase suggestion means" is a mechanism for checking ingredients required for the menu suggested by the suggestion means and prompting the user to purchase any ingredients that are missing.

[1473] "Emotion recognition means" is a technology that analyzes the user's facial expressions, tone of voice, etc. to identify the user's emotions.

[1474] The "emotion-based suggestion means" is a mechanism for providing the user with an optimal menu based on the user's emotional information identified by the emotion recognition means.

[1475] The system embodying this invention efficiently utilizes ingredients in the refrigerator, proposes menus based on the user's emotions, and assists in purchasing ingredients. The components that make up this system and their specific operations are described below.

[1476] Main components of the system

[1477] 1. Imaging equipment

[1478] The camera is installed inside the refrigerator and takes pictures of the food periodically after the refrigerator door is closed. The camera can also be a smartphone camera.

[1479] 2. Identification Method

[1480] The server receives the image data sent from the camera and identifies the ingredients using an image recognition algorithm (e.g., TensorFlow). The identified ingredient information is recorded in a database (e.g., Firebase Firestore).

[1481] 3. Proposal method

[1482] The server generates a menu that can be created using a generative AI model (e.g., GPT-3) based on the identified ingredients. For example, if the identified ingredients are "tomato," "cheese," "lettuce," and "bread," the following prompt sentence can be used:

[1483] Please suggest a menu that can be made if tomatoes, cheese, lettuce, and bread are available.

[1484] The generated menu options are sent to the device (smartphone or tablet).

[1485] 4. Emotion recognition means

[1486] The server uses the smartphone's camera and microphone to record the user's facial expressions and tone of voice, and recognizes the user's emotions using an emotion recognition engine (e.g., Affectiva SDK). The results of emotion recognition are fed back to the proposal method.

[1487] 5. Emotion-based suggestion tools

[1488] Based on the emotion information, the server adjusts the menu candidates generated by the suggestion tool and generates a new menu that suits the user's state. The server also generates prompts using an AI model:

[1489] If you have tomato, cheese, lettuce, and bread, suggest a menu recommendation for a situation where the user is tired.

[1490] For example, if the system detects that the user is tired, it will suggest a quick and easy recipe such as a tomato and cheese sandwich.

[1491] 6. Purchase suggestion method

[1492] Based on the menu proposed by the suggestion means, the server generates a list of necessary ingredients. The identification means compares this with ingredient inventory information and extracts any missing ingredients. The missing ingredients are then suggested to the user so that they can be purchased through a delivery service (e.g., Uber Eats API).

[1493] Specific examples

[1494] As a concrete example of using this system, let's consider the case where a housewife takes a picture of the inside of her refrigerator using her smartphone. The camera identifies "tomato," "cheese," "lettuce," and "bread," and the server uses this information to suggest a menu. When the user confirms the suggested menu, the emotion recognition means analyzes the user's facial expression and tone of voice and determines that the user is tired. In this case, the server generates and suggests a menu that is easy to prepare in a short time.

[1495] For example, if a user selects a "tomato and cheese sandwich," the server double-checks the ingredients for that menu and finds that "mayonnaise" is missing. The server then provides a link to purchase "mayonnaise" from a delivery service.

[1496] This makes it possible to suggest appropriate menus based on the user's emotions and to suggest purchasing ingredients that are in short supply, thereby reducing household food costs and the effort required for cooking.

[1497] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1498] Step 1:

[1499] Photographing ingredients using a photography device

[1500] When the user closes the refrigerator door, the camera (smartphone camera) automatically takes a picture of the inside of the refrigerator.

[1501] Input: Image of the inside of the refrigerator

[1502] Output: Photographed food image data

[1503] Step 2:

[1504] Image data analysis and food ingredient identification

[1505] The server receives the image data sent from the camera and identifies the ingredients using an image recognition algorithm (e.g., TensorFlow).

[1506] Input: Photographed food image data

[1507] Data processing / calculation: Identifying ingredients using image recognition models

[1508] Output: List of identified ingredients

[1509] Step 3:

[1510] Storage of identified ingredient information

[1511] The server stores the identified ingredient information in a database (e.g., Firebase Firestore).

[1512] Input: Identified ingredient list

[1513] Data processing / calculation: Writing data to the database

[1514] Output: Saved ingredient information

[1515] Step 4:

[1516] User Emotion Recognition

[1517] The device's camera and microphone are used to capture the user's facial expressions and voice, and the server uses an emotion recognition engine (e.g., Affectiva SDK) to analyze the user's emotions.

[1518] Input: User facial and voice data

[1519] Data processing / calculation: Emotion analysis using emotion recognition models

[1520] Output: User's emotional information

[1521] Step 5:

[1522] Generate menu suggestions

[1523] Based on the identified ingredient information and emotional information, the server uses a generative AI model (e.g., GPT-3) to generate prompts and suggest appropriate menu items.

[1524] Input: Identified ingredients list, user's emotional information

[1525] Data processing / calculation: Menu generation using generative AI models

[1526] Output: Suggested menu options

[1527] Step 6:

[1528] View Menu Suggestions

[1529] The application on the terminal receives the menu information sent from the server and displays it to the user.

[1530] Input: Suggested menu options

[1531] Data processing / calculation: Displaying menu information to the user

[1532] Output: Menu suggestions displayed on the device

[1533] Step 7:

[1534] Check for missing ingredients

[1535] Based on the menu selected by the user, the server generates a list of ingredients needed and compares it with the ingredient inventory information in the database. If there are any ingredients that are missing, it extracts them.

[1536] Input: Selected menu, identified ingredient list

[1537] Data processing / calculation: Creating a list of necessary ingredients, extracting ingredients that are lacking

[1538] Output: List of missing ingredients

[1539] Step 8:

[1540] Generate and display purchase offers

[1541] The server creates a purchasing suggestion based on the missing ingredients and displays it to the user through the device application. The user can then click a link on the device to purchase the missing ingredients through a delivery service (e.g., Uber Eats API).

[1542] Input: List of ingredients in short supply

[1543] Data processing / calculation: generating purchase proposals

[1544] Output: Purchase suggestions displayed on the device, links to delivery services

[1545] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1546] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1547] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1548] [Fourth embodiment]

[1549] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1550] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1551] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1552] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1553] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1554] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1555] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1556] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1557] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1558] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1559] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1560] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1561] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1562] The present invention is a system for efficiently utilizing food ingredients in a refrigerator, saving household food costs and reducing the effort required for cooking. The system includes the following main elements:

[1563] System configuration

[1564] 1. Fridge Camera

[1565] This camera is installed inside the refrigerator and takes pictures of food ingredients. The camera takes pictures periodically after the refrigerator door is closed.

[1566] 2. Server

[1567] It includes means for receiving image data transmitted from the camera and executing an image recognition algorithm to identify ingredients from the data.

[1568] 3. Terminal

[1569] A device used by a user, such as a smartphone or tablet, on which the application is installed.

[1570] Program processing

[1571] 1. Photographing food in the refrigerator

[1572] How the refrigerator camera works

[1573] The refrigerator camera periodically photographs the food inside the refrigerator after the door is closed.

[1574] 2. Analysis of image data

[1575] Server Operation

[1576] The server receives the image data sent from the refrigerator camera.

[1577] An image recognition algorithm is applied to the received image data to identify the ingredients in the refrigerator.

[1578] For example, the server identifies "tomato," "cheese," "lettuce," "bread," etc.

[1579] The identified ingredient information is recorded in a database.

[1580] 3. Create a composable menu

[1581] Server Operation

[1582] The server generates menu options that can be created using a generative AI model based on the identified ingredient information.

[1583] For example, using "tomato," "cheese," "lettuce," and "bread," it suggests "tomato and cheese sandwich" and "lettuce and tomato salad."

[1584] The generated menu candidates are sent to the terminal.

[1585] 4. Menu suggestions

[1586] Device behavior

[1587] The application on the device receives and displays the menu information sent from the server.

[1588] The user selects the desired menu from the proposed menus.

[1589] 5. Check for missing ingredients and suggest purchasing them

[1590] Server Operation

[1591] The server generates a list of ingredients required based on the menu selected by the user.

[1592] It compares the information on food inventory in the refrigerator and extracts any ingredients that are missing.

[1593] For example, if you select "tomato and cheese sandwich," it will identify that "tomato," "cheese," and "bread" are present in the refrigerator, but "mayonnaise" is missing.

[1594] A list of missing ingredients and a purchasing suggestion including a link to the online supermarket's purchasing page are sent to the device.

[1595] 6. Displaying and executing purchase offers

[1596] Device behavior

[1597] The app on the device receives and displays the information about the missing ingredients and purchasing suggestions sent from the server.

[1598] User Actions

[1599] The user checks the list of ingredients they are short of and clicks on a link to an online supermarket to purchase the ingredients they are short of.

[1600] Specific examples

[1601] Using this system, let's consider a case where a refrigerator contains "tomatoes," "cheese," "lettuce," and "bread." The camera takes a picture of these ingredients, and the server analyzes the image data to identify them. Based on the identified ingredients, the server generates menu items such as "tomato and cheese sandwich" and "lettuce and tomato salad," and sends this to the device. The user opens the app, checks the suggested menu items, and selects the desired "tomato and cheese sandwich." Based on this menu, the server identifies the missing "mayonnaise" and suggests a link to an online supermarket to the user. The user can click the link to purchase "mayonnaise."

[1602] In this way, the present invention makes efficient use of ingredients in the refrigerator and suggests suitable menus, thereby saving household food costs and reducing the effort required for cooking.

[1603] The processing flow will be explained below.

[1604] Step 1:

[1605] The refrigerator camera takes pictures of the inside of the refrigerator at regular intervals or after the refrigerator door is closed. The camera generates high-resolution image data and transmits the data to a server.

[1606] Step 2:

[1607] The server receives the image data sent from the refrigerator camera and stores it in a temporary storage area for saving the image data.

[1608] Step 3:

[1609] The server applies image recognition algorithms to identify ingredients in the image. For example, the server uses image recognition software to identify ingredients such as "tomato," "cheese," "lettuce," and "bread."

[1610] Step 4:

[1611] The server records the identified ingredient information in a database, which includes detailed information such as the type of ingredient, the quantity, and the date and time of identification.

[1612] Step 5:

[1613] The server uses a generative AI model to generate possible menu items based on the ingredient information stored in the database. For example, using "tomato," "cheese," "lettuce," and "bread," it can suggest "tomato and cheese sandwich" or "lettuce and tomato salad."

[1614] Step 6:

[1615] The server sends the generated menu options and necessary ingredient information to the terminal, which receives this information and prepares to display it to the user.

[1616] Step 7:

[1617] The app on the device receives the menu information sent from the server and displays it to the user. The user opens the app and checks the proposed menu.

[1618] Step 8:

[1619] The user selects the desired menu from the proposed menus, and the user's selection information is sent from the terminal to the server.

[1620] Step 9:

[1621] The server generates a list of ingredients required for the selected menu based on the user's selection, and compares it with ingredient inventory information to extract any ingredients that are in short supply.

[1622] Step 10:

[1623] The server compiles a list of ingredients that are missing, generates a link to the online supermarket's purchasing page, and sends this information to the device.

[1624] Step 11:

[1625] The app on the device receives the information about the missing ingredients and the purchase suggestions sent from the server and displays them to the user.

[1626] Step 12:

[1627] The user checks the list of ingredients they are short of and clicks on a link to an online supermarket, where they can purchase the ingredients they are short of.

[1628] Example 1

[1629] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1630] Modern households are seeking to effectively utilize the ingredients in their refrigerators to save on food costs and reduce the effort required for cooking. However, current systems lack the functionality to automatically recognize ingredients in the refrigerator and suggest menus based on that information, forcing users to manually manage ingredients. Furthermore, their functionality to support the purchase of ingredients that are in short supply is limited. To address these issues, the present invention aims to provide a series of processes for automatically identifying ingredients in the refrigerator, generating menus based on that information, and purchasing ingredients that are in short supply.

[1631] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1632] In this invention, the server includes a camera for taking pictures of ingredients in the refrigerator, a means for receiving and saving image data taken by the camera, a means for identifying ingredients by applying an image recognition algorithm to the received and saved image data, a means for recording information on ingredients identified by the camera in a database, a means for generating menu candidates that can be created using a generative AI model based on the identified ingredient information, a means for sending the generated menu candidates to a user terminal, a means for displaying the generated menu candidates on the user terminal, a means for generating a list of necessary ingredients based on the menu selected by the user, a means for comparing the list of necessary ingredients with information on ingredient inventory in the refrigerator and extracting missing ingredients, and a means for sending the list of missing ingredients and a link to a purchasing page to the user terminal. This allows the system to automatically manage ingredients in the refrigerator, suggest suitable menus to the user, and quickly and easily purchase missing ingredients.

[1633] The "photography device" is a device for photographing the environment inside the refrigerator, and mainly includes a camera.

[1634] "Image data" is digital data obtained by photographing the food items in the refrigerator.

[1635] The "receiving and saving means" refers to a means for receiving image data transmitted from the imaging device and saving it temporarily or permanently, and includes server storage and the like.

[1636] An "image recognition algorithm" is a computational method for analyzing image data and identifying specific objects or features, including approaches that utilize artificial intelligence techniques.

[1637] The "identification means" is a means for applying an image recognition algorithm to identify ingredients in the refrigerator and obtain information about them.

[1638] A "database" is a system for systematically storing and managing information on identified ingredients, etc., and includes SQL databases, NoSQL databases, etc.

[1639] A "generative AI model" is an artificial intelligence model that generates new information and suggestions based on input data, and is based on machine learning and deep learning.

[1640] "Menu candidates" are options for dishes and recipes suggested by the generative AI model based on identified ingredient information.

[1641] A "user terminal" is a device used by a user, including a smartphone, tablet, computer, etc.

[1642] The "list of necessary ingredients" is a list of all ingredients required to prepare the menu selected by the user.

[1643] "Inventory information" refers to information about all ingredients currently in the refrigerator, including details such as the quantity and type.

[1644] "Missing ingredients" refers to ingredients that are needed to prepare the selected menu but are not currently present in the refrigerator.

[1645] The "Purchase Page Link" is a hyperlink to a web page that allows the user to purchase the ingredients they are running low on online.

[1646] This invention is a system for efficiently utilizing ingredients in a refrigerator, saving household food costs and reducing the effort required for cooking. The system includes a camera for photographing ingredients in the refrigerator, a server for receiving and storing image data, a means for identifying ingredients using an image recognition algorithm based on the image data, a database for recording information on identified ingredients, a means for generating menu options using a generative AI model, and a means for extracting ingredients that are in short supply and suggesting their purchase.

[1647] First, a refrigerator camera is used to take pictures of the food inside the refrigerator. This camera automatically starts up after the refrigerator door is closed and periodically takes pictures of the food inside. For example, it takes a total of three images three seconds after the refrigerator door is closed. The captured image data is sent to a server via Wi-Fi.

[1648] Next, the server receives this image data and temporarily stores it in a database. Image recognition algorithms using TensorFlow, OpenCV, etc. are applied to the stored image data. This algorithm identifies the ingredients in the refrigerator, and ingredient information such as "tomato," "cheese," "lettuce," and "bread" is recorded in the database.

[1649] The server then uses this identified ingredient information to generate menu suggestions that can be created using a generative AI model (e.g., GPT-3). An example prompt used here is:

[1650] "There are tomatoes, cheese, lettuce, and bread in the refrigerator. Can you suggest a recipe using these?"

[1651] This generates menu options such as "tomato and cheese sandwich" and "lettuce and tomato salad," which are then sent to the user's device.

[1652] An application installed on the user's device receives this menu information and displays it on the screen. The user can then select the desired menu from the suggested menu. For example, if the user selects a "tomato and cheese sandwich," the server generates a list of ingredients based on this menu and compares it with the inventory information of ingredients in the refrigerator.

[1653] At this stage, the missing ingredients are identified, and it is determined that "mayonnaise," for example, is in short supply. The server calls the online supermarket's API, obtains a purchase link for "mayonnaise," and sends it to the user's device. The user can click this link to easily purchase the missing ingredients.

[1654] As described above, the present invention can improve user convenience by automatically managing ingredients in the refrigerator, suggesting suitable menus, and suggesting the purchase of ingredients that are in short supply, thereby saving food costs and reducing the effort required for cooking.

[1655] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1656] Step 1:

[1657] How the refrigerator camera works

[1658] After the refrigerator door is closed, the refrigerator camera automatically activates and takes a picture of the food inside.

[1659] input

[1660] A signal that the refrigerator door is closing.

[1661] output

[1662] High resolution image data of the inside of a refrigerator.

[1663] Specific actions

[1664] The camera takes three images three seconds after the door is closed and saves them as digital image files in JPEG format.

[1665] Step 2:

[1666] Receiving image data from the server

[1667] The image data captured by the refrigerator camera is sent via Wi-Fi to a server, which receives and stores it.

[1668] input

[1669] Image data sent from the refrigerator camera.

[1670] output

[1671] Saving received image data.

[1672] Specific actions

[1673] The server receives image files from the camera via the FTP server function and stores them in a specific directory.

[1674] Step 3:

[1675] Server image analysis

[1676] The server applies an image recognition algorithm (e.g., TensorFlow's Inception model) to the received image data to identify the ingredients.

[1677] input

[1678] Saved image data.

[1679] output

[1680] Identified ingredient information.

[1681] Specific actions

[1682] The server uses the TensorFlow library to perform object detection on the stored image data, identifying ingredients such as "tomato," "cheese," "lettuce," and "bread." The results are stored in a database in JSON format.

[1683] Step 4:

[1684] Server menu generation

[1685] Based on the identified ingredient information, the server generates menu options that can be created using a generative AI model (e.g., GPT-3).

[1686] input

[1687] Identified ingredient information.

[1688] output

[1689] The generated menu suggestions.

[1690] Specific actions

[1691] The server inputs the following prompt into the generative AI model:

[1692] "There are tomatoes, cheese, lettuce, and bread in the refrigerator. Can you suggest a recipe using these?"

[1693] The generated results are "tomato and cheese sandwich" and "lettuce and tomato salad", which are saved in JSON format.

[1694] Step 5:

[1695] Sending and displaying menu information

[1696] The server sends the generated menu suggestions to the user's terminal, which displays them.

[1697] input

[1698] The generated menu suggestions.

[1699] output

[1700] Displaying menu information on the user's terminal.

[1701] Specific actions

[1702] The server pushes the JSON data of the menu suggestions to the user's device, which receives and displays it in the device's application. The user can then select the desired menu from the suggested menus.

[1703] Step 6:

[1704] Identifying missing ingredients

[1705] Based on the menu selected by the user, the server generates a list of ingredients needed, compares it with the inventory information in the refrigerator, and extracts any ingredients that are missing.

[1706] input

[1707] The menu selected by the user.

[1708] output

[1709] Missing ingredient information.

[1710] Specific actions

[1711] The server does the following:

[1712] 1. Refer to the recipe database for "tomato and cheese sandwich" and generate a list of ingredients needed.

[1713] 2. Compare with the food information in the refrigerator and determine that there is a shortage of "mayonnaise."

[1714] Step 7:

[1715] Submit a Purchase Offer

[1716] The server sends a list of missing ingredients and a link to the purchasing page to the user's terminal.

[1717] input

[1718] Missing ingredient information.

[1719] output

[1720] Sending purchase suggestion information to the user terminal.

[1721] Specific actions

[1722] The server calls the online supermarket's API, obtains a purchase link for "mayonnaise," and sends this along with a list of ingredients that are in short supply to the user's terminal.

[1723] Step 8:

[1724] View purchase suggestions

[1725] The device's app receives the information about the missing ingredients and the purchase suggestions sent from the server and displays them on the screen.

[1726] input

[1727] Purchase offer information sent from the server.

[1728] output

[1729] Display of purchase suggestion information on user terminal.

[1730] Specific actions

[1731] The app displays a list of missing ingredients in a pop-up window. For example, "You're running low on mayonnaise. Here's a link to buy it." The user can click the link to buy the missing ingredients.

[1732] (Application example 1)

[1733] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1734] In modern households, it is important to efficiently manage food in the refrigerator and optimize consumption. However, currently, it is time-consuming to identify the presence of ingredients and identify what is missing, and it is difficult to suggest appropriate menus or replenish missing ingredients. Furthermore, particularly in the field of food delivery, there is no technology that automatically replenishes or suggests ingredients when a specific ingredient is missing. Therefore, there is a need for a system that can efficiently manage food in the refrigerator and even automatically replenish missing ingredients.

[1735] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1736] In this invention, the server includes an imaging device that photographs ingredients in the refrigerator, identification means that analyzes image data photographed by the imaging device to identify ingredients, suggestion means that proposes menus that can be created based on ingredient information identified by the identification means, purchase suggestion means that identifies ingredients that are lacking based on the menus proposed by the suggestion means and suggests their purchase, an auxiliary device that is installed in an automatic delivery machine associated with the refrigerator and that scans ingredients in the refrigerator with an imaging device, and replenishment means that the automatic delivery machine automatically replenishes ingredients that are lacking. This makes it possible to efficiently manage ingredients in the refrigerator and automatically replenish ingredients that are lacking.

[1737] An "imaging device that photographs food ingredients in a refrigerator" is a device that is installed inside a refrigerator and that captures image data of food ingredients in the refrigerator.

[1738] The "identification means" is a device or program that has the function of analyzing image data captured by the imaging device and identifying each food ingredient in the refrigerator.

[1739] The "suggestion means" is a device or program that has the function of generating and suggesting menus that can be made using ingredients based on the ingredient information identified by the identification means.

[1740] The "purchase suggestion means" is a device or program having the function of identifying ingredients that are in short supply from the ingredient information in the refrigerator based on the menu generated by the suggestion means and suggesting to the user that they be purchased.

[1741] A "delivery automaton" is a mechanical device, such as an automatically mobile robot, used to assist in the management of refrigerators.

[1742] The "auxiliary device" is an imaging device that is installed in the automated delivery machine and is used to scan ingredients in the refrigerator.

[1743] The "replenishment means" is a device or program that has the function of identifying a shortage of ingredients and automatically replenishing the ingredients when necessary.

[1744] The present invention is a system for efficiently managing ingredients in a refrigerator and automatically replenishing ingredients when they are in short supply. Detailed embodiments of the present invention will be described below.

[1745] The present invention includes an imaging device that photographs ingredients in a refrigerator, a server, a user terminal, and an automated delivery machine. The imaging device is installed inside the refrigerator and periodically photographs ingredients and sends the image data to the server. The server analyzes the received image data and has an identification means for identifying the ingredients.

[1746] The server generates a menu that can be prepared using the suggestion means based on the identified ingredient information and transmits it to the user's terminal. The user can check and select the suggested menu through the terminal. Based on the selected menu, the server's purchase suggestion means identifies ingredients that are in short supply and suggests their purchase to the user.

[1747] Furthermore, the delivery machine is equipped with an auxiliary device that can scan the ingredients in the refrigerator. As a means of replenishment, the delivery machine can automatically replenish any ingredients that are missing.

[1748] The system operates as follows.

[1749] 1. Scan food in the refrigerator

[1750] An image capture device installed in the delivery machine periodically scans the food items in the refrigerator and captures image data, for example, after the refrigerator door is closed.

[1751] 2. Analysis of image data

[1752] The server receives the image data sent from the imaging device and identifies the ingredients using image analysis software (e.g., TensorFlow or OpenCV). The identified ingredient information is recorded in a database.

[1753] 3. Creating a Menu

[1754] The server uses a generative AI model to generate menu options based on the identified ingredients. For example, if "tomato," "cheese," "lettuce," and "bread" are identified, the server will suggest options such as "tomato and cheese sandwich" or "lettuce and tomato salad."

[1755] 4. Identifying missing ingredients and suggesting purchases

[1756] The server generates a list of ingredients based on the user's menu selection and compares it with the refrigerator's inventory data. If any ingredients are missing, a purchase suggestion is sent to the user's device.

[1757] 5. Automatic replenishment function

[1758] The automated delivery machine follows instructions from the server, automatically orders missing ingredients from online supermarkets, etc., and replenishes the refrigerator if necessary.

[1759] As a concrete example, consider a situation where there are "tomatoes," "cheese," "lettuce," and "bread" in a refrigerator. The image capture device of the automated delivery machine takes a picture of these items, and the server analyzes and identifies the image data. Based on the identified ingredient information, the server generates and suggests menu items such as "tomato and cheese sandwich" or "lettuce and tomato salad." If the user selects "tomato and cheese sandwich," the server identifies the missing "mayonnaise" and suggests a link to purchase it from an online supermarket to the user. Furthermore, the automated delivery machine automatically purchases mayonnaise and replenishes the refrigerator.

[1760] Example prompt for generative AI model:

[1761] We will build an AI system that recognizes ingredients in the user's refrigerator and suggests ingredients that are missing. Please identify ingredients from the input image, compare them with the list of necessary ingredients, extract the missing ingredients, and generate a purchase link.

[1762] As described above, the present invention can improve the efficiency of food ingredient management in a refrigerator and realize automatic replenishment of ingredients that are running low.

[1763] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1764] Step 1:

[1765] Scan food in the refrigerator

[1766] Input: Image data from an imaging device (camera) installed inside the refrigerator

[1767] Specific operation: After the refrigerator door is closed, the image capture device takes an image of the inside of the refrigerator. The captured image data is sent to the server.

[1768] Output: Image data of ingredients in the refrigerator

[1769] Step 2:

[1770] Receiving and analyzing image data

[1771] Input: Image data of the inside of the refrigerator sent from the imaging device

[1772] Specific operation: The server receives the image data and identifies the ingredients using image analysis software (e.g., TensorFlow or OpenCV). The identified ingredient information is then recorded in a database.

[1773] Output: Identification information of ingredients (e.g., "tomato," "cheese," "lettuce," "bread")

[1774] Step 3:

[1775] Generate Menu

[1776] Input: Identification information of ingredients recorded on the server

[1777] Specific operation: The server uses the generative AI model to generate menu options based on the identified ingredients. The generated menu options are sent to the user's device.

[1778] Output: Possible menu items (e.g., "Tomato and cheese sandwich," "Lettuce and tomato salad")

[1779] Step 4:

[1780] Menu suggestions and selections

[1781] Input: Menu suggestions sent from the server to the user's device

[1782] Specific operation: The user's device displays the menu options received from the server, and the user selects the desired menu.

[1783] Output: The menu selected by the user

[1784] Step 5:

[1785] Identifying missing ingredients and suggesting purchases

[1786] Input: Menu selected by the user and ingredient information in the database

[1787] Specific operation: Based on the selected menu, the server generates a list of necessary ingredients and checks it against the refrigerator inventory in the database. If any ingredients are missing, it sends a purchase suggestion to the user's device.

[1788] Output: List of ingredients that are in short supply and purchase suggestions (e.g., "mayonnaise" purchase link)

[1789] Step 6:

[1790] Purchasing and Auto-Refill

[1791] Input: Purchase offers and delivery machine replenishment orders sent from the server to the user

[1792] Specific operation: When the user checks the purchase suggestion and clicks on the link to the online supermarket to purchase the missing ingredients, the delivery machine automatically replenishes the missing ingredients based on instructions from the server. The delivery machine is equipped with an auxiliary device that places the purchased ingredients in the refrigerator.

[1793] Output: Automatically replenished food in the refrigerator

[1794] Example prompt sentence:

[1795] We will build an AI system that recognizes ingredients in the user's refrigerator and suggests ingredients that are missing. Please identify ingredients from the input image, compare them with the list of necessary ingredients, extract the missing ingredients, and generate a purchase link.

[1796] This allows users to efficiently manage ingredients in their refrigerators and automatically replenish ingredients when they run out.

[1797] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1798] The present invention combines a system for efficiently utilizing ingredients in the refrigerator, saving household food costs and reducing cooking time, with an emotion engine that recognizes the user's emotions. The system includes the following main components:

[1799] System configuration

[1800] 1. Fridge Camera

[1801] This camera is installed inside the refrigerator and takes pictures of food ingredients. The camera takes pictures periodically after the refrigerator door is closed.

[1802] 2. Server

[1803] It includes means for receiving image data transmitted from the camera and executing an image recognition algorithm to identify ingredients from the data.

[1804] It is equipped with an emotion engine that recognizes the user's emotions and suggests appropriate menus based on those emotions.

[1805] 3. Terminal

[1806] A device used by a user, such as a smartphone or tablet, on which the application is installed.

[1807] Program processing

[1808] 1. Photographing food in the refrigerator

[1809] How the refrigerator camera works

[1810] The refrigerator camera periodically photographs the food inside the refrigerator after the door is closed.

[1811] 2. Analysis of image data

[1812] Server Operation

[1813] The server receives the image data sent from the refrigerator camera.

[1814] An image recognition algorithm is applied to the received image data to identify the ingredients in the refrigerator.

[1815] For example, the server identifies "tomato," "cheese," "lettuce," "bread," etc.

[1816] The identified ingredient information is recorded in a database.

[1817] 3. Create a composable menu

[1818] Server Operation

[1819] The server generates menu options that can be created using a generative AI model based on the identified ingredient information.

[1820] For example, using "tomato," "cheese," "lettuce," and "bread," it suggests "tomato and cheese sandwich" and "lettuce and tomato salad."

[1821] 4. Emotion Recognition by Emotion Engine

[1822] Server Operation

[1823] The server uses an emotion engine to recognize the user's emotions by analyzing the user's facial expressions and tone of voice.

[1824] The emotion engine also learns the user's past selection history and behavioral patterns to predict emotions.

[1825] 5. Menu suggestions based on emotions

[1826] Server Operation

[1827] The server proposes a menu according to the user's emotions based on the user's emotion information obtained from the emotion engine.

[1828] For example, if the server detects that the user is tired, it will suggest a menu that is simple and quick to prepare.

[1829] 6. Menu suggestions

[1830] Device behavior

[1831] The application on the device receives the menu information sent from the server and displays it to the user.

[1832] The user opens the app and sees the suggested menu.

[1833] 7. Check for missing ingredients and suggest purchasing them

[1834] Server Operation

[1835] The server generates a list of ingredients required based on the menu selected by the user.

[1836] The system compares the information with food stock information and extracts any ingredients that are in short supply.

[1837] 8. Displaying and executing purchase offers

[1838] Device behavior

[1839] The app on the device receives the information about the missing ingredients and the purchase suggestions sent from the server and displays them to the user.

[1840] User Actions

[1841] The user checks the list of ingredients they are short of and clicks on a link to an online supermarket to purchase the ingredients they are short of.

[1842] Specific examples

[1843] Using this system, let's assume that there are "tomatoes," "cheese," "lettuce," and "bread" in the refrigerator. The camera takes a picture of these ingredients, and the server analyzes the image data to identify them. Based on the identified ingredients, the server generates a menu such as "tomato and cheese sandwich" or "lettuce and tomato salad" and sends it to the terminal.

[1844] When a user opens the app and checks the suggested menu, the emotion engine analyzes the user's facial expressions and tone of voice. If it determines that the user is tired, it suggests a "tomato and cheese sandwich," which is easy and quick to make. When the user selects this suggestion, the server checks the ingredients needed for the "tomato and cheese sandwich" and identifies the missing "mayonnaise." The server then suggests the user purchase "mayonnaise" from an online supermarket, and the user can click a link to purchase it.

[1845] In this way, by combining emotion engines, it becomes possible to suggest menus that correspond to the user's emotions, enabling more individually tailored dish suggestions to be realized.

[1846] The processing flow will be explained below.

[1847] Step 1:

[1848] The refrigerator camera takes pictures of the inside of the refrigerator at regular intervals or after the refrigerator door is closed. The camera generates high-resolution image data and transmits the data to a server.

[1849] Step 2:

[1850] The server receives the image data sent from the refrigerator camera and stores it in a temporary storage area for saving the image data.

[1851] Step 3:

[1852] The server applies image recognition algorithms to identify ingredients in the image. For example, the server uses image recognition software to identify ingredients such as "tomato," "cheese," "lettuce," and "bread."

[1853] Step 4:

[1854] The server records the identified ingredient information in a database, which includes detailed information such as the type of ingredient, the quantity, and the date and time of identification.

[1855] Step 5:

[1856] The server uses a generative AI model to generate possible menu items based on the ingredient information stored in the database. For example, using "tomato," "cheese," "lettuce," and "bread," it can suggest "tomato and cheese sandwich" or "lettuce and tomato salad."

[1857] Step 6:

[1858] The server sends the generated menu options and necessary ingredient information to the terminal, which receives this information and prepares to display it to the user.

[1859] Step 7:

[1860] The app on the device receives the menu information sent from the server and displays it to the user. The user opens the app and checks the proposed menu.

[1861] Step 8:

[1862] The app on the device recognizes the user's facial expressions and tone of voice and sends that data to the emotion engine, which then analyzes facial expressions and voice to recognize the user's emotional state.

[1863] Step 9:

[1864] The emotion engine on the server recognizes the user's emotions based on the analysis results and adjusts the menu suggestions based on the user's emotional state. For example, if the system recognizes that the user is tired, it will prioritize suggestions of simple and quick meals.

[1865] Step 10:

[1866] The user selects the desired menu from the proposed menus, and the user's selection information is sent from the terminal to the server.

[1867] Step 11:

[1868] The server generates a list of ingredients required for the selected menu based on the user's selection, and compares it with ingredient inventory information to extract any ingredients that are in short supply.

[1869] Step 12:

[1870] The server compiles a list of ingredients that are missing, generates a link to the online supermarket's purchasing page, and sends this information to the device.

[1871] Step 13:

[1872] The app on the device receives the information about the missing ingredients and the purchase suggestions sent from the server and displays them to the user.

[1873] Step 14:

[1874] The user checks the list of ingredients they are short of and clicks on a link to an online supermarket, where they can purchase the ingredients they are short of.

[1875] Example 2

[1876] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1877] In modern households, efficiently utilizing the ingredients in the refrigerator to reduce food costs and cooking time is an important issue. However, it is difficult for users to properly manage the ingredients in their refrigerator, resulting in frequent waste and unused ingredients. In addition, menu suggestions are not tailored to the user's condition or mood, making it difficult for users to maintain their motivation to cook. Furthermore, if necessary ingredients are in short supply, users have to go shopping separately, which is a hassle.

[1878] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1879] In this invention, the server includes a camera that takes pictures of ingredients in the refrigerator, an image analysis unit that analyzes image data taken by the camera and identifies ingredients, a suggestion unit that suggests menus that can be made based on the ingredient information identified by the image analysis unit, a purchase suggestion unit that identifies ingredients that are lacking based on the menus suggested by the suggestion unit and suggests the purchase of those ingredients, and an emotion recognition unit that recognizes the user's emotions and suggests appropriate menus accordingly. This makes it possible to efficiently use ingredients in the refrigerator and reduce household food costs, and furthermore, by suggesting menus that correspond to the user's emotions, it is possible to improve motivation to cook and reduce the effort required.

[1880] "Photography device" refers to a device installed to photograph food items in a refrigerator.

[1881] "Image analysis means" refers to algorithms or software that analyzes image data acquired by the imaging device and identifies ingredients.

[1882] The "suggestion means" refers to a system for suggesting menus that can be prepared based on the ingredient information identified by the image analysis means.

[1883] The "purchase suggestion means" refers to a system for determining ingredients that are in short supply based on the menu suggested by the suggestion means and suggesting to the user to purchase those ingredients.

[1884] "Emotion recognition means" refers to algorithms or devices that analyze a user's facial expressions and tone of voice to recognize their emotions.

[1885] A "refrigerator" refers to a cooling device used in homes and businesses to preserve food.

[1886] "Ingredients" refers to food stored in the refrigerator for use in cooking.

[1887] "Image data" refers to digital data of photographs and videos captured by a photographing device.

[1888] "Ingredient information" refers to data relating to the type and quantity of ingredients identified by the image analysis means.

[1889] "Menu" refers to a list of dishes that can be made using ingredients.

[1890] The present invention combines a system for efficiently utilizing ingredients in a refrigerator, saving household food costs and reducing the effort required for cooking, with an emotion engine that recognizes the user's emotions. This system includes a photographing device for photographing ingredients in the refrigerator, image analysis means for analyzing image data and identifying ingredients, suggestion means for suggesting menus based on identified ingredient information, purchase suggestion means for identifying ingredients that are in short supply and suggesting their purchase, and emotion recognition means for recognizing the user's emotions and suggesting appropriate menus.

[1891] Specific hardware and software:

[1892] 1. Camera: A camera installed inside the refrigerator takes pictures periodically after the refrigerator door is closed. For example, a commercially available webcam or a special-purpose camera can be used.

[1893] 2. Image analysis means: Uses image recognition algorithms installed on the server. Specifically, open source image processing libraries and machine learning models (e.g., TensorFlow and OpenCV) can be used.

[1894] 3. Proposal method: Menu suggestions are made using a generative AI model. Specifically, a generative AI model that performs natural language processing (e.g., GPT-3) is used to generate a menu based on ingredient information.

[1895] 4. Purchase suggestion means: Using a database management system installed on the server, the missing ingredients are identified based on the identified ingredient information and the proposed menu, and a purchase suggestion is made.

[1896] 5. Emotion Recognition: An emotion engine is used to analyze the user's facial expressions and tone of voice. For example, a deep learning model (e.g., FaceNet) can be used for facial expression analysis, and a speech recognition engine (e.g., Google Cloud Speech-to-Text) can be used for voice analysis.

[1897] Operational flow and data calculation:

[1898] The server receives image data sent from the refrigerator camera and performs image analysis to identify ingredients from the data. The analyzed ingredient information is recorded in a database within the server. The server then uses a generative AI model based on this ingredient information to generate possible menu options. When the user uses the application to check the proposed menu, the emotion engine analyzes the user's facial expressions and tone of voice and presents a menu that corresponds to the user's emotions. Furthermore, based on the proposed menu, the server determines which ingredients are in short supply and suggests that the user purchase them.

[1899] Example 1:

[1900] If there are "tomatoes," "cheese," "lettuce," and "bread" in the refrigerator, the camera takes a picture of these ingredients, and the server analyzes the image data to identify them. Based on the identified ingredient information, the server generates menu items such as "tomato and cheese sandwich" and "lettuce and tomato salad," and sends them to the device. When the user opens the app and checks the suggested menu items, the emotion engine analyzes the user's facial expressions and tone of voice, and if it determines that the user is tired, it suggests "tomato and cheese sandwich," which is easy to make and can be made quickly. If the user selects this suggestion, the server checks the ingredients needed for the "tomato and cheese sandwich" and identifies the missing "mayonnaise." The server then suggests the user purchase "mayonnaise" from an online supermarket, and the user can click a link to purchase it.

[1901] Example 2:

[1902] Suggest a menu that is easy and quick to make using the "chicken," "carrots," and "onions" in the refrigerator. Also, if the user is tired, prioritize items that are easy and nutritious. An example of a prompt sentence is, "Suggest a menu that is easy and quick to make using the "chicken," "carrots," and "onions" in the refrigerator. Also, if the user is tired, prioritize items that are easy and nutritious."

[1903] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1904] Step 1:

[1905] Photographing food in the refrigerator

[1906] How the refrigerator camera works

[1907] The refrigerator camera periodically takes pictures of the inside of the refrigerator after the door is closed.

[1908] For example, a refrigerator is set to take a photo every hour.

[1909] Input: Image of the inside of the refrigerator

[1910] Output: Captured image data

[1911] Step 2:

[1912] Image data analysis

[1913] Server Operation

[1914] The server receives the image data sent from the refrigerator camera.

[1915] An image recognition algorithm is applied to the received image data to identify the ingredients inside.

[1916] For example, TensorFlow or OpenCV is used as an image recognition algorithm.

[1917] Input: Photographed image data

[1918] Data processing and calculation: Apply image recognition algorithms to identify the type of food

[1919] Output: Identified ingredients (e.g., "tomato," "cheese," "lettuce," "bread," etc.)

[1920] Step 3:

[1921] Generate composable menus

[1922] Server Operation

[1923] The server generates menu options that can be created using a generative AI model based on the identified ingredient information.

[1924] Specifically, we use a generative AI model (e.g., GPT-3) that performs natural language processing.

[1925] Input: Identified ingredient information

[1926] Data processing and data calculation: Generate menu suggestions using generative AI models

[1927] Output: Generated menu options (e.g., "tomato and cheese sandwich," "lettuce and tomato salad," etc.)

[1928] Step 4:

[1929] Emotion recognition by emotion recognition means

[1930] Server Operation

[1931] The server uses an emotion recognition means to recognize the user's emotion.

[1932] A deep learning model (e.g., FaceNet) is used for facial expression analysis, and a speech recognition engine (e.g., Google Cloud Speech-to-Text) is used for voice analysis.

[1933] Input: User's facial image and voice data

[1934] Data processing and data calculation: Applying emotion recognition algorithms to analyze user emotions

[1935] Output: Analyzed user's emotional information (e.g., "Tired," "Relaxed," etc.)

[1936] Step 5:

[1937] Menu suggestions based on emotions

[1938] Server Operation

[1939] The server proposes a menu according to the user's emotion based on the emotion information obtained from the emotion recognition means.

[1940] For example, if the user is tired, the system will prioritize suggesting menus that are simple and can be prepared in a short time.

[1941] Input: Analyzed user emotion information and generated menu suggestions

[1942] Data processing and data calculation: determining menu priorities based on emotional information

[1943] Output: Menu suggestions based on emotions (e.g., "Tomato and cheese sandwich")

[1944] Step 6:

[1945] Menu suggestions

[1946] Device behavior

[1947] The application on the device receives the menu information sent from the server and displays it to the user.

[1948] The user opens the app and sees the suggested menu.

[1949] For example, a smartphone or tablet app might display menu items such as "Tomato and Cheese Sandwich" or "Lettuce and Tomato Salad."

[1950] Input: Emotion-based menu suggestions

[1951] Output: Displayed menu information

[1952] Step 7:

[1953] Check for missing ingredients and suggest purchasing

[1954] Server Operation

[1955] The server generates a list of ingredients required based on the menu selected by the user.

[1956] The generated ingredient list is compared with the inventory information in the refrigerator to extract any ingredients that are missing.

[1957] For example, it may identify that a tomato and cheese sandwich requires mayonnaise, but that it is not in stock in the refrigerator.

[1958] Input: Selected menu, refrigerator inventory information

[1959] Data processing and calculation: Generate a list of necessary ingredients and extract missing ingredients

[1960] Output: Missing ingredient information (e.g., "mayonnaise")

[1961] Step 8:

[1962] View and execute purchase offers

[1963] Device behavior

[1964] The app on the device receives the information about the missing ingredients and the purchase suggestions sent from the server and displays them to the user.

[1965] For example, a message may appear saying, "We're low on mayonnaise. Would you like to purchase some from the online supermarket?"

[1966] Input: Missing ingredient information

[1967] Output: Displayed purchase suggestions

[1968] User Actions

[1969] The user checks the list of ingredients they are short of and clicks on a link to an online supermarket to purchase the ingredients they are short of.

[1970] For example, when you click, you are redirected to an online supermarket website, where the missing ingredients are automatically added to your cart.

[1971] Input: Purchase suggestion for ingredients that are in short supply

[1972] Output: The purchase is completed and the required ingredients are added back to inventory.

[1973] (Application example 2)

[1974] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1975] Conventional refrigerator management systems have technologies that identify ingredients in a refrigerator and suggest menus based on that information, but they are unable to suggest optimal menus because they are unable to make suggestions based on the user's emotions or physical condition. In particular, in busy daily lives and under stressful circumstances, menu suggestions based on the user's emotions are more desirable. Therefore, the present invention aims to provide a system that recognizes the user's emotions and suggests optimal menus based on emotional information.

[1976] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1977] In this invention, the server includes a photographing device that photographs ingredients in the refrigerator, an identification means that analyzes image data photographed by the photographing device to identify ingredients, a suggestion means that suggests menus that can be made based on the ingredient information identified by the identification means, a purchase suggestion means that identifies ingredients that are in short supply based on the menu suggested by the suggestion means and suggests their purchase, an emotion recognition means that recognizes the user's emotion, and an emotion-based suggestion means that suggests menus based on the emotion information recognized by the emotion recognition means. This makes it possible to suggest optimal menus according to the user's emotions.

[1978] The "photography device" is a device used to take pictures of ingredients in the refrigerator.

[1979] The "identification means" is a technology that analyzes image data acquired by the imaging device and identifies ingredients.

[1980] The "proposing means" is a mechanism for providing the user with menus that can be prepared based on the ingredient information identified by the identifying means.

[1981] The "purchase suggestion means" is a mechanism for checking ingredients required for the menu suggested by the suggestion means and prompting the user to purchase any ingredients that are missing.

[1982] "Emotion recognition means" is a technology that analyzes the user's facial expressions, tone of voice, etc. to identify the user's emotions.

[1983] The "emotion-based suggestion means" is a mechanism for providing the user with an optimal menu based on the user's emotional information identified by the emotion recognition means.

[1984] The system embodying this invention efficiently utilizes ingredients in the refrigerator, proposes menus based on the user's emotions, and assists in purchasing ingredients. The components that make up this system and their specific operations are described below.

[1985] Main components of the system

[1986] 1. Imaging equipment

[1987] The camera is installed inside the refrigerator and takes pictures of the food periodically after the refrigerator door is closed. The camera can also be a smartphone camera.

[1988] 2. Identification Method

[1989] The server receives the image data sent from the camera and identifies the ingredients using an image recognition algorithm (e.g., TensorFlow). The identified ingredient information is recorded in a database (e.g., Firebase Firestore).

[1990] 3. Proposal method

[1991] The server generates a menu that can be created using a generative AI model (e.g., GPT-3) based on the identified ingredients. For example, if the identified ingredients are "tomato," "cheese," "lettuce," and "bread," the following prompt sentence can be used:

[1992] Please suggest a menu that can be made if tomatoes, cheese, lettuce, and bread are available.

[1993] The generated menu options are sent to the device (smartphone or tablet).

[1994] 4. Emotion recognition means

[1995] The server uses the smartphone's camera and microphone to record the user's facial expressions and tone of voice, and recognizes the user's emotions using an emotion recognition engine (e.g., Affectiva SDK). The results of emotion recognition are fed back to the proposal method.

[1996] 5. Emotion-based suggestion tools

[1997] Based on the emotion information, the server adjusts the menu candidates generated by the suggestion tool and generates a new menu that suits the user's state. The server also generates prompts using an AI model:

[1998] If you have tomato, cheese, lettuce, and bread, suggest a menu recommendation for a situation where the user is tired.

[1999] For example, if the system detects that the user is tired, it will suggest a quick and easy recipe such as a tomato and cheese sandwich.

[2000] 6. Purchase suggestion method

[2001] Based on the menu proposed by the suggestion means, the server generates a list of necessary ingredients. The identification means compares this with ingredient inventory information and extracts any missing ingredients. The missing ingredients are then suggested to the user so that they can be purchased through a delivery service (e.g., Uber Eats API).

[2002] Specific examples

[2003] As a concrete example of using this system, let's consider the case where a housewife takes a picture of the inside of her refrigerator using her smartphone. The camera identifies "tomato," "cheese," "lettuce," and "bread," and the server uses this information to suggest a menu. When the user confirms the suggested menu, the emotion recognition means analyzes the user's facial expression and tone of voice and determines that the user is tired. In this case, the server generates and suggests a menu that is easy to prepare in a short time.

[2004] For example, if a user selects a "tomato and cheese sandwich," the server double-checks the ingredients for that menu and finds that "mayonnaise" is missing. The server then provides a link to purchase "mayonnaise" from a delivery service.

[2005] This makes it possible to suggest appropriate menus based on the user's emotions and to suggest purchasing ingredients that are in short supply, thereby reducing household food costs and the effort required for cooking.

[2006] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2007] Step 1:

[2008] Photographing ingredients using a photography device

[2009] When the user closes the refrigerator door, the camera (smartphone camera) automatically takes a picture of the inside of the refrigerator.

[2010] Input: Image of the inside of the refrigerator

[2011] Output: Photographed food image data

[2012] Step 2:

[2013] Image data analysis and food ingredient identification

[2014] The server receives the image data sent from the camera and identifies the ingredients using an image recognition algorithm (e.g., TensorFlow).

[2015] Input: Photographed food image data

[2016] Data processing / calculation: Identifying ingredients using image recognition models

[2017] Output: List of identified ingredients

[2018] Step 3:

[2019] Storage of identified ingredient information

[2020] The server stores the identified ingredient information in a database (e.g., Firebase Firestore).

[2021] Input: Identified ingredient list

[2022] Data processing / calculation: Writing data to the database

[2023] Output: Saved ingredient information

[2024] Step 4:

[2025] User Emotion Recognition

[2026] The device's camera and microphone are used to capture the user's facial expressions and voice, and the server uses an emotion recognition engine (e.g., Affectiva SDK) to analyze the user's emotions.

[2027] Input: User facial and voice data

[2028] Data processing / calculation: Emotion analysis using emotion recognition models

[2029] Output: User's emotional information

[2030] Step 5:

[2031] Generate menu suggestions

[2032] Based on the identified ingredient information and emotional information, the server uses a generative AI model (e.g., GPT-3) to generate prompts and suggest appropriate menu items.

[2033] Input: Identified ingredients list, user's emotional information

[2034] Data processing / calculation: Menu generation using generative AI models

[2035] Output: Suggested menu options

[2036] Step 6:

[2037] View Menu Suggestions

[2038] The application on the terminal receives the menu information sent from the server and displays it to the user.

[2039] Input: Suggested menu options

[2040] Data processing / calculation: Displaying menu information to the user

[2041] Output: Menu suggestions displayed on the device

[2042] Step 7:

[2043] Check for missing ingredients

[2044] Based on the menu selected by the user, the server generates a list of ingredients needed and compares it with the ingredient inventory information in the database. If there are any ingredients that are missing, it extracts them.

[2045] Input: Selected menu, identified ingredient list

[2046] Data processing / calculation: Creating a list of necessary ingredients, extracting ingredients that are lacking

[2047] Output: List of missing ingredients

[2048] Step 8:

[2049] Generate and display purchase offers

[2050] The server creates a purchasing suggestion based on the missing ingredients and displays it to the user through the device application. The user can then click a link on the device to purchase the missing ingredients through a delivery service (e.g., Uber Eats API).

[2051] Input: List of ingredients in short supply

[2052] Data processing / calculation: generating purchase proposals

[2053] Output: Purchase suggestions displayed on the device, links to delivery services

[2054] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2055] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2056] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2057] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2058] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2059] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2060] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2061] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2062] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2063] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[2064] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[2065] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[2066] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[2067] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[2068] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[2069] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[2070] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[2071] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[2072] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[2073] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[2074] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[2075] The following is further disclosed regarding the above embodiment.

[2076] (Claim 1)

[2077] A camera that takes pictures of the food in the refrigerator,

[2078] an identification means for analyzing image data captured by the camera to identify ingredients;

[2079] a suggestion means for suggesting a menu that can be prepared based on the ingredient information identified by the identification means;

[2080] The system includes a purchase suggestion means for identifying ingredients that are in short supply and suggesting their purchase based on the menu suggested by the suggestion means.

[2081] (Claim 2)

[2082] 2. The system according to claim 1, wherein the suggestion means also suggests a menu according to cooking time and the user's preferences.

[2083] (Claim 3)

[2084] 2. The system according to claim 1, wherein the camera is a camera that takes pictures periodically after the refrigerator door is closed.

[2085] "Example 1"

[2086] (Claim 1)

[2087] a photographing device for photographing food ingredients in a refrigerator;

[2088] means for receiving and storing image data captured by the imaging device;

[2089] an identification means for applying an image recognition algorithm to the received and stored image data to identify ingredients;

[2090] a means for recording the ingredient information identified by the identification means in a database;

[2091] A means for generating menu candidates that can be created using a generative AI model based on the identified ingredient information;

[2092] means for transmitting the generated menu candidates to a user terminal;

[2093] means for displaying the generated menu candidates on a user terminal;

[2094] means for generating a list of ingredients required based on the menu selected by the user;

[2095] a means for comparing the list of necessary ingredients with information on ingredient inventory in the refrigerator and extracting ingredients that are in short supply;

[2096] means for transmitting the list of missing ingredients and a link to a purchasing page to a user terminal;

[2097] A system including:

[2098] (Claim 2)

[2099] The system of claim 1, wherein a prompt sentence is input to the generative AI model to generate menu candidates.

[2100] (Claim 3)

[2101] 2. The system according to claim 1, wherein the photographing device is a device that periodically photographs the refrigerator after the door is closed.

[2102] "Application Example 1"

[2103] (Claim 1)

[2104] an imaging device that photographs food ingredients in the refrigerator;

[2105] an identification means for analyzing image data captured by the imaging device to identify ingredients;

[2106] a suggestion means for suggesting a menu that can be prepared based on the ingredient information identified by the identification means;

[2107] a purchase suggestion means for identifying ingredients that are in short supply based on the menu suggested by the suggestion means and suggesting their purchase;

[2108] an auxiliary device for scanning ingredients in the refrigerator using an imaging device installed in an automated delivery machine associated with the refrigerator;

[2109] A means for automatically replenishing the missing ingredients by the automatic delivery machine

[2110] A system including:

[2111] (Claim 2)

[2112] 2. The system according to claim 1, wherein the suggestion means also suggests a menu according to the cooking time required and the user's preferences.

[2113] (Claim 3)

[2114] 2. The system according to claim 1, wherein the imaging device is a camera that takes images periodically after the refrigerator door is closed.

[2115] "Example 2: Combining Emotion Engines"

[2116] (Claim 1)

[2117] a photographing device for photographing food ingredients in a refrigerator;

[2118] an image analysis means for analyzing image data captured by the imaging device to identify ingredients;

[2119] a suggestion means for suggesting a menu that can be prepared based on the ingredient information identified by the image analysis means;

[2120] a purchase suggestion means for identifying ingredients that are in short supply based on the menu suggested by the suggestion means and suggesting their purchase;

[2121] emotion recognition means for recognizing a user's emotion and suggesting an appropriate menu accordingly;

[2122] A system including:

[2123] (Claim 2)

[2124] 2. The system according to claim 1, wherein the suggestion means also suggests a menu according to cooking time and the user's preferences.

[2125] (Claim 3)

[2126] 2. The system according to claim 1, wherein the photographing device is a device that periodically photographs the refrigerator after the door is closed.

[2127] "Application example 2 when combining emotion engines"

[2128] (Claim 1)

[2129] a photographing device for photographing food ingredients in a refrigerator;

[2130] an identification means for analyzing image data captured by the imaging device and identifying ingredients;

[2131] a suggestion means for suggesting a menu that can be prepared based on the ingredient information identified by the identification means;

[2132] a purchase suggestion means for identifying ingredients that are in short supply based on the menu suggested by the suggestion means and suggesting their purchase;

[2133] emotion recognition means for recognizing an emotion of a user;

[2134] The system further includes an emotion-based suggestion means for suggesting a menu based on the emotion information recognized by the emotion recognition means.

[2135] (Claim 2)

[2136] 2. The system according to claim 1, wherein the suggestion means suggests a menu according to emotion information in addition to cooking time and user preferences.

[2137] (Claim 3)

[2138] 2. The system according to claim 1, wherein the photographing device is a device that periodically photographs the refrigerator after the door is closed. [Explanation of symbols]

[2139] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A camera that takes pictures of the food in the refrigerator, an identification means for analyzing image data captured by the camera to identify ingredients; a suggestion means for suggesting a menu that can be prepared based on the ingredient information identified by the identification means; The system includes a purchase suggestion means for identifying ingredients that are in short supply and suggesting their purchase based on the menu suggested by the suggestion means.

2. The system according to claim 1 , wherein the suggestion means also suggests a menu according to cooking time and the user's preferences.

3. 2. The system according to claim 1, wherein the camera is a camera that takes pictures periodically after the refrigerator door is closed.

Images