System

Abstract

A system is provided.SOLUTION: A system comprising: means for recognizing ingredients in a refrigerator by analyzing an image acquired by a camera installed in the refrigerator; means for proposing a recipe that can be made only with the ingredients in the refrigerator based on the recognized ingredients; means for proposing a recipe that can be made by adding additional ingredients to the recognized ingredients; means for learning a user's past recipe selection history and proposing a recipe based on the user's preference; and means for displaying a recipe based on the ingredients in the refrigerator, a recipe requiring additional ingredients, and a recipe based on the user's preference on a user interface.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Deciding what to cook at home is a major concern for many people. They often waste food because they are unable to make effective use of the ingredients in their refrigerators. Furthermore, they tend to cook only a limited number of dishes, leaving them with few opportunities to try new dishes. To solve these problems, a system is needed that efficiently uses the ingredients in the refrigerator and quickly provides recipes suited to the user. [Means for solving the problem]

[0005] The present invention provides a system that recognizes ingredients in a refrigerator by analyzing images captured by a camera installed inside the refrigerator. Based on the recognized ingredients, the system suggests recipes that can be made using only the ingredients in the refrigerator and recipes that can be made by adding additional ingredients. The system also learns the user's past recipe selection history and suggests recipes based on the user's preferences. The system also includes a means for suggesting new recipes based on the user's past recipe selection history. This reduces the time it takes to decide on a dish, prevents waste of ingredients in the refrigerator, and enables the provision of a wide variety of dishes.

[0006] A "camera" is a device that uses light to capture images and acquires the image data in digital form.

[0007] "Analysis" is the process of extracting specific information (e.g., type and quantity of ingredients) from the acquired image data.

[0008] "Ingredients" is a general term for fresh foods and processed foods used to prepare a dish.

[0009] "Recognition" is the process of identifying a specific ingredient from image data obtained through analysis.

[0010] "Suggestion" is the act of showing the optimal recipe to the user based on the analysis results and learning data.

[0011] A "recipe" is a set of instructions that lists the ingredients and steps for making a particular dish.

[0012] "User" refers to a person who uses the system to receive recipe suggestions.

[0013] "Preferences" refers to individual taste trends based on the user's past recipe selections and cooking preferences.

[0014] "Learning" is the data processing that the system performs to predict user preferences and trends based on past data and selection history.

[0015] "User interface" refers to the screen display and operating means by which a user interacts with a system.

[0016] A "list" is an ordered list of items such as recipes and additional ingredients.

[0017] "Extra ingredients" refers to ingredients that are not in your refrigerator but are needed to make a particular dish.

[0018] "Selection history" refers to a record of recipes and dishes that a user has selected in the past.

[0019] A "refrigerator" refers to a household electrical appliance used to store food in a low-temperature environment.

[0020] "Image data" refers to visual information captured by a camera and stored in digital format. [Brief explanation of the drawings]

[0021] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8]FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0022] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0023] First, the terms used in the following description will be explained.

[0024] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0025] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0026] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0027] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0028] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0029] [First embodiment]

[0030] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0031] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0032] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0033] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0034] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0035] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0036] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0037] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0038] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0039] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0040] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0041] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0042] This invention relates to a system that uses an image analysis system with a camera installed inside a refrigerator to recognize ingredients in the refrigerator and then suggests optimal recipes to the user based on that information. This system is composed of a server, a terminal, and user interaction.

[0043] Server Processing

[0044] 1. Acquiring camera images

[0045] A camera installed inside the refrigerator periodically takes pictures of the inside of the refrigerator, and the server captures and receives these pictures.

[0046] 2. Image analysis and food ingredient recognition

[0047] The server preprocesses the received image data to remove noise and adjust the resolution, then applies image analysis algorithms (e.g., deep learning models for object recognition) to identify the ingredients in the refrigerator.

[0048] The type and quantity of recognized ingredients are stored in a database, and the freshness and expiration date of the ingredients are also recorded.

[0049] 3. Referencing the recipe database

[0050] The server references the ingredient database to search for recipes that can be made using only the ingredients in the refrigerator. It also searches for recipes that require additional ingredients and generates a list of the additional ingredients required.

[0051] 4. Learning user preference data

[0052] The server collects the user's past recipe selection history and rating data, and trains a machine learning model to predict the user's preferences. Based on this data, it suggests recipes that suit the user's tastes and new recipes that the user has not tried yet.

[0053] 5. Generating Recipe Suggestions

[0054] The server creates the following recipe list:

[0055] Recipes that can be made using only ingredients in your refrigerator

[0056] Recipes that require additional ingredients and a list of the additional ingredients required

[0057] Recipes based on user preferences

[0058] New recipes

[0059] 6. Data transmission

[0060] The server sends the generated recipe list to the terminal.

[0061] Terminal handling

[0062] 1. Display the recipe list

[0063] The terminal displays the recipe list received from the server on the user interface.

[0064] 2. View detailed recipe information

[0065] When a user selects a recipe, the device displays detailed information about the recipe, including ingredients, steps, and any additional ingredients needed.

[0066] 3. Generate a shopping list

[0067] The terminal displays a shopping list of additional ingredients based on the recipe selected by the user.

[0068] User operations

[0069] 1. Recipe selection

[0070] The user selects the recipe they want to make from the recipe list displayed on the terminal.

[0071] 2. Check and purchase ingredients

[0072] Users check the ingredients needed based on the recipe and purchase any missing ingredients.

[0073] 3. Enter your feedback

[0074] After the user executes the selected recipe, they input their feedback into the device, which then sends this data to the server and updates the user preference database.

[0075] Specific examples

[0076] Example 1: Recipes that can be made using only ingredients in your refrigerator

[0077] situation

[0078] The refrigerator contains "eggs, milk, butter, flour, and bacon."

[0079] Server Processing

[0080] These ingredients are recognized using image analysis and the ingredient database is updated.

[0081] Search the database for "hotcake recipes" and "bacon and egg recipes" and add them to the list.

[0082] Terminal display

[0083] "Hotcakes" and "Bacon and eggs" are displayed to the user as options.

[0084] Example 2: Recipe that requires additional ingredients

[0085] situation

[0086] The refrigerator contains "tomatoes, lettuce, cheese, and bread."

[0087] Server Processing

[0088] These ingredients are recognized using image analysis and the ingredient database is updated.

[0089] Search the database for "sandwich recipes" and see that they call for ham.

[0090] Terminal display

[0091] Display "Sandwich (additional ham required)" as a suggestion to the user.

[0092] Example 3: Recipes based on user preferences

[0093] situation

[0094] The user has often selected "Japanese food" in the past.

[0095] Server Processing

[0096] It learns user preference data and searches for Japanese recipes (e.g., "meat and potato stew" and "miso soup").

[0097] As a new suggestion, add "lasagna" to the list of Western food options.

[0098] Terminal display

[0099] The user is prompted to select from "nikujaga," "miso soup," and "lasagna."

[0100] This system allows users to maximize the use of ingredients in their refrigerators while also allowing them to try new dishes as needed, reducing the time it takes to decide what to cook, reducing food waste, and enabling a wider variety of meals.

[0101] The processing flow will be explained below.

[0102] Step 1:

[0103] Camera image acquisition

[0104] The server periodically acquires images from a camera installed inside the refrigerator.

[0105] The camera takes pictures of each shelf and compartment in the refrigerator and generates image data.

[0106] Step 2:

[0107] Image data preprocessing

[0108] The server performs noise removal and resolution adjustment on the acquired image data.

[0109] The preprocessed image data is saved in preparation for analysis.

[0110] Step 3:

[0111] Image analysis and food ingredient recognition

[0112] The server uses deep learning models for object recognition to analyze the pre-processed image data.

[0113] The server uses the model to recognize ingredients in the image and identify the type and quantity of each ingredient.

[0114] Step 4:

[0115] Ingredient database update

[0116] The server stores the recognized ingredient information in an ingredient database.

[0117] The freshness and expiration date of ingredients are also recorded at the same time, keeping the database up to date.

[0118] Step 5:

[0119] Recipe database reference

[0120] The server refers to the ingredient database and searches for recipes that can be made using only the ingredients in the refrigerator.

[0121] It also searches for recipes that require additional ingredients and generates a list of the required additional ingredients.

[0122] Step 6:

[0123] Learning user preference data

[0124] The server collects the user's past recipe selection history and evaluation data.

[0125] The server uses this data to train machine learning models to predict user preferences.

[0126] Step 7:

[0127] Creating a recipe list

[0128] The server creates the following recipe list:

[0129] Recipes that can be made using only ingredients in your refrigerator

[0130] Recipes that require additional ingredients and a list of the additional ingredients required

[0131] Recipes based on user preferences

[0132] New recipes

[0133] Step 8:

[0134] Sending data

[0135] The server sends the generated recipe list to the terminal.

[0136] Step 9:

[0137] Displaying the recipe list

[0138] The terminal displays the recipe list received from the server on the user interface.

[0139] To make it easier for users to select a recipe, the app also displays an overview of each recipe, its difficulty level, and the time required.

[0140] Step 10:

[0141] Viewing detailed recipe information

[0142] When a user selects a particular recipe, the device displays detailed information about that recipe, including ingredients, steps, and any additional ingredients needed.

[0143] Step 11:

[0144] Generate a shopping list

[0145] The terminal displays a shopping list of ingredients that are missing based on the recipe selected by the user.

[0146] The user checks the shopping list and prepares to purchase the necessary ingredients.

[0147] Step 12:

[0148] Enter your feedback

[0149] After the user executes the selected recipe, the results are input as feedback to the terminal.

[0150] Feedback includes the quality of the dish, its difficulty, and individual ratings.

[0151] Step 13:

[0152] Sending feedback data

[0153] The terminal transmits feedback data from the user to the server.

[0154] Step 14:

[0155] Database Update

[0156] The server updates the user's preference database based on the received feedback data.

[0157] Use the updated data to improve your next recipe suggestion.

[0158] Through this series of processing steps, the system will suggest optimal recipes to the user, enabling them to make efficient use of the ingredients in their refrigerator.

[0159] Example 1

[0160] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0161] In modern life, people need to effectively manage the ingredients in their refrigerators and use them up without wasting any. However, visually checking the contents of the refrigerator and planning recipes based on that information is time-consuming and labor-intensive. It is also difficult to suggest optimal recipes based on the user's preferences and past cooking history. Furthermore, it is time-consuming to have to check and purchase ingredients that are missing. To solve these issues, a system is needed that can efficiently recognize ingredients in the refrigerator and suggest optimal recipes to users.

[0162] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0163] In this invention, the server includes means for analyzing images captured by an image capture device installed in the refrigerator and recognizing food in the refrigerator, means for suggesting dishes that can be made using only the food in the refrigerator based on the recognized food, means for suggesting dishes that can be made by adding additional ingredients to the recognized food, means for learning the user's past dish selection history and suggesting dishes based on the user's preferences, means for predicting the user's preferences using a machine learning model and suggesting dishes based on the predicted preferences, and means for displaying on a display device dishes based on the food in the refrigerator, dishes requiring additional ingredients, and dishes based on the user's preferences. This allows the user to efficiently manage ingredients in the refrigerator and receive optimal recipe suggestions, thereby reducing the effort required for cooking and reducing food waste.

[0164] An "image capture device" is a device that is installed inside a refrigerator and is used to capture images.

[0165] "Analysis" refers to the act of processing acquired image data and extracting necessary information.

[0166] "Food" refers to all food ingredients and beverages stored in the refrigerator.

[0167] The "means of suggestion" is a means of selecting appropriate recipes and information based on predetermined conditions and providing them to the user.

[0168] "Cuisine" refers to food prepared using specific ingredients.

[0169] "Additional ingredients" are new ingredients or seasonings that are needed in addition to the ingredients already in the refrigerator.

[0170] "User preferences" refer to preferences and favorite foods that are inferred based on the user's past selections and ratings of dishes.

[0171] A "machine learning model" refers to an algorithm or mathematical model that analyzes large amounts of data and finds patterns.

[0172] "Prediction methods" are methods for predicting future trends and results based on past data and analytical results.

[0173] A "display device" is a device that functions as a user interface and visually presents various pieces of information.

[0174] The present invention relates to a system that efficiently manages food in a refrigerator and suggests optimal recipes to users. The system is composed of a server, a terminal, and user interaction. The following describes in detail an embodiment of the present invention.

[0175] Server configuration and operation

[0176] The server receives and analyzes images captured by the image capture device installed inside the refrigerator. Specifically, it uses the following hardware and software:

[0177] Hardware:

[0178] Camera installed inside the refrigerator

[0179] Server itself (equipped with a high-performance CPU, sufficient memory, and network interface)

[0180] software:

[0181] Image processing library (e.g. OpenCV)

[0182] Object recognition library (e.g., TENSORFLOW(R), YOLO)

[0183] Database management system (e.g., MySQL®)

[0184] Machine learning libraries (e.g., scikit-learn)

[0185] The server performs the following steps to recognize the food in the refrigerator:

[0186] 1. Image Acquisition: A camera installed inside the refrigerator periodically takes images and sends them to a server, which receives the image data via Wi-Fi or a wired connection.

[0187] 2. Image analysis: The received image data is first pre-processed to remove noise and adjust the resolution, then an object recognition algorithm is used to identify food in the image.

[0188] 3. Storage in database: Information such as the type, quantity, freshness, and expiration date of the recognized food is stored in a database.

[0189] Recipe suggestion function

[0190] The server suggests suitable recipes to the user by:

[0191] 1. Recipes that can be made using only food in the refrigerator: Search for relevant recipes based on food information in the database.

[0192] 2. Recipes that require additional ingredients: Search for recipes that can be made by combining ingredients in the refrigerator with additional ingredients, and generate a list of the additional ingredients required.

[0193] 3. Recipes based on user preferences: We collect users' past recipe selection history and rating data, and use machine learning models to predict their preferences. Based on this data, we then suggest optimal recipes.

[0194] Terminal configuration and operation

[0195] The terminal has the following user interface features:

[0196] 1. Displaying the recipe list: The recipe list received from the server is displayed in the user interface. The recipe list is dynamically generated and displayed using HTML and JavaScript (registered trademark).

[0197] 2. Displaying detailed recipe information: When a user selects a recipe, detailed information (ingredients, steps, required additional ingredients, etc.) is displayed. This is done by retrieving data from the server using Ajax.

[0198] 3. Generate shopping list: Based on the recipe selected by the user, additional ingredients are displayed as a shopping list.

[0199] User operations

[0200] Users use this system through the following operations:

[0201] 1. Recipe selection: Select the recipe you want to make from the recipe list displayed on your device.

[0202] 2. Check and purchase ingredients: Check the ingredients needed based on the recipe and purchase any that are missing.

[0203] 3. Entering feedback: After executing the selected recipe, the user enters the feedback into the device, which then sends this data to the server and updates the user's preference database.

[0204] Specific examples

[0205] Here are some concrete usage examples:

[0206] Example of a recipe that can be made using only ingredients in the refrigerator: If the refrigerator contains "eggs, milk, butter, flour, and bacon," the server will recognize these ingredients and suggest recipes for pancakes or bacon and eggs.

[0207] Example of a recipe that requires additional ingredients: If the refrigerator contains "tomatoes, lettuce, cheese, and bread," the server suggests a sandwich and informs the user that additional ham is needed.

[0208] Example of recipes based on user preferences: If the user has frequently selected "Japanese food" in the past, the server will suggest recipes for Nikujaga (meat and potato stew) and Miso soup. It will also add Lasagna to the list as a new suggestion.

[0209] Examples of prompt statements

[0210] "Please tell me a recipe I can make using the eggs, milk, butter, flour, and bacon I have in my fridge."

[0211] This invention allows users to efficiently manage food in their refrigerators and reduce the effort required for cooking. In addition, the system continuously learns through feedback, enabling it to suggest more accurate recipes.

[0212] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0213] Server Processing

[0214] Step 1: Acquire an image

[0215] Operation: A camera placed inside the refrigerator periodically takes images of the inside of the refrigerator.

[0216] Input: Image data captured by the camera.

[0217] How it works: The camera takes pictures of the inside of the refrigerator and sends them over WiFi or a wired connection to a server, which then downloads the image data using HTTP requests or the FTP protocol.

[0218] Output: Image data received on the server side.

[0219] Step 2: Image analysis

[0220] Operation: The server preprocesses the received image data and recognizes the food.

[0221] Input: The image data received in step 1.

[0222] How it works: The server uses image processing libraries such as OpenCV to remove noise and adjust the resolution, then uses TensorFlow to apply an object recognition model to identify and classify food items in the image.

[0223] Output: Recognized food types and quantities.

[0224] Step 3: Saving to the database

[0225] Operation: The server stores the recognized food data in a database.

[0226] Input: Food type and quantity data obtained in step 2.

[0227] How it works: The server connects to a database management system such as MySQL and uses the "INSERT" statement to record food names, quantities, freshness information, etc. into the database.

[0228] Output: Food data stored in a database.

[0229] Step 4: Recipe suggestions

[0230] Operation: The server refers to the food database to search and suggest recipes suitable for the user.

[0231] Input: Food data saved in step 3.

[0232] How it works: The server uses an SQL query to search the recipe table in the database to find recipes that match the food list in the refrigerator. It also searches for recipes that require additional ingredients and generates a list of the additional ingredients in JSON format.

[0233] Output: A list of recipes that can be made using only the food in the refrigerator, and a list of recipes that require additional ingredients.

[0234] Step 5: Learning user preferences

[0235] Operation: The server trains a machine learning model based on the user's past recipe selection history.

[0236] Input: User's past recipe selection history data.

[0237] How it works: The server uses pandas to read historical data and scikit-learn to train a machine learning model that predicts user preferences and generates recipes based on them.

[0238] Output: A list of recipes based on the user's preferences.

[0239] Step 6: Sending data

[0240] Operation: The server sends a recipe list to the device.

[0241] Input: The recipe list generated in step 4 and step 5.

[0242] How it works: The server sends an HTTP POST request to the device via a RESTful API, sending a list of recipes in JSON format.

[0243] Output: Recipe list sent to the device.

[0244] Terminal handling

[0245] Step 1: View the recipe list

[0246] Operation: The terminal displays the recipe list received from the server on the user interface.

[0247] Input: Recipe list sent from the server.

[0248] How it works: Using HTML and JavaScript, the received recipe list is dynamically generated and displayed on a web page.

[0249] Output: The recipe list displayed in the user interface.

[0250] Step 2: Viewing Recipe Details

[0251] Action: The user selects a recipe.

[0252] Input: User's choice.

[0253] How it works: The device uses Ajax to retrieve detailed information about the selected recipe from the server and displays it, including ingredients, steps, and any additional ingredients needed.

[0254] Output: Detailed information about the recipe.

[0255] Step 3: Generate a shopping list

[0256] Operation: Generate a shopping list based on the user's selected recipes.

[0257] Input: The recipe data selected by the user.

[0258] What it does: Dynamically generates a list of additional ingredients using JavaScript and displays it as a shopping list.

[0259] Output: Purchase list.

[0260] User operations

[0261] Step 1: Recipe Selection

[0262] Operation: The user selects the recipe they want to make from the recipe list displayed on the device.

[0263] Input: A list of recipes displayed on the terminal.

[0264] Action: The user selects a recipe via a select box or click event.

[0265] Output: The selected recipe.

[0266] Step 2: Check and purchase ingredients

[0267] Operation: The user checks the ingredients needed based on the recipe and purchases any missing ingredients.

[0268] Input: Recipe details.

[0269] What it does: The user looks at the shopping list and notes or buys the ingredients they need.

[0270] Output: A list of ingredients confirmed.

[0271] Step 3: Provide feedback

[0272] Operation: The user executes the selected recipe and then enters their feedback into the terminal.

[0273] Input: User feedback data.

[0274] Behavior: A feedback input form is provided, and after the user enters their rating and comments, it is sent to the server.

[0275] Output: Feedback data sent to the server.

[0276] This allows users to efficiently manage the ingredients in their refrigerators and receive appropriate recipe suggestions, reducing the effort required for cooking and reducing food waste. In addition, the system continues to learn through feedback, allowing it to make even more accurate suggestions.

[0277] (Application example 1)

[0278] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0279] Conventional refrigerator food management systems only allowed users to track the ingredients in their refrigerators and suggested recipes. Furthermore, users had to go shopping separately to get ingredients they were running low on, which was inconvenient. Furthermore, the system lacked the ability to suggest new recipes based on the user's preferences. This limited the opportunities for users to efficiently use the ingredients in their refrigerators and try new recipes.

[0280] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0281] In this invention, the server includes means for analyzing images captured by a camera installed inside the refrigerator and recognizing ingredients in the refrigerator, means for suggesting recipes that can be made using only the ingredients in the refrigerator based on the recognized ingredients, means for suggesting recipes that can be made by adding additional ingredients to the recognized ingredients, means for learning the user's past recipe selection history and suggesting recipes based on the user's preferences, means for displaying on a user interface recipes based on the ingredients in the refrigerator, recipes that require additional ingredients, and recipes based on the user's preferences, means for recognizing ingredients that are missing from a recipe selected by the user and generating a list of those ingredients, and means for enabling the user to order missing ingredients for delivery based on the generated list of missing ingredients. This allows the user to use the ingredients in the refrigerator efficiently, expand recipe options, and easily procure missing ingredients.

[0282] A "camera installed inside a refrigerator" is a camera device that is attached inside a refrigerator and captures video or images of the interior.

[0283] "Analyzing images" is the process of recognizing and identifying specific objects or elements based on the acquired image data.

[0284] "Means for recognizing ingredients" refers to a method or device for identifying specific foods or ingredients in a refrigerator through image analysis.

[0285] The "means for suggesting recipes that can be made using only ingredients in the refrigerator" is a method or device that presents to the user recipes for dishes that do not require additional ingredients, based on recognized ingredients.

[0286] The "means for suggesting recipes that can be made by adding additional ingredients" is a method or device that presents to the user how to make dishes that can be made by adding missing ingredients to the recognized ingredients.

[0287] "Learning a user's past recipe selection history" is the process of collecting data about recipes previously selected by a user and analyzing that data.

[0288] The "means for proposing recipes based on user preferences" is a method or device for presenting recipes that match the collected and learned preferences of the user.

[0289] The "means for displaying on a user interface" refers to a method or apparatus for displaying suggested recipes and information on the screen of a device used by a user.

[0290] The "means for recognizing missing ingredients and generating a list thereof" is a method or device that identifies ingredients that are required for a recipe selected by the user but are not present in the refrigerator and lists them.

[0291] A "means for enabling delivery ordering of missing ingredients" is a method or device that enables identified missing ingredients to be purchased directly via an online ordering system.

[0292] An embodiment of the present invention uses hardware such as a server, a smartphone, and a camera inside a refrigerator, and software such as an image analysis algorithm, a recipe database, a user interface, and delivery service collaboration software.

[0293] The server analyzes images captured by a camera installed inside the refrigerator and recognizes the ingredients inside. A deep learning model is used as the image analysis algorithm for recognition. For example, frameworks such as TensorFlow and PyTorch can be used. Preprocessing involves noise removal and resolution adjustment, followed by object recognition to identify the ingredients. Information about the recognized ingredients is stored in a database.

[0294] The server then suggests recipes that can be made using only the ingredients in the refrigerator based on the recognized ingredients. To do this, the server refers to the ingredient database and searches for relevant recipes.

[0295] The server then suggests recipes that can be made by adding additional ingredients to the recognized ingredients, searches the database for the necessary additional ingredients, and generates a list of these ingredients, which is then displayed together with the recipe selected by the user.

[0296] The server also learns the user's past recipe selection history and suggests recipes based on the user's preferences. By using a machine learning model, the server can learn the user's preference patterns and provide recipes tailored to their preferences.

[0297] The user interface displays these recipes on the smartphone screen. The user interacts with the server through the smartphone and selects from the suggested recipes. A list of ingredients missing from the selected recipe is generated, and this information can be used to order delivery via delivery service integration software.

[0298] For example, a user can take a photo of the ingredients in their refrigerator with their smartphone and send the image to a server via the app. The server analyzes the image and recognizes the ingredients. The server then suggests recipes that can be made with the recognized ingredients, as well as dishes that require additional ingredients, and displays these to the user via a user interface. When the user selects a recipe, a list of missing ingredients is displayed, allowing them to order delivery.

[0299] Specifically, this involves suggesting new dishes (such as Western food if the user mostly chooses Japanese food) based on the user's past recipe preferences, and ordering missing ingredients through a delivery service.

[0300] An example of a specific prompt using a generative AI model is, "Based on the results of image analysis of the inside of the refrigerator, please suggest recipes that can be made with the ingredients I currently have. Also, please generate a list of ingredients that I am running low on."

[0301] This allows users to efficiently use ingredients in their refrigerator, expand their recipe options, and easily procure ingredients that they are running low on.

[0302] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0303] Step 1:

[0304] The user takes a picture of the inside of the refrigerator with the smartphone camera and sends it to the server via the app. The input is image data of the inside of the refrigerator, and the output is the image data sent to the server. The user records the state of the inside of the refrigerator using the smartphone camera and uploads the image to the server using the send button on the app.

[0305] Step 2:

[0306] The server preprocesses the received image data. The input is the transmitted image data, and the output is the preprocessed image data. Specific operations include noise reduction and resolution adjustment. For example, it applies a filter to reduce image noise and resizes the image to a resolution suitable for analysis.

[0307] Step 3:

[0308] The server analyzes the preprocessed image data using a deep learning model to recognize the ingredients in the refrigerator. The input is the preprocessed image data, and the output is a list of recognized ingredients. Specifically, it uses an object detection algorithm to identify each ingredient in the image and extract the name and quantity of each ingredient.

[0309] Step 4:

[0310] The server saves the recognized ingredient list in a database. The input is the recognized ingredient list, and the output is the latest ingredient database. Specifically, the operation involves connecting to the database and adding or updating new ingredient information.

[0311] Step 5:

[0312] The server references the ingredient database to search for recipes that can be made using only the ingredients in the refrigerator. The input is the ingredient database, and the output is a list of matching recipes. Specifically, it executes a database query to retrieve recipes based on the currently available ingredients.

[0313] Step 6:

[0314] The server searches for recipes that require additional ingredients and generates a list of missing ingredients. The input is an ingredient database and a list of recognized ingredients, and the output is a list of recipes with the additional ingredients and a list of missing ingredients. Specifically, the server performs a database query to identify ingredients that are missing from existing ingredients and then lists them.

[0315] Step 7:

[0316] The server learns the user's past recipe selection history and suggests recipes based on the user's preferences. The input is the user's selection history data, and the output is a personalized recipe list. Specifically, it uses a machine learning algorithm to analyze past data and identify the user's preference patterns.

[0317] Step 8:

[0318] The device displays the recipe list received from the server on the user interface. The input is the recipe list sent from the server, and the output is the recipe list displayed on the smartphone screen. The user can select the desired recipe from the displayed list.

[0319] Step 9:

[0320] The system checks the ingredients missing from the recipe selected by the user and displays a shopping list of the additional ingredients needed. The input is the selected recipe and the output is the shopping list. The user checks the displayed list and identifies the missing ingredients.

[0321] Step 10:

[0322] The terminal allows users to order missing ingredients for delivery based on the shopping list. The input is the shopping list, and the output is the execution of the delivery order. Specifically, the order details are sent via the delivery service's API, and the ingredients are delivered to the user.

[0323] Through the above processing steps, the user can efficiently utilize ingredients in the refrigerator, find suitable recipes, and easily procure ingredients that are in short supply.

[0324] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0325] This invention is a system that combines an image analysis system using a camera installed inside the refrigerator with an emotion engine that recognizes the user's emotions to make optimal use of ingredients in the refrigerator and suggest recipes based on the user's preferences and emotions. This system is composed of interactions between a server, terminals, and users, and realizes more personalized recipe suggestions.

[0326] Server Processing

[0327] 1. Acquiring camera images

[0328] The server periodically acquires images from a camera installed inside the refrigerator.

[0329] The camera takes pictures of each shelf and compartment inside the refrigerator and generates image data.

[0330] 2. Image data preprocessing

[0331] The server preprocesses the acquired image data, removing noise and adjusting the resolution.

[0332] The preprocessed image data is used for analysis.

[0333] 3. Image analysis and food ingredient recognition

[0334] The server analyzes the image data using a deep learning model to identify the type and quantity of ingredients.

[0335] The recognition results are stored in a food ingredient database, and freshness and expiration date are also recorded.

[0336] 4. Referencing the recipe database

[0337] The server refers to the ingredient database and searches for recipes that can be made using only the ingredients in the refrigerator.

[0338] It also searches for recipes that require additional ingredients and generates a list of the ingredients needed.

[0339] 5. Learning User Preference Data

[0340] The server collects past recipe selection history and rating data to train a preference prediction model.

[0341] Used to suggest recipes based on the user's preferences.

[0342] 6. Use of Emotion Engine

[0343] The server uses an emotion engine to recognize the user's emotions in real time.

[0344] The recognized emotion data is reflected in recipe suggestions.

[0345] 7. Generating Recipe Suggestions

[0346] The server generates the following recipe list:

[0347] Recipes that can be made using only ingredients in your refrigerator

[0348] Recipes and lists that require additional ingredients

[0349] Recipes based on user preferences

[0350] New recipes

[0351] Recipes based on user sentiment

[0352] 8. Data transmission

[0353] The server sends the generated recipe list to the terminal.

[0354] Terminal handling

[0355] 1. Display the recipe list

[0356] The terminal displays the recipe list received from the server on the user interface.

[0357] It also displays an overview of the recipe, its difficulty level, and the time required.

[0358] 2. View detailed recipe information

[0359] When a user selects a particular recipe, the device displays detailed information, including ingredients, steps, and any additional ingredients needed.

[0360] 3. Generate a shopping list

[0361] The device displays a shopping list of additional ingredients based on the selected recipe.

[0362] The user checks the list and prepares to purchase the necessary ingredients.

[0363] User operations

[0364] 1. Recipe selection

[0365] The user selects the recipe they want to make from the recipe list displayed on the terminal.

[0366] 2. Check and purchase ingredients

[0367] Users check the ingredients needed based on the recipe and purchase any missing ingredients.

[0368] 3. Providing Emotion Data

[0369] The system recognizes emotions from the user's facial expressions and voice in real time and transmits them to the server.

[0370] 4. Enter your feedback

[0371] After executing the selected recipe, the user inputs the results and impressions as feedback into the terminal.

[0372] Specific examples

[0373] Example 1: Recipe suggestions based on ingredients in the refrigerator and emotions

[0374] situation

[0375] The refrigerator contains "eggs, milk, butter, flour, and bacon," and the user is relaxed.

[0376] Server Processing

[0377] Ingredients are recognized through image analysis and the ingredient database is updated.

[0378] The emotion engine recognizes the user's relaxed state.

[0379] It searches the database for "hotcakes" and "bacon and eggs" and recommends "hotcakes" that best suits the user's condition.

[0380] Terminal display

[0381] "Hotcakes" is displayed to the user, along with detailed recipe information.

[0382] Example 2: User preferences and new recipe suggestions

[0383] situation

[0384] The user has often selected "Japanese food" in the past and is now interested in new cuisine.

[0385] Server Processing

[0386] It learns user preference data and suggests Japanese dishes such as "meat and potato stew" and "miso soup."

[0387] Add "lasagna" as a new suggestion.

[0388] Terminal display

[0389] It displays dishes such as "nikujaga" (beef and potato stew), "miso soup" and "lasagna" to encourage people to try new dishes.

[0390] This system allows users to make the most of the ingredients in their refrigerator and easily find dishes that suit their tastes and emotions, significantly reducing the time it takes to decide what to cook, minimizing food waste, and increasing the variety of their daily meals.

[0391] The processing flow will be explained below.

[0392] Step 1:

[0393] Camera image acquisition

[0394] The server periodically acquires images from a camera installed inside the refrigerator.

[0395] The camera takes pictures of each shelf and compartment inside the refrigerator and generates image data.

[0396] Step 2:

[0397] Image data preprocessing

[0398] The server preprocesses the acquired image data, removing noise and adjusting the resolution.

[0399] Save the preprocessed image data.

[0400] Step 3:

[0401] Image analysis and food ingredient recognition

[0402] The server analyzes the preprocessed image data using a deep learning model to identify the type and quantity of ingredients.

[0403] The results of ingredient recognition are stored in an ingredient database, and freshness and expiration date are also recorded.

[0404] Step 4:

[0405] Acquiring emotion data

[0406] The device captures the user's facial expressions and voice in real time to recognize their emotions.

[0407] An emotion engine analyzes the captured data to determine the user's current emotional state.

[0408] Step 5:

[0409] Recipe database reference

[0410] The server refers to the ingredient database and searches for recipes that can be made using only the ingredients in the refrigerator.

[0411] It also searches for recipes that require additional ingredients and generates a list of the ingredients needed.

[0412] Step 6:

[0413] Learning user preference data

[0414] The server collects past recipe selection history and rating data to train the machine learning model.

[0415] Predict recipes based on user preference data.

[0416] Step 7:

[0417] Emotional Data Integration

[0418] The server collects the recognized emotion data and adjusts the recipe suggestion options based on it.

[0419] If the emotion indicates relaxation, customizations such as suggesting simple and relaxing dishes are performed.

[0420] Step 8:

[0421] Creating a recipe list

[0422] The server generates the following recipe list:

[0423] Recipes that can be made using only ingredients in your refrigerator

[0424] Recipes that require additional ingredients and a list of the additional ingredients required

[0425] Recipes based on user preferences

[0426] New recipes

[0427] Recipes based on user sentiment

[0428] Step 9:

[0429] Sending data

[0430] The server sends the generated recipe list to the terminal.

[0431] Step 10:

[0432] Displaying the recipe list

[0433] The terminal displays the recipe list received from the server on the user interface.

[0434] It also displays an overview of the recipe, its difficulty level, and the time required.

[0435] Step 11:

[0436] Viewing detailed recipe information

[0437] When a user selects a particular recipe, the device displays detailed information, including ingredients, steps, and any additional ingredients needed.

[0438] Step 12:

[0439] Generate a shopping list

[0440] The device displays a shopping list of additional ingredients based on the selected recipe.

[0441] The user checks the list and prepares to purchase the necessary ingredients.

[0442] Step 13:

[0443] Cooking and getting emotional feedback

[0444] The user executes the selected recipe and inputs emotional feedback into the device during the process.

[0445] The emotion engine collects the user's emotion data during and after cooking.

[0446] Step 14:

[0447] Sending feedback data

[0448] The terminal transmits feedback data from the user to the server.

[0449] Step 15:

[0450] Database Update

[0451] The server updates the user's preference database and emotion database based on the received feedback data.

[0452] Use the updated data to improve your next recipe suggestion.

[0453] This system allows users to make the most of the ingredients in their refrigerator and easily find dishes that suit their tastes and emotions. It also utilizes real-time emotional data to provide more personalized recipe suggestions.

[0454] Example 2

[0455] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0456] Conventional refrigerator ingredient management and recipe suggestion systems are limited to suggesting recipes that can be made using only the ingredients in the refrigerator or recipes that require additional ingredients. This does not allow for personalized suggestions based on the user's preferences and emotions, leading to users always making the same dishes and wasting ingredients. Furthermore, the lack of increased cooking options tends to decrease users' motivation to cook. Therefore, the present invention aims to provide more personalized cooking suggestions by taking the user's preferences and emotions into account, thereby improving the enjoyment and efficiency of cooking.

[0457] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0458] In this invention, the server includes means for analyzing images captured by a camera installed inside the refrigerator and recognizing ingredients in the refrigerator, means for suggesting recipes that can be made using only the ingredients in the refrigerator based on the recognized ingredients, means for suggesting recipes that can be made by adding additional ingredients to the recognized ingredients, means for learning the user's past recipe selection history and suggesting recipes based on the user's preferences, means for recognizing the user's emotional state and suggesting recipes based on the emotions, and means for displaying on a user interface recipes based on the ingredients in the refrigerator, recipes that require additional ingredients, recipes based on the user's preferences, and recipes based on the user's emotions. This allows the user to effectively use the ingredients in the refrigerator and select recipes that suit their preferences and emotions.

[0459] A "camera installed inside a refrigerator" is a device installed inside the refrigerator for taking images of food items inside the refrigerator.

[0460] "Means for analyzing images and recognizing ingredients in the refrigerator" refers to technology and algorithms that use images acquired from a camera to identify the type and quantity of ingredients present in the refrigerator.

[0461] The "means for suggesting recipes that can be made using only ingredients in the refrigerator" is a function that searches for recipes that can be made using only the ingredients in the recognized refrigerator and suggests them to the user.

[0462] "Means for suggesting recipes that can be made by adding additional ingredients" is a function that provides recipes for dishes that can be made by adding other ingredients in addition to the ingredients in the refrigerator.

[0463] "Means for learning a user's past recipe selection history and suggesting recipes based on the user's preferences" refers to technology that collects and analyzes recipes previously selected by the user and their evaluation data, and then suggests recipes that suit the user's preferences based on that data.

[0464] "Means for recognizing the user's emotional state and suggesting recipes based on those emotions" is a function that analyzes the user's emotions in real time from their facial expressions, voice, etc., and suggests recipes that are appropriate for those emotions.

[0465] "Means for displaying on the user interface" refers to the technology or mechanism for displaying suggested recipes and related information on the screen of the device used by the user.

[0466] This invention is a system that combines an image analysis system using a camera installed inside the refrigerator with an emotion engine that recognizes the user's emotions to make optimal use of ingredients in the refrigerator and suggest recipes based on the user's preferences and emotions. This system is composed of interactions between a server, terminals, and users, and realizes more personalized recipe suggestions.

[0467] The server periodically acquires images from a camera installed inside the refrigerator and analyzes the image data to recognize ingredients. This analysis uses a deep learning model. The acquired image data is preprocessed to remove noise and adjust the resolution. This improves the accuracy of the image data, enabling accurate recognition of ingredients.

[0468] Once the ingredients are recognized, the server updates the ingredient database, recording the type, quantity, freshness, and expiration date of each ingredient. Based on this database, the server searches for recipes that can be made using only the ingredients in the refrigerator and suggests them to the user. It also searches for recipes that can be made by adding additional ingredients to the ingredients in the refrigerator and generates a list of the necessary ingredients.

[0469] Furthermore, the server collects the user's past recipe selection history and rating data to train a preference prediction model. This model allows it to suggest recipes based on the user's preferences. Meanwhile, the server uses an emotion engine to recognize the user's emotions in real time and reflects this emotion data in the recipe suggestions. This allows it to suggest simple dishes when the user is relaxed and quick dishes when the user is busy.

[0470] As a concrete example, suppose the refrigerator contains "eggs, milk, butter, flour, and bacon," and the user is in a relaxed state. When the server recognizes these ingredients and the emotion engine understands the user's relaxed state, it searches for recipes for "hotcakes" and "bacon and eggs." Among them, it suggests "hotcakes," which matches the user's relaxed state. The server sends this suggested recipe list to the terminal, which then displays it on the user interface.

[0471] The device accepts input from the user, and when a specific recipe is selected, it displays detailed information about it, allowing the user to check the ingredients needed and cooking steps. If additional ingredients are needed, they are displayed as a shopping list, which the user can use to purchase them. For example, the tomatoes needed to make a "tomato omelette" are added to the shopping list.

[0472] The user selects the recipe they want to make from a list of recipes displayed on the device, checks the ingredients needed, and purchases them. Emotional data from the user's facial expressions and voice is also collected in real time and sent to the server. After cooking, the user enters the results and their impressions as feedback into the device, and this data is used to suggest future recipes.

[0473] As an example of a prompt sentence, if you enter "I'm wondering what to have for dinner tonight," the system will suggest the best recipe based on the ingredients in the refrigerator and the user's preferences and emotions.

[0474] As described above, the system of the present invention makes effective use of ingredients in the refrigerator and proposes recipes that match the user's tastes and emotions, thereby reducing the time it takes to decide on a dish and minimizing the waste of ingredients, making everyday meals more varied and enjoyable.

[0475] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0476] Server Processing Steps

[0477] Step 1: Acquire camera images

[0478] The server periodically acquires images from a camera installed inside the refrigerator. The input is image data captured by the camera, which is sent to the server. The output is the acquired image data, which is stored in the server. Subsequent image processing is performed based on this image data.

[0479] Step 2: Preprocessing the image data

[0480] The server performs pre-processing on the acquired image data. Specific operations include noise removal, resolution adjustment, and image cropping. The input is raw image data acquired from the camera, and the output is clear, pre-processed image data. This processing improves the accuracy of image analysis.

[0481] Step 3: Image analysis and ingredient recognition

[0482] The server uses a deep learning model to recognize ingredients in the refrigerator from preprocessed image data. The input is the preprocessed image data, and the output is data recording the type and quantity of recognized ingredients. This data is stored in an ingredient database. Specifically, it identifies ingredients such as eggs, milk, and butter.

[0483] Step 4: Referencing the recipe database

[0484] The server refers to the ingredient database and searches for recipes that can be made using only the ingredients in the refrigerator. At the same time, it searches for recipes that require additional ingredients and generates a list of the necessary ingredients. The input is the recognition data from the ingredient database, and the output is a list of recommended recipes and a list of additional ingredients. Specifically, it includes recipes that can be made with "eggs and milk," and recipes that can be made by adding "tomatoes."

[0485] Step 5: Learning user preference data

[0486] The server collects the user's past recipe selection history and rating data and trains a preference prediction model. The input is the past recipe selection history and rating data, and the output is a model that learns the user's preference patterns. This model makes it possible to suggest recipes that suit the user's preferences.

[0487] Step 6: Use the Emotion Engine

[0488] The server uses an emotion engine to recognize the user's emotions in real time. The input is emotional data obtained from the user's facial expressions and voice, and the output is the recognized emotional state. Based on this, a recipe appropriate to the user's emotions is selected. For example, if the user is relaxed, a recipe that suits that state is suggested.

[0489] Step 7: Generate recipe suggestions

[0490] The server generates the following recipe list based on the analysis data so far:

[0491] Recipes that can be made using only ingredients in your refrigerator

[0492] Recipes and lists that require additional ingredients

[0493] Recipes based on user preferences

[0494] Recipes based on user sentiment

[0495] The input is food recognition data, preference data, and emotion data, and the output is a list of recommended recipes.

[0496] Step 8: Sending Data

[0497] The server sends the generated recipe list to the terminal. The input is the recommended recipe list, and the output is the list sent to the terminal. This list includes a summary of the recipe, the ingredients required, and cooking instructions.

[0498] Terminal processing steps

[0499] Step 1: View the recipe list

[0500] The terminal displays the recipe list received from the server on the user interface. The input is the recipe list from the server, and the output is the recipe information displayed on the user interface, including the recipe summary, difficulty, and time.

[0501] Step 2: Viewing Recipe Details

[0502] When the user selects a particular recipe, the device displays its details. The input is the user's selected recipe, and the output is details such as ingredients, steps, and any additional ingredients needed.

[0503] Step 3: Generate a shopping list

[0504] The terminal creates and displays a shopping list based on the selected recipe. The input is the selected recipe data, and the output is a shopping list listing the additional ingredients needed. The user can check this list and prepare to purchase the necessary ingredients.

[0505] User operation steps

[0506] Step 1: Recipe Selection

[0507] The user selects the recipe they want to make from the recipe list displayed on the terminal. The input is the recipe list displayed on the terminal, and the output is the selected recipe.

[0508] Step 2: Check and purchase ingredients

[0509] The user checks the ingredients needed based on the recipe and purchases any missing ingredients. The input is the list of ingredients for the recipe, and the output is the purchased ingredients.

[0510] Step 3: Provide emotion data

[0511] The system recognizes emotions from the user's facial expressions and voice in real time and transmits them to the server. The input is the user's facial expressions and voice, and the output is emotional data sent to the server.

[0512] Step 4: Provide feedback

[0513] After executing the selected recipe, the user inputs the results and their impressions as feedback into the terminal. The input is the recipe execution results and impressions, and the output is feedback data sent to the server.

[0514] (Application example 2)

[0515] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0516] In recent years, there has been a demand for making effective use of ingredients in the refrigerator and reducing waste, but it has been difficult for users to find recipes that suit their preferences and emotions. Furthermore, when using delivery services, food options are not optimized because suggestions are not made that take into account the ingredients in the refrigerator or the user's emotions. This has led to problems such as wasted ingredients in the refrigerator and inappropriate food delivery suggestions. The present invention aims to solve these problems by efficiently utilizing ingredients in the refrigerator and realizing appropriate recipe and delivery suggestions that match the user's emotions and preferences.

[0517] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0518] In this invention, the server includes means for analyzing images captured by a camera installed inside the refrigerator and recognizing ingredients in the refrigerator, means for suggesting recipes that can be made using only the ingredients in the refrigerator based on the recognized ingredients, means for suggesting recipes that can be made by adding additional ingredients to the recognized ingredients, means for learning the user's past recipe selection history and suggesting recipes based on the user's preferences, means for analyzing the user's emotions and suggesting recipes based on the analyzed emotions, and means for displaying on a user interface recipes based on the ingredients in the refrigerator, recipes that require additional ingredients, recipes based on the user's preferences, and recipes based on the user's emotions.This makes it possible to efficiently use ingredients in the refrigerator without waste and to suggest personalized recipes and delivery options that suit the user's emotions and preferences.

[0519] A "camera installed inside a refrigerator" is a device that takes pictures of the inside of a refrigerator and obtains information about the internal condition and ingredients.

[0520] "Means for analyzing images and recognizing ingredients in the refrigerator" refers to algorithms or software that process image data captured by a camera and identify the types and quantities of ingredients contained within.

[0521] "Means to suggest recipes that can be made using only ingredients in the refrigerator" is a function that presents users with recipes for dishes that can be made without using any additional ingredients, based on the information of recognized ingredients.

[0522] "Means for suggesting recipes that can be made by adding additional ingredients" is a function that presents users with recipes for dishes that can be made by taking into account the ingredients in the refrigerator and other ingredients that can be added from outside.

[0523] "Means for learning the user's past recipe selection history and suggesting recipes based on the user's preferences" is a function that analyzes the history of recipes that the user has selected in the past and suggests recipes that suit the user's preferences based on the results.

[0524] "Means for analyzing the user's emotions and suggesting recipes based on the analyzed emotions" is a function that evaluates the user's emotions from their facial expressions, voice, etc., and suggests the optimal recipe based on their emotional state.

[0525] "Means for displaying information in a user interface" refers to a display or screen display device that provides information in a form that can be seen by the user.

[0526] The "means for generating a list of additional ingredients" is a function that lists ingredients required for the recipe selected by the user that are in short supply in the refrigerator and presents them as a shopping list.

[0527] "Means to suggest new recipes" is a function that suggests recipes for new genres or types of food that the user has not tried before, based on the user's past recipe selection history.

[0528] "Means for suggesting ingredients based on the user's purchase history" is a function that analyzes the history of ingredients that the user has purchased and suggests new ingredients that should be purchased based on that history.

[0529] The system for realizing this invention consists of an algorithm that analyzes images from a camera installed inside the refrigerator and recognizes ingredients, an engine that analyzes the user's preferences and emotions, and a server and terminal that suggests recipes and delivery options based on this data.

[0530] Server Processing

[0531] The server first periodically acquires image data from a camera installed inside the refrigerator and preprocesses the images. The hardware used includes a camera and server, and software such as OpenCV and TensorFlow. OpenCV is used to remove noise and adjust the resolution of the images, and TensorFlow is used to analyze the images using a deep learning model and recognize ingredients.

[0532] The system then stores the recognized ingredient data in an ingredient database, and uses that data to reference a recipe database to search for recipes that can be made using only the ingredients in the refrigerator, as well as recipes that require additional ingredients. It also learns the user's preferences using the user's past recipe selection history and rating data. Furthermore, it uses an emotion engine to analyze the user's emotional state in real time and collects emotion data.

[0533] We combine these data to generate the following recipe list:

[0534] Recipes that can be made using only ingredients in your refrigerator

[0535] Recipes and lists that require additional ingredients

[0536] Recipes based on user preferences

[0537] New recipes

[0538] Recipes based on user sentiment

[0539] Terminal handling

[0540] The device displays the recipe list sent from the server on a user interface. A smartphone or tablet is used for the display, and detailed information such as a recipe summary, difficulty level, and required time is also provided. When the user selects a specific recipe, a list of additional ingredients is generated and presented to the user.

[0541] User operations

[0542] The user selects the recipe they want to make from a list of recipes displayed on their device and purchases additional ingredients as needed. The system also recognizes the user's emotions in real time from their facial expressions and voice, and sends this data to the server. After executing the recipe, the system provides feedback on the results and impressions, which are used to learn the user's preference data.

[0543] Specific examples

[0544] As a concrete example, consider a situation where the user has "eggs, milk, and cheese" in the refrigerator and is feeling "stressed." Based on this, the server can suggest recipes and delivery options such as a cheese omelet or stress-relieving herbal tea. Also, based on the user's past preference for "Japanese food," the server can suggest a new type of "Western food" recipe this time.

[0545] Prompt Sentence Examples

[0546] "Please recognize ingredients from camera images inside the refrigerator and analyze emotions from the user's facial expressions and voice. Based on the results, please generate a program that will suggest the best recipes and food delivery options for the user."

[0547] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0548] Processing Steps

[0549] Step 1:

[0550] The server periodically acquires image data from a camera installed inside the refrigerator. The image data captured by the camera is provided as input and is used for image pre-processing in the next step.

[0551] Step 2:

[0552] The server preprocesses the acquired image data. Here, OpenCV is used to perform basic image processing such as removing noise from the image and adjusting the resolution. The input is raw image data from the camera, and the output is image data that has been denoised and adjusted for resolution.

[0553] Step 3:

[0554] The server analyzes the preprocessed image data to recognize ingredients. A deep learning model using TensorFlow identifies the type and quantity of ingredients and stores the results in an ingredient database. The input is the preprocessed image data, and the output is the type and quantity data of ingredients.

[0555] Step 4:

[0556] Based on the information in the ingredient database, the server searches for recipes that can be made using only the ingredients in the refrigerator and recipes that require additional ingredients. It also references the recipe database and generates a list of ingredients required for each recipe. The input is the type and quantity data of ingredients, and the output is a list of recipes and a list of required ingredients.

[0557] Step 5:

[0558] The server learns the user's past recipe selection history and suggests recipes based on their preferences. Using the past selection data as learning data, it trains a preference prediction model and suggests recipes that match the user's preferences. The input is the past recipe selection history, and the output is a recipe based on the user's preferences.

[0559] Step 6:

[0560] The server uses an emotion engine to analyze the user's emotions. It uses facial expression images and voice data obtained by a camera and microphone as input to identify the user's emotional state. The input is the user's facial expression images and voice data, and the output is emotion data.

[0561] Step 7:

[0562] The server generates a recipe list by integrating recipes based on ingredients in the refrigerator, recipes that require additional ingredients, recipes based on preferences, and recipes based on emotions. This creates personalized recipe suggestions that take various data into consideration. The input is ingredient data, preference data, and emotion data, and the output is a recipe list.

[0563] Step 8:

[0564] The server sends the generated recipe list to the terminal, along with detailed recipe information and a list of required ingredients. The input is the recipe list, and the output is the recipe data sent to the terminal.

[0565] Step 9:

[0566] The terminal displays the recipe list received from the server, and when the user selects a specific recipe, displays its detailed information. If necessary, it also generates and displays additional ingredient lists. The input is the recipe list received from the server, and the output is the display on the user interface.

[0567] Step 10:

[0568] The user selects the recipe they want to make from the recipe list displayed on the device, and checks and purchases any additional ingredients they need. The user provides emotional data from their facial expressions and voice, and also inputs feedback. The input is the displayed recipe list and a list of additional ingredients, and the output is the selected recipe and feedback data.

[0569] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0570] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0571] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0572] [Second embodiment]

[0573] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0574] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0575] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0576] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0577] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0578] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0579] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0580] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0581] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0582] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0583] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0584] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0585] This invention relates to a system that uses an image analysis system with a camera installed inside a refrigerator to recognize ingredients in the refrigerator and then suggests optimal recipes to the user based on that information. This system is composed of a server, a terminal, and user interaction.

[0586] Server Processing

[0587] 1. Acquiring camera images

[0588] A camera installed inside the refrigerator periodically takes pictures of the inside of the refrigerator, and the server captures and receives these pictures.

[0589] 2. Image analysis and food ingredient recognition

[0590] The server preprocesses the received image data to remove noise and adjust the resolution, then applies image analysis algorithms (e.g., deep learning models for object recognition) to identify the ingredients in the refrigerator.

[0591] The type and quantity of recognized ingredients are stored in a database, and the freshness and expiration date of the ingredients are also recorded.

[0592] 3. Referencing the recipe database

[0593] The server references the ingredient database to search for recipes that can be made using only the ingredients in the refrigerator. It also searches for recipes that require additional ingredients and generates a list of the additional ingredients required.

[0594] 4. Learning user preference data

[0595] The server collects the user's past recipe selection history and rating data, and trains a machine learning model to predict the user's preferences. Based on this data, it suggests recipes that suit the user's tastes and new recipes that the user has not tried yet.

[0596] 5. Generating Recipe Suggestions

[0597] The server creates the following recipe list:

[0598] Recipes that can be made using only ingredients in your refrigerator

[0599] Recipes that require additional ingredients and a list of the additional ingredients required

[0600] Recipes based on user preferences

[0601] New recipes

[0602] 6. Data transmission

[0603] The server sends the generated recipe list to the terminal.

[0604] Terminal handling

[0605] 1. Display the recipe list

[0606] The terminal displays the recipe list received from the server on the user interface.

[0607] 2. View detailed recipe information

[0608] When a user selects a recipe, the device displays detailed information about the recipe, including ingredients, steps, and any additional ingredients needed.

[0609] 3. Generate a shopping list

[0610] The terminal displays a shopping list of additional ingredients based on the recipe selected by the user.

[0611] User operations

[0612] 1. Recipe selection

[0613] The user selects the recipe they want to make from the recipe list displayed on the terminal.

[0614] 2. Check and purchase ingredients

[0615] Users check the ingredients needed based on the recipe and purchase any missing ingredients.

[0616] 3. Enter your feedback

[0617] After the user executes the selected recipe, they input their feedback into the device, which then sends this data to the server and updates the user preference database.

[0618] Specific examples

[0619] Example 1: Recipes that can be made using only ingredients in your refrigerator

[0620] situation

[0621] The refrigerator contains "eggs, milk, butter, flour, and bacon."

[0622] Server Processing

[0623] These ingredients are recognized using image analysis and the ingredient database is updated.

[0624] Search the database for "hotcake recipes" and "bacon and egg recipes" and add them to the list.

[0625] Terminal display

[0626] "Hotcakes" and "Bacon and eggs" are displayed to the user as options.

[0627] Example 2: Recipe that requires additional ingredients

[0628] situation

[0629] The refrigerator contains "tomatoes, lettuce, cheese, and bread."

[0630] Server Processing

[0631] These ingredients are recognized using image analysis and the ingredient database is updated.

[0632] Search the database for "sandwich recipes" and see that they call for ham.

[0633] Terminal display

[0634] Display "Sandwich (additional ham required)" as a suggestion to the user.

[0635] Example 3: Recipes based on user preferences

[0636] situation

[0637] The user has often selected "Japanese food" in the past.

[0638] Server Processing

[0639] It learns user preference data and searches for Japanese recipes (e.g., "meat and potato stew" and "miso soup").

[0640] As a new suggestion, add "lasagna" to the list of Western food options.

[0641] Terminal display

[0642] The user is prompted to select from "nikujaga," "miso soup," and "lasagna."

[0643] This system allows users to maximize the use of ingredients in their refrigerators while also allowing them to try new dishes as needed, reducing the time it takes to decide what to cook, reducing food waste, and enabling a wider variety of meals.

[0644] The processing flow will be explained below.

[0645] Step 1:

[0646] Camera image acquisition

[0647] The server periodically acquires images from a camera installed inside the refrigerator.

[0648] The camera takes pictures of each shelf and compartment in the refrigerator and generates image data.

[0649] Step 2:

[0650] Image data preprocessing

[0651] The server performs noise removal and resolution adjustment on the acquired image data.

[0652] The preprocessed image data is saved in preparation for analysis.

[0653] Step 3:

[0654] Image analysis and food ingredient recognition

[0655] The server uses deep learning models for object recognition to analyze the pre-processed image data.

[0656] The server uses the model to recognize ingredients in the image and identify the type and quantity of each ingredient.

[0657] Step 4:

[0658] Ingredient database update

[0659] The server stores the recognized ingredient information in an ingredient database.

[0660] The freshness and expiration date of ingredients are also recorded at the same time, keeping the database up to date.

[0661] Step 5:

[0662] Recipe database reference

[0663] The server refers to the ingredient database and searches for recipes that can be made using only the ingredients in the refrigerator.

[0664] It also searches for recipes that require additional ingredients and generates a list of the required additional ingredients.

[0665] Step 6:

[0666] Learning user preference data

[0667] The server collects the user's past recipe selection history and evaluation data.

[0668] The server uses this data to train machine learning models to predict user preferences.

[0669] Step 7:

[0670] Creating a recipe list

[0671] The server creates the following recipe list:

[0672] Recipes that can be made using only ingredients in your refrigerator

[0673] Recipes that require additional ingredients and a list of the additional ingredients required

[0674] Recipes based on user preferences

[0675] New recipes

[0676] Step 8:

[0677] Sending data

[0678] The server sends the generated recipe list to the terminal.

[0679] Step 9:

[0680] Displaying the recipe list

[0681] The terminal displays the recipe list received from the server on the user interface.

[0682] To make it easier for users to select a recipe, the app also displays an overview of each recipe, its difficulty level, and the time required.

[0683] Step 10:

[0684] Viewing detailed recipe information

[0685] When a user selects a particular recipe, the device displays detailed information about that recipe, including ingredients, steps, and any additional ingredients needed.

[0686] Step 11:

[0687] Generate a shopping list

[0688] The terminal displays a shopping list of ingredients that are missing based on the recipe selected by the user.

[0689] The user checks the shopping list and prepares to purchase the necessary ingredients.

[0690] Step 12:

[0691] Enter your feedback

[0692] After the user executes the selected recipe, the results are input as feedback to the terminal.

[0693] Feedback includes the quality of the dish, its difficulty, and individual ratings.

[0694] Step 13:

[0695] Sending feedback data

[0696] The terminal transmits feedback data from the user to the server.

[0697] Step 14:

[0698] Database Update

[0699] The server updates the user's preference database based on the received feedback data.

[0700] Use the updated data to improve your next recipe suggestion.

[0701] Through this series of processing steps, the system will suggest optimal recipes to the user, enabling them to make efficient use of the ingredients in their refrigerator.

[0702] Example 1

[0703] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0704] In modern life, people need to effectively manage the ingredients in their refrigerators and use them up without wasting any. However, visually checking the contents of the refrigerator and planning recipes based on that information is time-consuming and labor-intensive. It is also difficult to suggest optimal recipes based on the user's preferences and past cooking history. Furthermore, it is time-consuming to have to check and purchase ingredients that are missing. To solve these issues, a system is needed that can efficiently recognize ingredients in the refrigerator and suggest optimal recipes to users.

[0705] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0706] In this invention, the server includes means for analyzing images captured by an image capture device installed in the refrigerator and recognizing food in the refrigerator, means for suggesting dishes that can be made using only the food in the refrigerator based on the recognized food, means for suggesting dishes that can be made by adding additional ingredients to the recognized food, means for learning the user's past dish selection history and suggesting dishes based on the user's preferences, means for predicting the user's preferences using a machine learning model and suggesting dishes based on the predicted preferences, and means for displaying on a display device dishes based on the food in the refrigerator, dishes requiring additional ingredients, and dishes based on the user's preferences. This allows the user to efficiently manage ingredients in the refrigerator and receive optimal recipe suggestions, thereby reducing the effort required for cooking and reducing food waste.

[0707] An "image capture device" is a device that is installed inside a refrigerator and is used to capture images.

[0708] "Analysis" refers to the act of processing acquired image data and extracting necessary information.

[0709] "Food" refers to all food ingredients and beverages stored in the refrigerator.

[0710] The "means of suggestion" is a means of selecting appropriate recipes and information based on predetermined conditions and providing them to the user.

[0711] "Cuisine" refers to food prepared using specific ingredients.

[0712] "Additional ingredients" are new ingredients or seasonings that are needed in addition to the ingredients already in the refrigerator.

[0713] "User preferences" refer to preferences and favorite foods that are inferred based on the user's past selections and ratings of dishes.

[0714] A "machine learning model" refers to an algorithm or mathematical model that analyzes large amounts of data and finds patterns.

[0715] "Prediction methods" are methods for predicting future trends and results based on past data and analytical results.

[0716] A "display device" is a device that functions as a user interface and visually presents various pieces of information.

[0717] The present invention relates to a system that efficiently manages food in a refrigerator and suggests optimal recipes to users. The system is composed of a server, a terminal, and user interaction. The following describes in detail an embodiment of the present invention.

[0718] Server configuration and operation

[0719] The server receives and analyzes images captured by the image capture device installed inside the refrigerator. Specifically, it uses the following hardware and software:

[0720] Hardware:

[0721] Camera installed inside the refrigerator

[0722] Server itself (equipped with a high-performance CPU, sufficient memory, and network interface)

[0723] software:

[0724] Image processing library (e.g. OpenCV)

[0725] Object recognition libraries (e.g. TensorFlow, YOLO)

[0726] Database management system (e.g. MySQL)

[0727] Machine learning libraries (e.g., scikit-learn)

[0728] The server performs the following steps to recognize the food in the refrigerator:

[0729] 1. Image Acquisition: A camera installed inside the refrigerator periodically takes images and sends them to a server, which receives the image data via Wi-Fi or a wired connection.

[0730] 2. Image analysis: The received image data is first pre-processed to remove noise and adjust the resolution, then an object recognition algorithm is used to identify food in the image.

[0731] 3. Storage in database: Information such as the type, quantity, freshness, and expiration date of the recognized food is stored in a database.

[0732] Recipe suggestion function

[0733] The server suggests suitable recipes to the user by:

[0734] 1. Recipes that can be made using only food in the refrigerator: Search for relevant recipes based on food information in the database.

[0735] 2. Recipes that require additional ingredients: Search for recipes that can be made by combining ingredients in the refrigerator with additional ingredients, and generate a list of the additional ingredients required.

[0736] 3. Recipes based on user preferences: We collect users' past recipe selection history and rating data, and use machine learning models to predict their preferences. Based on this data, we then suggest optimal recipes.

[0737] Terminal configuration and operation

[0738] The terminal has the following user interface features:

[0739] 1. Displaying the recipe list: Display the recipe list received from the server in the user interface. The recipe list is dynamically generated and displayed using HTML and JavaScript.

[0740] 2. Displaying detailed recipe information: When a user selects a recipe, detailed information (ingredients, steps, required additional ingredients, etc.) is displayed. This is done by retrieving data from the server using Ajax.

[0741] 3. Generate shopping list: Based on the recipe selected by the user, additional ingredients are displayed as a shopping list.

[0742] User operations

[0743] Users use this system through the following operations:

[0744] 1. Recipe selection: Select the recipe you want to make from the recipe list displayed on your device.

[0745] 2. Check and purchase ingredients: Check the ingredients needed based on the recipe and purchase any that are missing.

[0746] 3. Entering feedback: After executing the selected recipe, the user enters the feedback into the device, which then sends this data to the server and updates the user's preference database.

[0747] Specific examples

[0748] Here are some concrete usage examples:

[0749] Example of a recipe that can be made using only ingredients in the refrigerator: If the refrigerator contains "eggs, milk, butter, flour, and bacon," the server will recognize these ingredients and suggest recipes for pancakes or bacon and eggs.

[0750] Example of a recipe that requires additional ingredients: If the refrigerator contains "tomatoes, lettuce, cheese, and bread," the server suggests a sandwich and informs the user that additional ham is needed.

[0751] Example of recipes based on user preferences: If the user has frequently selected "Japanese food" in the past, the server will suggest recipes for Nikujaga (meat and potato stew) and Miso soup. It will also add Lasagna to the list as a new suggestion.

[0752] Examples of prompt statements

[0753] "Please tell me a recipe I can make using the eggs, milk, butter, flour, and bacon I have in my fridge."

[0754] This invention allows users to efficiently manage food in their refrigerators and reduce the effort required for cooking. In addition, the system continuously learns through feedback, enabling it to suggest more accurate recipes.

[0755] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0756] Server Processing

[0757] Step 1: Acquire an image

[0758] Operation: A camera placed inside the refrigerator periodically takes images of the inside of the refrigerator.

[0759] Input: Image data captured by the camera.

[0760] How it works: The camera takes pictures of the inside of the refrigerator and sends them over WiFi or a wired connection to a server, which then downloads the image data using HTTP requests or the FTP protocol.

[0761] Output: Image data received on the server side.

[0762] Step 2: Image analysis

[0763] Operation: The server preprocesses the received image data and recognizes the food.

[0764] Input: The image data received in step 1.

[0765] How it works: The server uses image processing libraries such as OpenCV to remove noise and adjust the resolution, then uses TensorFlow to apply an object recognition model to identify and classify food items in the image.

[0766] Output: Recognized food types and quantities.

[0767] Step 3: Saving to the database

[0768] Operation: The server stores the recognized food data in a database.

[0769] Input: Food type and quantity data obtained in step 2.

[0770] How it works: The server connects to a database management system such as MySQL and uses the "INSERT" statement to record food names, quantities, freshness information, etc. into the database.

[0771] Output: Food data stored in a database.

[0772] Step 4: Recipe suggestions

[0773] Operation: The server refers to the food database to search and suggest recipes suitable for the user.

[0774] Input: Food data saved in step 3.

[0775] How it works: The server uses an SQL query to search the recipe table in the database to find recipes that match the food list in the refrigerator. It also searches for recipes that require additional ingredients and generates a list of the additional ingredients in JSON format.

[0776] Output: A list of recipes that can be made using only the food in the refrigerator, and a list of recipes that require additional ingredients.

[0777] Step 5: Learning user preferences

[0778] Operation: The server trains a machine learning model based on the user's past recipe selection history.

[0779] Input: User's past recipe selection history data.

[0780] How it works: The server uses pandas to read historical data and scikit-learn to train a machine learning model that predicts user preferences and generates recipes based on them.

[0781] Output: A list of recipes based on the user's preferences.

[0782] Step 6: Sending data

[0783] Operation: The server sends a recipe list to the device.

[0784] Input: The recipe list generated in step 4 and step 5.

[0785] How it works: The server sends an HTTP POST request to the device via a RESTful API, sending a list of recipes in JSON format.

[0786] Output: Recipe list sent to the device.

[0787] Terminal handling

[0788] Step 1: View the recipe list

[0789] Operation: The terminal displays the recipe list received from the server on the user interface.

[0790] Input: Recipe list sent from the server.

[0791] How it works: Using HTML and JavaScript, the received recipe list is dynamically generated and displayed on a web page.

[0792] Output: The recipe list displayed in the user interface.

[0793] Step 2: Viewing Recipe Details

[0794] Action: The user selects a recipe.

[0795] Input: User's choice.

[0796] How it works: The device uses Ajax to retrieve detailed information about the selected recipe from the server and displays it, including ingredients, steps, and any additional ingredients needed.

[0797] Output: Detailed information about the recipe.

[0798] Step 3: Generate a shopping list

[0799] Operation: Generate a shopping list based on the user's selected recipes.

[0800] Input: The recipe data selected by the user.

[0801] What it does: Dynamically generates a list of additional ingredients using JavaScript and displays it as a shopping list.

[0802] Output: Purchase list.

[0803] User operations

[0804] Step 1: Recipe Selection

[0805] Operation: The user selects the recipe they want to make from the recipe list displayed on the device.

[0806] Input: A list of recipes displayed on the terminal.

[0807] Action: The user selects a recipe via a select box or click event.

[0808] Output: The selected recipe.

[0809] Step 2: Check and purchase ingredients

[0810] Operation: The user checks the ingredients needed based on the recipe and purchases any missing ingredients.

[0811] Input: Recipe details.

[0812] What it does: The user looks at the shopping list and notes or buys the ingredients they need.

[0813] Output: A list of ingredients confirmed.

[0814] Step 3: Provide feedback

[0815] Operation: The user executes the selected recipe and then enters their feedback into the terminal.

[0816] Input: User feedback data.

[0817] Behavior: A feedback input form is provided, and after the user enters their rating and comments, it is sent to the server.

[0818] Output: Feedback data sent to the server.

[0819] This allows users to efficiently manage the ingredients in their refrigerators and receive appropriate recipe suggestions, reducing the effort required for cooking and reducing food waste. In addition, the system continues to learn through feedback, allowing it to make even more accurate suggestions.

[0820] (Application example 1)

[0821] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0822] Conventional refrigerator food management systems only allowed users to track the ingredients in their refrigerators and suggested recipes. Furthermore, users had to go shopping separately to get ingredients they were running low on, which was inconvenient. Furthermore, the system lacked the ability to suggest new recipes based on the user's preferences. This limited the opportunities for users to efficiently use the ingredients in their refrigerators and try new recipes.

[0823] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0824] In this invention, the server includes means for analyzing images captured by a camera installed inside the refrigerator and recognizing ingredients in the refrigerator, means for suggesting recipes that can be made using only the ingredients in the refrigerator based on the recognized ingredients, means for suggesting recipes that can be made by adding additional ingredients to the recognized ingredients, means for learning the user's past recipe selection history and suggesting recipes based on the user's preferences, means for displaying on a user interface recipes based on the ingredients in the refrigerator, recipes that require additional ingredients, and recipes based on the user's preferences, means for recognizing ingredients that are missing from a recipe selected by the user and generating a list of those ingredients, and means for enabling the user to order missing ingredients for delivery based on the generated list of missing ingredients. This allows the user to use the ingredients in the refrigerator efficiently, expand recipe options, and easily procure missing ingredients.

[0825] A "camera installed inside a refrigerator" is a camera device that is attached inside a refrigerator and captures video or images of the interior.

[0826] "Analyzing images" is the process of recognizing and identifying specific objects or elements based on the acquired image data.

[0827] "Means for recognizing ingredients" refers to a method or device for identifying specific foods or ingredients in a refrigerator through image analysis.

[0828] The "means for suggesting recipes that can be made using only ingredients in the refrigerator" is a method or device that presents to the user recipes for dishes that do not require additional ingredients, based on recognized ingredients.

[0829] The "means for suggesting recipes that can be made by adding additional ingredients" is a method or device that presents to the user how to make dishes that can be made by adding missing ingredients to the recognized ingredients.

[0830] "Learning a user's past recipe selection history" is the process of collecting data about recipes previously selected by a user and analyzing that data.

[0831] The "means for proposing recipes based on user preferences" is a method or device for presenting recipes that match the collected and learned preferences of the user.

[0832] The "means for displaying on a user interface" refers to a method or apparatus for displaying suggested recipes and information on the screen of a device used by a user.

[0833] The "means for recognizing missing ingredients and generating a list thereof" is a method or device that identifies ingredients that are required for a recipe selected by the user but are not present in the refrigerator and lists them.

[0834] A "means for enabling delivery ordering of missing ingredients" is a method or device that enables identified missing ingredients to be purchased directly via an online ordering system.

[0835] An embodiment of the present invention uses hardware such as a server, a smartphone, and a camera inside a refrigerator, and software such as an image analysis algorithm, a recipe database, a user interface, and delivery service collaboration software.

[0836] The server analyzes images captured by a camera installed inside the refrigerator and recognizes the ingredients inside. A deep learning model is used as the image analysis algorithm for recognition. For example, frameworks such as TensorFlow and PyTorch can be used. Preprocessing involves noise removal and resolution adjustment, followed by object recognition to identify the ingredients. Information about the recognized ingredients is stored in a database.

[0837] The server then suggests recipes that can be made using only the ingredients in the refrigerator based on the recognized ingredients. To do this, the server refers to the ingredient database and searches for relevant recipes.

[0838] The server then suggests recipes that can be made by adding additional ingredients to the recognized ingredients, searches the database for the necessary additional ingredients, and generates a list of these ingredients, which is then displayed together with the recipe selected by the user.

[0839] The server also learns the user's past recipe selection history and suggests recipes based on the user's preferences. By using a machine learning model, the server can learn the user's preference patterns and provide recipes tailored to their preferences.

[0840] The user interface displays these recipes on the smartphone screen. The user interacts with the server through the smartphone and selects from the suggested recipes. A list of ingredients missing from the selected recipe is generated, and this information can be used to order delivery via delivery service integration software.

[0841] For example, a user can take a photo of the ingredients in their refrigerator with their smartphone and send the image to a server via the app. The server analyzes the image and recognizes the ingredients. The server then suggests recipes that can be made with the recognized ingredients, as well as dishes that require additional ingredients, and displays these to the user via a user interface. When the user selects a recipe, a list of missing ingredients is displayed, allowing them to order delivery.

[0842] Specifically, this involves suggesting new dishes (such as Western food if the user mostly chooses Japanese food) based on the user's past recipe preferences, and ordering missing ingredients through a delivery service.

[0843] An example of a specific prompt using a generative AI model is, "Based on the results of image analysis of the inside of the refrigerator, please suggest recipes that can be made with the ingredients I currently have. Also, please generate a list of ingredients that I am running low on."

[0844] This allows users to efficiently use ingredients in their refrigerator, expand their recipe options, and easily procure ingredients that they are running low on.

[0845] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0846] Step 1:

[0847] The user takes a picture of the inside of the refrigerator with the smartphone camera and sends it to the server via the app. The input is image data of the inside of the refrigerator, and the output is the image data sent to the server. The user records the state of the inside of the refrigerator using the smartphone camera and uploads the image to the server using the send button on the app.

[0848] Step 2:

[0849] The server preprocesses the received image data. The input is the transmitted image data, and the output is the preprocessed image data. Specific operations include noise reduction and resolution adjustment. For example, it applies a filter to reduce image noise and resizes the image to a resolution suitable for analysis.

[0850] Step 3:

[0851] The server analyzes the preprocessed image data using a deep learning model to recognize the ingredients in the refrigerator. The input is the preprocessed image data, and the output is a list of recognized ingredients. Specifically, it uses an object detection algorithm to identify each ingredient in the image and extract the name and quantity of each ingredient.

[0852] Step 4:

[0853] The server saves the recognized ingredient list in a database. The input is the recognized ingredient list, and the output is the latest ingredient database. Specifically, the operation involves connecting to the database and adding or updating new ingredient information.

[0854] Step 5:

[0855] The server references the ingredient database to search for recipes that can be made using only the ingredients in the refrigerator. The input is the ingredient database, and the output is a list of matching recipes. Specifically, it executes a database query to retrieve recipes based on the currently available ingredients.

[0856] Step 6:

[0857] The server searches for recipes that require additional ingredients and generates a list of missing ingredients. The input is an ingredient database and a list of recognized ingredients, and the output is a list of recipes with the additional ingredients and a list of missing ingredients. Specifically, the server performs a database query to identify ingredients that are missing from existing ingredients and then lists them.

[0858] Step 7:

[0859] The server learns the user's past recipe selection history and suggests recipes based on the user's preferences. The input is the user's selection history data, and the output is a personalized recipe list. Specifically, it uses a machine learning algorithm to analyze past data and identify the user's preference patterns.

[0860] Step 8:

[0861] The device displays the recipe list received from the server on the user interface. The input is the recipe list sent from the server, and the output is the recipe list displayed on the smartphone screen. The user can select the desired recipe from the displayed list.

[0862] Step 9:

[0863] The system checks the ingredients missing from the recipe selected by the user and displays a shopping list of the additional ingredients needed. The input is the selected recipe and the output is the shopping list. The user checks the displayed list and identifies the missing ingredients.

[0864] Step 10:

[0865] The terminal allows users to order missing ingredients for delivery based on the shopping list. The input is the shopping list, and the output is the execution of the delivery order. Specifically, the order details are sent via the delivery service's API, and the ingredients are delivered to the user.

[0866] Through the above processing steps, the user can efficiently utilize ingredients in the refrigerator, find suitable recipes, and easily procure ingredients that are in short supply.

[0867] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0868] This invention is a system that combines an image analysis system using a camera installed inside the refrigerator with an emotion engine that recognizes the user's emotions to make optimal use of ingredients in the refrigerator and suggest recipes based on the user's preferences and emotions. This system is composed of interactions between a server, terminals, and users, and realizes more personalized recipe suggestions.

[0869] Server Processing

[0870] 1. Acquiring camera images

[0871] The server periodically acquires images from a camera installed inside the refrigerator.

[0872] The camera takes pictures of each shelf and compartment inside the refrigerator and generates image data.

[0873] 2. Image data preprocessing

[0874] The server preprocesses the acquired image data, removing noise and adjusting the resolution.

[0875] The preprocessed image data is used for analysis.

[0876] 3. Image analysis and food ingredient recognition

[0877] The server analyzes the image data using a deep learning model to identify the type and quantity of ingredients.

[0878] The recognition results are stored in a food ingredient database, and freshness and expiration date are also recorded.

[0879] 4. Referencing the recipe database

[0880] The server refers to the ingredient database and searches for recipes that can be made using only the ingredients in the refrigerator.

[0881] It also searches for recipes that require additional ingredients and generates a list of the ingredients needed.

[0882] 5. Learning User Preference Data

[0883] The server collects past recipe selection history and rating data to train a preference prediction model.

[0884] Used to suggest recipes based on the user's preferences.

[0885] 6. Use of Emotion Engine

[0886] The server uses an emotion engine to recognize the user's emotions in real time.

[0887] The recognized emotion data is reflected in recipe suggestions.

[0888] 7. Generating Recipe Suggestions

[0889] The server generates the following recipe list:

[0890] Recipes that can be made using only ingredients in your refrigerator

[0891] Recipes and lists that require additional ingredients

[0892] Recipes based on user preferences

[0893] New recipes

[0894] Recipes based on user sentiment

[0895] 8. Data transmission

[0896] The server sends the generated recipe list to the terminal.

[0897] Terminal handling

[0898] 1. Display the recipe list

[0899] The terminal displays the recipe list received from the server on the user interface.

[0900] It also displays an overview of the recipe, its difficulty level, and the time required.

[0901] 2. View detailed recipe information

[0902] When a user selects a particular recipe, the device displays detailed information, including ingredients, steps, and any additional ingredients needed.

[0903] 3. Generate a shopping list

[0904] The device displays a shopping list of additional ingredients based on the selected recipe.

[0905] The user checks the list and prepares to purchase the necessary ingredients.

[0906] User operations

[0907] 1. Recipe selection

[0908] The user selects the recipe they want to make from the recipe list displayed on the terminal.

[0909] 2. Check and purchase ingredients

[0910] Users check the ingredients needed based on the recipe and purchase any missing ingredients.

[0911] 3. Providing Emotion Data

[0912] The system recognizes emotions from the user's facial expressions and voice in real time and transmits them to the server.

[0913] 4. Enter your feedback

[0914] After executing the selected recipe, the user inputs the results and impressions as feedback into the terminal.

[0915] Specific examples

[0916] Example 1: Recipe suggestions based on ingredients in the refrigerator and emotions

[0917] situation

[0918] The refrigerator contains "eggs, milk, butter, flour, and bacon," and the user is relaxed.

[0919] Server Processing

[0920] Ingredients are recognized through image analysis and the ingredient database is updated.

[0921] The emotion engine recognizes the user's relaxed state.

[0922] It searches the database for "hotcakes" and "bacon and eggs" and recommends "hotcakes" that best suits the user's condition.

[0923] Terminal display

[0924] "Hotcakes" is displayed to the user, along with detailed recipe information.

[0925] Example 2: User preferences and new recipe suggestions

[0926] situation

[0927] The user has often selected "Japanese food" in the past and is now interested in new cuisine.

[0928] Server Processing

[0929] It learns user preference data and suggests Japanese dishes such as "meat and potato stew" and "miso soup."

[0930] Add "lasagna" as a new suggestion.

[0931] Terminal display

[0932] It displays dishes such as "nikujaga" (beef and potato stew), "miso soup" and "lasagna" to encourage people to try new dishes.

[0933] This system allows users to make the most of the ingredients in their refrigerator and easily find dishes that suit their tastes and emotions, significantly reducing the time it takes to decide what to cook, minimizing food waste, and increasing the variety of their daily meals.

[0934] The processing flow will be explained below.

[0935] Step 1:

[0936] Camera image acquisition

[0937] The server periodically acquires images from a camera installed inside the refrigerator.

[0938] The camera takes pictures of each shelf and compartment inside the refrigerator and generates image data.

[0939] Step 2:

[0940] Image data preprocessing

[0941] The server preprocesses the acquired image data, removing noise and adjusting the resolution.

[0942] Save the preprocessed image data.

[0943] Step 3:

[0944] Image analysis and food ingredient recognition

[0945] The server analyzes the preprocessed image data using a deep learning model to identify the type and quantity of ingredients.

[0946] The results of ingredient recognition are stored in an ingredient database, and freshness and expiration date are also recorded.

[0947] Step 4:

[0948] Acquiring emotion data

[0949] The device captures the user's facial expressions and voice in real time to recognize their emotions.

[0950] An emotion engine analyzes the captured data to determine the user's current emotional state.

[0951] Step 5:

[0952] Recipe database reference

[0953] The server refers to the ingredient database and searches for recipes that can be made using only the ingredients in the refrigerator.

[0954] It also searches for recipes that require additional ingredients and generates a list of the ingredients needed.

[0955] Step 6:

[0956] Learning user preference data

[0957] The server collects past recipe selection history and rating data to train the machine learning model.

[0958] Predict recipes based on user preference data.

[0959] Step 7:

[0960] Emotional Data Integration

[0961] The server collects the recognized emotion data and adjusts the recipe suggestion options based on it.

[0962] If the emotion indicates relaxation, customizations such as suggesting simple and relaxing dishes are performed.

[0963] Step 8:

[0964] Creating a recipe list

[0965] The server generates the following recipe list:

[0966] Recipes that can be made using only ingredients in your refrigerator

[0967] Recipes that require additional ingredients and a list of the additional ingredients required

[0968] Recipes based on user preferences

[0969] New recipes

[0970] Recipes based on user sentiment

[0971] Step 9:

[0972] Sending data

[0973] The server sends the generated recipe list to the terminal.

[0974] Step 10:

[0975] Displaying the recipe list

[0976] The terminal displays the recipe list received from the server on the user interface.

[0977] It also displays an overview of the recipe, its difficulty level, and the time required.

[0978] Step 11:

[0979] Viewing detailed recipe information

[0980] When a user selects a particular recipe, the device displays detailed information, including ingredients, steps, and any additional ingredients needed.

[0981] Step 12:

[0982] Generate a shopping list

[0983] The device displays a shopping list of additional ingredients based on the selected recipe.

[0984] The user checks the list and prepares to purchase the necessary ingredients.

[0985] Step 13:

[0986] Cooking and getting emotional feedback

[0987] The user executes the selected recipe and inputs emotional feedback into the device during the process.

[0988] The emotion engine collects the user's emotion data during and after cooking.

[0989] Step 14:

[0990] Sending feedback data

[0991] The terminal transmits feedback data from the user to the server.

[0992] Step 15:

[0993] Database Update

[0994] The server updates the user's preference database and emotion database based on the received feedback data.

[0995] Use the updated data to improve your next recipe suggestion.

[0996] This system allows users to make the most of the ingredients in their refrigerator and easily find dishes that suit their tastes and emotions. It also utilizes real-time emotional data to provide more personalized recipe suggestions.

[0997] Example 2

[0998] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0999] Conventional refrigerator ingredient management and recipe suggestion systems are limited to suggesting recipes that can be made using only the ingredients in the refrigerator or recipes that require additional ingredients. This does not allow for personalized suggestions based on the user's preferences and emotions, leading to users always making the same dishes and wasting ingredients. Furthermore, the lack of increased cooking options tends to decrease users' motivation to cook. Therefore, the present invention aims to provide more personalized cooking suggestions by taking the user's preferences and emotions into account, thereby improving the enjoyment and efficiency of cooking.

[1000] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1001] In this invention, the server includes means for analyzing images captured by a camera installed inside the refrigerator and recognizing ingredients in the refrigerator, means for suggesting recipes that can be made using only the ingredients in the refrigerator based on the recognized ingredients, means for suggesting recipes that can be made by adding additional ingredients to the recognized ingredients, means for learning the user's past recipe selection history and suggesting recipes based on the user's preferences, means for recognizing the user's emotional state and suggesting recipes based on the emotions, and means for displaying on a user interface recipes based on the ingredients in the refrigerator, recipes that require additional ingredients, recipes based on the user's preferences, and recipes based on the user's emotions. This allows the user to effectively use the ingredients in the refrigerator and select recipes that suit their preferences and emotions.

[1002] A "camera installed inside a refrigerator" is a device installed inside the refrigerator for taking images of food items inside the refrigerator.

[1003] "Means for analyzing images and recognizing ingredients in the refrigerator" refers to technology and algorithms that use images acquired from a camera to identify the type and quantity of ingredients present in the refrigerator.

[1004] The "means for suggesting recipes that can be made using only ingredients in the refrigerator" is a function that searches for recipes that can be made using only the ingredients in the recognized refrigerator and suggests them to the user.

[1005] "Means for suggesting recipes that can be made by adding additional ingredients" is a function that provides recipes for dishes that can be made by adding other ingredients in addition to the ingredients in the refrigerator.

[1006] "Means for learning a user's past recipe selection history and suggesting recipes based on the user's preferences" refers to technology that collects and analyzes recipes previously selected by the user and their evaluation data, and then suggests recipes that suit the user's preferences based on that data.

[1007] "Means for recognizing the user's emotional state and suggesting recipes based on those emotions" is a function that analyzes the user's emotions in real time from their facial expressions, voice, etc., and suggests recipes that are appropriate for those emotions.

[1008] "Means for displaying on the user interface" refers to the technology or mechanism for displaying suggested recipes and related information on the screen of the device used by the user.

[1009] This invention is a system that combines an image analysis system using a camera installed inside the refrigerator with an emotion engine that recognizes the user's emotions to make optimal use of ingredients in the refrigerator and suggest recipes based on the user's preferences and emotions. This system is composed of interactions between a server, terminals, and users, and realizes more personalized recipe suggestions.

[1010] The server periodically acquires images from a camera installed inside the refrigerator and analyzes the image data to recognize ingredients. This analysis uses a deep learning model. The acquired image data is preprocessed to remove noise and adjust the resolution. This improves the accuracy of the image data, enabling accurate recognition of ingredients.

[1011] Once the ingredients are recognized, the server updates the ingredient database, recording the type, quantity, freshness, and expiration date of each ingredient. Based on this database, the server searches for recipes that can be made using only the ingredients in the refrigerator and suggests them to the user. It also searches for recipes that can be made by adding additional ingredients to the ingredients in the refrigerator and generates a list of the necessary ingredients.

[1012] Furthermore, the server collects the user's past recipe selection history and rating data to train a preference prediction model. This model allows it to suggest recipes based on the user's preferences. Meanwhile, the server uses an emotion engine to recognize the user's emotions in real time and reflects this emotion data in the recipe suggestions. This allows it to suggest simple dishes when the user is relaxed and quick dishes when the user is busy.

[1013] As a concrete example, suppose the refrigerator contains "eggs, milk, butter, flour, and bacon," and the user is in a relaxed state. When the server recognizes these ingredients and the emotion engine understands the user's relaxed state, it searches for recipes for "hotcakes" and "bacon and eggs." Among them, it suggests "hotcakes," which matches the user's relaxed state. The server sends this suggested recipe list to the terminal, which then displays it on the user interface.

[1014] The device accepts input from the user, and when a specific recipe is selected, it displays detailed information about it, allowing the user to check the ingredients needed and cooking steps. If additional ingredients are needed, they are displayed as a shopping list, which the user can use to purchase them. For example, the tomatoes needed to make a "tomato omelette" are added to the shopping list.

[1015] The user selects the recipe they want to make from a list of recipes displayed on the device, checks the ingredients needed, and purchases them. Emotional data from the user's facial expressions and voice is also collected in real time and sent to the server. After cooking, the user enters the results and their impressions as feedback into the device, and this data is used to suggest future recipes.

[1016] As an example of a prompt sentence, if you enter "I'm wondering what to have for dinner tonight," the system will suggest the best recipe based on the ingredients in the refrigerator and the user's preferences and emotions.

[1017] As described above, the system of the present invention makes effective use of ingredients in the refrigerator and proposes recipes that match the user's tastes and emotions, thereby reducing the time it takes to decide on a dish and minimizing the waste of ingredients, making everyday meals more varied and enjoyable.

[1018] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1019] Server Processing Steps

[1020] Step 1: Acquire camera images

[1021] The server periodically acquires images from a camera installed inside the refrigerator. The input is image data captured by the camera, which is sent to the server. The output is the acquired image data, which is stored in the server. Subsequent image processing is performed based on this image data.

[1022] Step 2: Preprocessing the image data

[1023] The server performs pre-processing on the acquired image data. Specific operations include noise removal, resolution adjustment, and image cropping. The input is raw image data acquired from the camera, and the output is clear, pre-processed image data. This processing improves the accuracy of image analysis.

[1024] Step 3: Image analysis and ingredient recognition

[1025] The server uses a deep learning model to recognize ingredients in the refrigerator from preprocessed image data. The input is the preprocessed image data, and the output is data recording the type and quantity of recognized ingredients. This data is stored in an ingredient database. Specifically, it identifies ingredients such as eggs, milk, and butter.

[1026] Step 4: Referencing the recipe database

[1027] The server refers to the ingredient database and searches for recipes that can be made using only the ingredients in the refrigerator. At the same time, it searches for recipes that require additional ingredients and generates a list of the necessary ingredients. The input is the recognition data from the ingredient database, and the output is a list of recommended recipes and a list of additional ingredients. Specifically, it includes recipes that can be made with "eggs and milk," and recipes that can be made by adding "tomatoes."

[1028] Step 5: Learning user preference data

[1029] The server collects the user's past recipe selection history and rating data and trains a preference prediction model. The input is the past recipe selection history and rating data, and the output is a model that learns the user's preference patterns. This model makes it possible to suggest recipes that suit the user's preferences.

[1030] Step 6: Use the Emotion Engine

[1031] The server uses an emotion engine to recognize the user's emotions in real time. The input is emotional data obtained from the user's facial expressions and voice, and the output is the recognized emotional state. Based on this, a recipe appropriate to the user's emotions is selected. For example, if the user is relaxed, a recipe that suits that state is suggested.

[1032] Step 7: Generate recipe suggestions

[1033] The server generates the following recipe list based on the analysis data so far:

[1034] Recipes that can be made using only ingredients in your refrigerator

[1035] Recipes and lists that require additional ingredients

[1036] Recipes based on user preferences

[1037] Recipes based on user sentiment

[1038] The input is food recognition data, preference data, and emotion data, and the output is a list of recommended recipes.

[1039] Step 8: Sending Data

[1040] The server sends the generated recipe list to the terminal. The input is the recommended recipe list, and the output is the list sent to the terminal. This list includes a summary of the recipe, the ingredients required, and cooking instructions.

[1041] Terminal processing steps

[1042] Step 1: View the recipe list

[1043] The terminal displays the recipe list received from the server on the user interface. The input is the recipe list from the server, and the output is the recipe information displayed on the user interface, including the recipe summary, difficulty, and time.

[1044] Step 2: Viewing Recipe Details

[1045] When the user selects a particular recipe, the device displays its details. The input is the user's selected recipe, and the output is details such as ingredients, steps, and any additional ingredients needed.

[1046] Step 3: Generate a shopping list

[1047] The terminal creates and displays a shopping list based on the selected recipe. The input is the selected recipe data, and the output is a shopping list listing the additional ingredients needed. The user can check this list and prepare to purchase the necessary ingredients.

[1048] User operation steps

[1049] Step 1: Recipe Selection

[1050] The user selects the recipe they want to make from the recipe list displayed on the terminal. The input is the recipe list displayed on the terminal, and the output is the selected recipe.

[1051] Step 2: Check and purchase ingredients

[1052] The user checks the ingredients needed based on the recipe and purchases any missing ingredients. The input is the list of ingredients for the recipe, and the output is the purchased ingredients.

[1053] Step 3: Provide emotion data

[1054] The system recognizes emotions from the user's facial expressions and voice in real time and transmits them to the server. The input is the user's facial expressions and voice, and the output is emotional data sent to the server.

[1055] Step 4: Provide feedback

[1056] After executing the selected recipe, the user inputs the results and their impressions as feedback into the terminal. The input is the recipe execution results and impressions, and the output is feedback data sent to the server.

[1057] (Application example 2)

[1058] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1059] In recent years, there has been a demand for making effective use of ingredients in the refrigerator and reducing waste, but it has been difficult for users to find recipes that suit their preferences and emotions. Furthermore, when using delivery services, food options are not optimized because suggestions are not made that take into account the ingredients in the refrigerator or the user's emotions. This has led to problems such as wasted ingredients in the refrigerator and inappropriate food delivery suggestions. The present invention aims to solve these problems by efficiently utilizing ingredients in the refrigerator and realizing appropriate recipe and delivery suggestions that match the user's emotions and preferences.

[1060] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1061] In this invention, the server includes means for analyzing images captured by a camera installed inside the refrigerator and recognizing ingredients in the refrigerator, means for suggesting recipes that can be made using only the ingredients in the refrigerator based on the recognized ingredients, means for suggesting recipes that can be made by adding additional ingredients to the recognized ingredients, means for learning the user's past recipe selection history and suggesting recipes based on the user's preferences, means for analyzing the user's emotions and suggesting recipes based on the analyzed emotions, and means for displaying on a user interface recipes based on the ingredients in the refrigerator, recipes that require additional ingredients, recipes based on the user's preferences, and recipes based on the user's emotions.This makes it possible to efficiently use ingredients in the refrigerator without waste and to suggest personalized recipes and delivery options that suit the user's emotions and preferences.

[1062] A "camera installed inside a refrigerator" is a device that takes pictures of the inside of a refrigerator and obtains information about the internal condition and ingredients.

[1063] "Means for analyzing images and recognizing ingredients in the refrigerator" refers to algorithms or software that process image data captured by a camera and identify the types and quantities of ingredients contained within.

[1064] "Means to suggest recipes that can be made using only ingredients in the refrigerator" is a function that presents users with recipes for dishes that can be made without using any additional ingredients, based on the information of recognized ingredients.

[1065] "Means for suggesting recipes that can be made by adding additional ingredients" is a function that presents users with recipes for dishes that can be made by taking into account the ingredients in the refrigerator and other ingredients that can be added from outside.

[1066] "Means for learning the user's past recipe selection history and suggesting recipes based on the user's preferences" is a function that analyzes the history of recipes that the user has selected in the past and suggests recipes that suit the user's preferences based on the results.

[1067] "Means for analyzing the user's emotions and suggesting recipes based on the analyzed emotions" is a function that evaluates the user's emotions from their facial expressions, voice, etc., and suggests the optimal recipe based on their emotional state.

[1068] "Means for displaying information in a user interface" refers to a display or screen display device that provides information in a form that can be seen by the user.

[1069] The "means for generating a list of additional ingredients" is a function that lists ingredients required for the recipe selected by the user that are in short supply in the refrigerator and presents them as a shopping list.

[1070] "Means to suggest new recipes" is a function that suggests recipes for new genres or types of food that the user has not tried before, based on the user's past recipe selection history.

[1071] "Means for suggesting ingredients based on the user's purchase history" is a function that analyzes the history of ingredients that the user has purchased and suggests new ingredients that should be purchased based on that history.

[1072] The system for realizing this invention consists of an algorithm that analyzes images from a camera installed inside the refrigerator and recognizes ingredients, an engine that analyzes the user's preferences and emotions, and a server and terminal that suggests recipes and delivery options based on this data.

[1073] Server Processing

[1074] The server first periodically acquires image data from a camera installed inside the refrigerator and preprocesses the images. The hardware used includes a camera and server, and software such as OpenCV and TensorFlow. OpenCV is used to remove noise and adjust the resolution of the images, and TensorFlow is used to analyze the images using a deep learning model and recognize ingredients.

[1075] The system then stores the recognized ingredient data in an ingredient database, and uses that data to reference a recipe database to search for recipes that can be made using only the ingredients in the refrigerator, as well as recipes that require additional ingredients. It also learns the user's preferences using the user's past recipe selection history and rating data. Furthermore, it uses an emotion engine to analyze the user's emotional state in real time and collects emotion data.

[1076] We combine these data to generate the following recipe list:

[1077] Recipes that can be made using only ingredients in your refrigerator

[1078] Recipes and lists that require additional ingredients

[1079] Recipes based on user preferences

[1080] New recipes

[1081] Recipes based on user sentiment

[1082] Terminal handling

[1083] The device displays the recipe list sent from the server on a user interface. A smartphone or tablet is used for the display, and detailed information such as a recipe summary, difficulty level, and required time is also provided. When the user selects a specific recipe, a list of additional ingredients is generated and presented to the user.

[1084] User operations

[1085] The user selects the recipe they want to make from a list of recipes displayed on their device and purchases additional ingredients as needed. The system also recognizes the user's emotions in real time from their facial expressions and voice, and sends this data to the server. After executing the recipe, the system provides feedback on the results and impressions, which are used to learn the user's preference data.

[1086] Specific examples

[1087] As a concrete example, consider a situation where the user has "eggs, milk, and cheese" in the refrigerator and is feeling "stressed." Based on this, the server can suggest recipes and delivery options such as a cheese omelet or stress-relieving herbal tea. Also, based on the user's past preference for "Japanese food," the server can suggest a new type of "Western food" recipe this time.

[1088] Prompt Sentence Examples

[1089] "Please recognize ingredients from camera images inside the refrigerator and analyze emotions from the user's facial expressions and voice. Based on the results, please generate a program that will suggest the best recipes and food delivery options for the user."

[1090] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1091] Processing Steps

[1092] Step 1:

[1093] The server periodically acquires image data from a camera installed inside the refrigerator. The image data captured by the camera is provided as input and is used for image pre-processing in the next step.

[1094] Step 2:

[1095] The server preprocesses the acquired image data. Here, OpenCV is used to perform basic image processing such as removing noise from the image and adjusting the resolution. The input is raw image data from the camera, and the output is image data that has been denoised and adjusted for resolution.

[1096] Step 3:

[1097] The server analyzes the preprocessed image data to recognize ingredients. A deep learning model using TensorFlow identifies the type and quantity of ingredients and stores the results in an ingredient database. The input is the preprocessed image data, and the output is the type and quantity data of ingredients.

[1098] Step 4:

[1099] Based on the information in the ingredient database, the server searches for recipes that can be made using only the ingredients in the refrigerator and recipes that require additional ingredients. It also references the recipe database and generates a list of ingredients required for each recipe. The input is the type and quantity data of ingredients, and the output is a list of recipes and a list of required ingredients.

[1100] Step 5:

[1101] The server learns the user's past recipe selection history and suggests recipes based on their preferences. Using the past selection data as learning data, it trains a preference prediction model and suggests recipes that match the user's preferences. The input is the past recipe selection history, and the output is a recipe based on the user's preferences.

[1102] Step 6:

[1103] The server uses an emotion engine to analyze the user's emotions. It uses facial expression images and voice data obtained by a camera and microphone as input to identify the user's emotional state. The input is the user's facial expression images and voice data, and the output is emotion data.

[1104] Step 7:

[1105] The server generates a recipe list by integrating recipes based on ingredients in the refrigerator, recipes that require additional ingredients, recipes based on preferences, and recipes based on emotions. This creates personalized recipe suggestions that take various data into consideration. The input is ingredient data, preference data, and emotion data, and the output is a recipe list.

[1106] Step 8:

[1107] The server sends the generated recipe list to the terminal, along with detailed recipe information and a list of required ingredients. The input is the recipe list, and the output is the recipe data sent to the terminal.

[1108] Step 9:

[1109] The terminal displays the recipe list received from the server, and when the user selects a specific recipe, displays its detailed information. If necessary, it also generates and displays additional ingredient lists. The input is the recipe list received from the server, and the output is the display on the user interface.

[1110] Step 10:

[1111] The user selects the recipe they want to make from the recipe list displayed on the device, and checks and purchases any additional ingredients they need. The user provides emotional data from their facial expressions and voice, and also inputs feedback. The input is the displayed recipe list and a list of additional ingredients, and the output is the selected recipe and feedback data.

[1112] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1113] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1114] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1115] [Third embodiment]

[1116] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1117] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1118] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1119] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1120] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1121] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1122] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1123] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1124] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1125] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1126] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1127] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1128] This invention relates to a system that uses an image analysis system with a camera installed inside a refrigerator to recognize ingredients in the refrigerator and then suggests optimal recipes to the user based on that information. This system is composed of a server, a terminal, and user interaction.

[1129] Server Processing

[1130] 1. Acquiring camera images

[1131] A camera installed inside the refrigerator periodically takes pictures of the inside of the refrigerator, and the server captures and receives these pictures.

[1132] 2. Image analysis and food ingredient recognition

[1133] The server preprocesses the received image data to remove noise and adjust the resolution, then applies image analysis algorithms (e.g., deep learning models for object recognition) to identify the ingredients in the refrigerator.

[1134] The type and quantity of recognized ingredients are stored in a database, and the freshness and expiration date of the ingredients are also recorded.

[1135] 3. Referencing the recipe database

[1136] The server references the ingredient database to search for recipes that can be made using only the ingredients in the refrigerator. It also searches for recipes that require additional ingredients and generates a list of the additional ingredients required.

[1137] 4. Learning user preference data

[1138] The server collects the user's past recipe selection history and rating data, and trains a machine learning model to predict the user's preferences. Based on this data, it suggests recipes that suit the user's tastes and new recipes that the user has not tried yet.

[1139] 5. Generating Recipe Suggestions

[1140] The server creates the following recipe list:

[1141] Recipes that can be made using only ingredients in your refrigerator

[1142] Recipes that require additional ingredients and a list of the additional ingredients required

[1143] Recipes based on user preferences

[1144] New recipes

[1145] 6. Data transmission

[1146] The server sends the generated recipe list to the terminal.

[1147] Terminal handling

[1148] 1. Display the recipe list

[1149] The terminal displays the recipe list received from the server on the user interface.

[1150] 2. View detailed recipe information

[1151] When a user selects a recipe, the device displays detailed information about the recipe, including ingredients, steps, and any additional ingredients needed.

[1152] 3. Generate a shopping list

[1153] The terminal displays a shopping list of additional ingredients based on the recipe selected by the user.

[1154] User operations

[1155] 1. Recipe selection

[1156] The user selects the recipe they want to make from the recipe list displayed on the terminal.

[1157] 2. Check and purchase ingredients

[1158] Users check the ingredients needed based on the recipe and purchase any missing ingredients.

[1159] 3. Enter your feedback

[1160] After the user executes the selected recipe, they input their feedback into the device, which then sends this data to the server and updates the user preference database.

[1161] Specific examples

[1162] Example 1: Recipes that can be made using only ingredients in your refrigerator

[1163] situation

[1164] The refrigerator contains "eggs, milk, butter, flour, and bacon."

[1165] Server Processing

[1166] These ingredients are recognized using image analysis and the ingredient database is updated.

[1167] Search the database for "hotcake recipes" and "bacon and egg recipes" and add them to the list.

[1168] Terminal display

[1169] "Hotcakes" and "Bacon and eggs" are displayed to the user as options.

[1170] Example 2: Recipe that requires additional ingredients

[1171] situation

[1172] The refrigerator contains "tomatoes, lettuce, cheese, and bread."

[1173] Server Processing

[1174] These ingredients are recognized using image analysis and the ingredient database is updated.

[1175] Search the database for "sandwich recipes" and see that they call for ham.

[1176] Terminal display

[1177] Display "Sandwich (additional ham required)" as a suggestion to the user.

[1178] Example 3: Recipes based on user preferences

[1179] situation

[1180] The user has often selected "Japanese food" in the past.

[1181] Server Processing

[1182] It learns user preference data and searches for Japanese recipes (e.g., "meat and potato stew" and "miso soup").

[1183] As a new suggestion, add "lasagna" to the list of Western food options.

[1184] Terminal display

[1185] The user is prompted to select from "nikujaga," "miso soup," and "lasagna."

[1186] This system allows users to maximize the use of ingredients in their refrigerators while also allowing them to try new dishes as needed, reducing the time it takes to decide what to cook, reducing food waste, and enabling a wider variety of meals.

[1187] The processing flow will be explained below.

[1188] Step 1:

[1189] Camera image acquisition

[1190] The server periodically acquires images from a camera installed inside the refrigerator.

[1191] The camera takes pictures of each shelf and compartment in the refrigerator and generates image data.

[1192] Step 2:

[1193] Image data preprocessing

[1194] The server performs noise removal and resolution adjustment on the acquired image data.

[1195] The preprocessed image data is saved in preparation for analysis.

[1196] Step 3:

[1197] Image analysis and food ingredient recognition

[1198] The server uses deep learning models for object recognition to analyze the pre-processed image data.

[1199] The server uses the model to recognize ingredients in the image and identify the type and quantity of each ingredient.

[1200] Step 4:

[1201] Ingredient database update

[1202] The server stores the recognized ingredient information in an ingredient database.

[1203] The freshness and expiration date of ingredients are also recorded at the same time, keeping the database up to date.

[1204] Step 5:

[1205] Recipe database reference

[1206] The server refers to the ingredient database and searches for recipes that can be made using only the ingredients in the refrigerator.

[1207] It also searches for recipes that require additional ingredients and generates a list of the required additional ingredients.

[1208] Step 6:

[1209] Learning user preference data

[1210] The server collects the user's past recipe selection history and evaluation data.

[1211] The server uses this data to train machine learning models to predict user preferences.

[1212] Step 7:

[1213] Creating a recipe list

[1214] The server creates the following recipe list:

[1215] Recipes that can be made using only ingredients in your refrigerator

[1216] Recipes that require additional ingredients and a list of the additional ingredients required

[1217] Recipes based on user preferences

[1218] New recipes

[1219] Step 8:

[1220] Sending data

[1221] The server sends the generated recipe list to the terminal.

[1222] Step 9:

[1223] Displaying the recipe list

[1224] The terminal displays the recipe list received from the server on the user interface.

[1225] To make it easier for users to select a recipe, the app also displays an overview of each recipe, its difficulty level, and the time required.

[1226] Step 10:

[1227] Viewing detailed recipe information

[1228] When a user selects a particular recipe, the device displays detailed information about that recipe, including ingredients, steps, and any additional ingredients needed.

[1229] Step 11:

[1230] Generate a shopping list

[1231] The terminal displays a shopping list of ingredients that are missing based on the recipe selected by the user.

[1232] The user checks the shopping list and prepares to purchase the necessary ingredients.

[1233] Step 12:

[1234] Enter your feedback

[1235] After the user executes the selected recipe, the results are input as feedback to the terminal.

[1236] Feedback includes the quality of the dish, its difficulty, and individual ratings.

[1237] Step 13:

[1238] Sending feedback data

[1239] The terminal transmits feedback data from the user to the server.

[1240] Step 14:

[1241] Database Update

[1242] The server updates the user's preference database based on the received feedback data.

[1243] Use the updated data to improve your next recipe suggestion.

[1244] Through this series of processing steps, the system will suggest optimal recipes to the user, enabling them to make efficient use of the ingredients in their refrigerator.

[1245] Example 1

[1246] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1247] In modern life, people need to effectively manage the ingredients in their refrigerators and use them up without wasting any. However, visually checking the contents of the refrigerator and planning recipes based on that information is time-consuming and labor-intensive. It is also difficult to suggest optimal recipes based on the user's preferences and past cooking history. Furthermore, it is time-consuming to have to check and purchase ingredients that are missing. To solve these issues, a system is needed that can efficiently recognize ingredients in the refrigerator and suggest optimal recipes to users.

[1248] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1249] In this invention, the server includes means for analyzing images captured by an image capture device installed in the refrigerator and recognizing food in the refrigerator, means for suggesting dishes that can be made using only the food in the refrigerator based on the recognized food, means for suggesting dishes that can be made by adding additional ingredients to the recognized food, means for learning the user's past dish selection history and suggesting dishes based on the user's preferences, means for predicting the user's preferences using a machine learning model and suggesting dishes based on the predicted preferences, and means for displaying on a display device dishes based on the food in the refrigerator, dishes requiring additional ingredients, and dishes based on the user's preferences. This allows the user to efficiently manage ingredients in the refrigerator and receive optimal recipe suggestions, thereby reducing the effort required for cooking and reducing food waste.

[1250] An "image capture device" is a device that is installed inside a refrigerator and is used to capture images.

[1251] "Analysis" refers to the act of processing acquired image data and extracting necessary information.

[1252] "Food" refers to all food ingredients and beverages stored in the refrigerator.

[1253] The "means of suggestion" is a means of selecting appropriate recipes and information based on predetermined conditions and providing them to the user.

[1254] "Cuisine" refers to food prepared using specific ingredients.

[1255] "Additional ingredients" are new ingredients or seasonings that are needed in addition to the ingredients already in the refrigerator.

[1256] "User preferences" refer to preferences and favorite foods that are inferred based on the user's past selections and ratings of dishes.

[1257] A "machine learning model" refers to an algorithm or mathematical model that analyzes large amounts of data and finds patterns.

[1258] "Prediction methods" are methods for predicting future trends and results based on past data and analytical results.

[1259] A "display device" is a device that functions as a user interface and visually presents various pieces of information.

[1260] The present invention relates to a system that efficiently manages food in a refrigerator and suggests optimal recipes to users. The system is composed of a server, a terminal, and user interaction. The following describes in detail an embodiment of the present invention.

[1261] Server configuration and operation

[1262] The server receives and analyzes images captured by the image capture device installed inside the refrigerator. Specifically, it uses the following hardware and software:

[1263] Hardware:

[1264] Camera installed inside the refrigerator

[1265] Server itself (equipped with a high-performance CPU, sufficient memory, and network interface)

[1266] software:

[1267] Image processing library (e.g. OpenCV)

[1268] Object recognition libraries (e.g. TensorFlow, YOLO)

[1269] Database management system (e.g. MySQL)

[1270] Machine learning libraries (e.g., scikit-learn)

[1271] The server performs the following steps to recognize the food in the refrigerator:

[1272] 1. Image Acquisition: A camera installed inside the refrigerator periodically takes images and sends them to a server, which receives the image data via Wi-Fi or a wired connection.

[1273] 2. Image analysis: The received image data is first pre-processed to remove noise and adjust the resolution, then an object recognition algorithm is used to identify food in the image.

[1274] 3. Storage in database: Information such as the type, quantity, freshness, and expiration date of the recognized food is stored in a database.

[1275] Recipe suggestion function

[1276] The server suggests suitable recipes to the user by:

[1277] 1. Recipes that can be made using only food in the refrigerator: Search for relevant recipes based on food information in the database.

[1278] 2. Recipes that require additional ingredients: Search for recipes that can be made by combining ingredients in the refrigerator with additional ingredients, and generate a list of the additional ingredients required.

[1279] 3. Recipes based on user preferences: We collect users' past recipe selection history and rating data, and use machine learning models to predict their preferences. Based on this data, we then suggest optimal recipes.

[1280] Terminal configuration and operation

[1281] The terminal has the following user interface features:

[1282] 1. Displaying the recipe list: Display the recipe list received from the server in the user interface. The recipe list is dynamically generated and displayed using HTML and JavaScript.

[1283] 2. Displaying detailed recipe information: When a user selects a recipe, detailed information (ingredients, steps, required additional ingredients, etc.) is displayed. This is done by retrieving data from the server using Ajax.

[1284] 3. Generate shopping list: Based on the recipe selected by the user, additional ingredients are displayed as a shopping list.

[1285] User operations

[1286] Users use this system through the following operations:

[1287] 1. Recipe selection: Select the recipe you want to make from the recipe list displayed on your device.

[1288] 2. Check and purchase ingredients: Check the ingredients needed based on the recipe and purchase any that are missing.

[1289] 3. Entering feedback: After executing the selected recipe, the user enters the feedback into the device, which then sends this data to the server and updates the user's preference database.

[1290] Specific examples

[1291] Here are some concrete usage examples:

[1292] Example of a recipe that can be made using only ingredients in the refrigerator: If the refrigerator contains "eggs, milk, butter, flour, and bacon," the server will recognize these ingredients and suggest recipes for pancakes or bacon and eggs.

[1293] Example of a recipe that requires additional ingredients: If the refrigerator contains "tomatoes, lettuce, cheese, and bread," the server suggests a sandwich and informs the user that additional ham is needed.

[1294] Example of recipes based on user preferences: If the user has frequently selected "Japanese food" in the past, the server will suggest recipes for Nikujaga (meat and potato stew) and Miso soup. It will also add Lasagna to the list as a new suggestion.

[1295] Examples of prompt statements

[1296] "Please tell me a recipe I can make using the eggs, milk, butter, flour, and bacon I have in my fridge."

[1297] This invention allows users to efficiently manage food in their refrigerators and reduce the effort required for cooking. In addition, the system continuously learns through feedback, enabling it to suggest more accurate recipes.

[1298] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1299] Server Processing

[1300] Step 1: Acquire an image

[1301] Operation: A camera placed inside the refrigerator periodically takes images of the inside of the refrigerator.

[1302] Input: Image data captured by the camera.

[1303] How it works: The camera takes pictures of the inside of the refrigerator and sends them over WiFi or a wired connection to a server, which then downloads the image data using HTTP requests or the FTP protocol.

[1304] Output: Image data received on the server side.

[1305] Step 2: Image analysis

[1306] Operation: The server preprocesses the received image data and recognizes the food.

[1307] Input: The image data received in step 1.

[1308] How it works: The server uses image processing libraries such as OpenCV to remove noise and adjust the resolution, then uses TensorFlow to apply an object recognition model to identify and classify food items in the image.

[1309] Output: Recognized food types and quantities.

[1310] Step 3: Saving to the database

[1311] Operation: The server stores the recognized food data in a database.

[1312] Input: Food type and quantity data obtained in step 2.

[1313] How it works: The server connects to a database management system such as MySQL and uses the "INSERT" statement to record food names, quantities, freshness information, etc. into the database.

[1314] Output: Food data stored in a database.

[1315] Step 4: Recipe suggestions

[1316] Operation: The server refers to the food database to search and suggest recipes suitable for the user.

[1317] Input: Food data saved in step 3.

[1318] How it works: The server uses an SQL query to search the recipe table in the database to find recipes that match the food list in the refrigerator. It also searches for recipes that require additional ingredients and generates a list of the additional ingredients in JSON format.

[1319] Output: A list of recipes that can be made using only the food in the refrigerator, and a list of recipes that require additional ingredients.

[1320] Step 5: Learning user preferences

[1321] Operation: The server trains a machine learning model based on the user's past recipe selection history.

[1322] Input: User's past recipe selection history data.

[1323] How it works: The server uses pandas to read historical data and scikit-learn to train a machine learning model that predicts user preferences and generates recipes based on them.

[1324] Output: A list of recipes based on the user's preferences.

[1325] Step 6: Sending data

[1326] Operation: The server sends a recipe list to the device.

[1327] Input: The recipe list generated in step 4 and step 5.

[1328] How it works: The server sends an HTTP POST request to the device via a RESTful API, sending a list of recipes in JSON format.

[1329] Output: Recipe list sent to the device.

[1330] Terminal handling

[1331] Step 1: View the recipe list

[1332] Operation: The terminal displays the recipe list received from the server on the user interface.

[1333] Input: Recipe list sent from the server.

[1334] How it works: Using HTML and JavaScript, the received recipe list is dynamically generated and displayed on a web page.

[1335] Output: The recipe list displayed in the user interface.

[1336] Step 2: Viewing Recipe Details

[1337] Action: The user selects a recipe.

[1338] Input: User's choice.

[1339] How it works: The device uses Ajax to retrieve detailed information about the selected recipe from the server and displays it, including ingredients, steps, and any additional ingredients needed.

[1340] Output: Detailed information about the recipe.

[1341] Step 3: Generate a shopping list

[1342] Operation: Generate a shopping list based on the user's selected recipes.

[1343] Input: The recipe data selected by the user.

[1344] What it does: Dynamically generates a list of additional ingredients using JavaScript and displays it as a shopping list.

[1345] Output: Purchase list.

[1346] User operations

[1347] Step 1: Recipe Selection

[1348] Operation: The user selects the recipe they want to make from the recipe list displayed on the device.

[1349] Input: A list of recipes displayed on the terminal.

[1350] Action: The user selects a recipe via a select box or click event.

[1351] Output: The selected recipe.

[1352] Step 2: Check and purchase ingredients

[1353] Operation: The user checks the ingredients needed based on the recipe and purchases any missing ingredients.

[1354] Input: Recipe details.

[1355] What it does: The user looks at the shopping list and notes or buys the ingredients they need.

[1356] Output: A list of ingredients confirmed.

[1357] Step 3: Provide feedback

[1358] Operation: The user executes the selected recipe and then enters their feedback into the terminal.

[1359] Input: User feedback data.

[1360] Behavior: A feedback input form is provided, and after the user enters their rating and comments, it is sent to the server.

[1361] Output: Feedback data sent to the server.

[1362] This allows users to efficiently manage the ingredients in their refrigerators and receive appropriate recipe suggestions, reducing the effort required for cooking and reducing food waste. In addition, the system continues to learn through feedback, allowing it to make even more accurate suggestions.

[1363] (Application example 1)

[1364] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1365] Conventional refrigerator food management systems only allowed users to track the ingredients in their refrigerators and suggested recipes. Furthermore, users had to go shopping separately to get ingredients they were running low on, which was inconvenient. Furthermore, the system lacked the ability to suggest new recipes based on the user's preferences. This limited the opportunities for users to efficiently use the ingredients in their refrigerators and try new recipes.

[1366] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1367] In this invention, the server includes means for analyzing images captured by a camera installed inside the refrigerator and recognizing ingredients in the refrigerator, means for suggesting recipes that can be made using only the ingredients in the refrigerator based on the recognized ingredients, means for suggesting recipes that can be made by adding additional ingredients to the recognized ingredients, means for learning the user's past recipe selection history and suggesting recipes based on the user's preferences, means for displaying on a user interface recipes based on the ingredients in the refrigerator, recipes that require additional ingredients, and recipes based on the user's preferences, means for recognizing ingredients that are missing from a recipe selected by the user and generating a list of those ingredients, and means for enabling the user to order missing ingredients for delivery based on the generated list of missing ingredients. This allows the user to use the ingredients in the refrigerator efficiently, expand recipe options, and easily procure missing ingredients.

[1368] A "camera installed inside a refrigerator" is a camera device that is attached inside a refrigerator and captures video or images of the interior.

[1369] "Analyzing images" is the process of recognizing and identifying specific objects or elements based on the acquired image data.

[1370] "Means for recognizing ingredients" refers to a method or device for identifying specific foods or ingredients in a refrigerator through image analysis.

[1371] The "means for suggesting recipes that can be made using only ingredients in the refrigerator" is a method or device that presents to the user recipes for dishes that do not require additional ingredients, based on recognized ingredients.

[1372] The "means for suggesting recipes that can be made by adding additional ingredients" is a method or device that presents to the user how to make dishes that can be made by adding missing ingredients to the recognized ingredients.

[1373] "Learning a user's past recipe selection history" is the process of collecting data about recipes previously selected by a user and analyzing that data.

[1374] The "means for proposing recipes based on user preferences" is a method or device for presenting recipes that match the collected and learned preferences of the user.

[1375] The "means for displaying on a user interface" refers to a method or apparatus for displaying suggested recipes and information on the screen of a device used by a user.

[1376] The "means for recognizing missing ingredients and generating a list thereof" is a method or device that identifies ingredients that are required for a recipe selected by the user but are not present in the refrigerator and lists them.

[1377] A "means for enabling delivery ordering of missing ingredients" is a method or device that enables identified missing ingredients to be purchased directly via an online ordering system.

[1378] An embodiment of the present invention uses hardware such as a server, a smartphone, and a camera inside a refrigerator, and software such as an image analysis algorithm, a recipe database, a user interface, and delivery service collaboration software.

[1379] The server analyzes images captured by a camera installed inside the refrigerator and recognizes the ingredients inside. A deep learning model is used as the image analysis algorithm for recognition. For example, frameworks such as TensorFlow and PyTorch can be used. Preprocessing involves noise removal and resolution adjustment, followed by object recognition to identify the ingredients. Information about the recognized ingredients is stored in a database.

[1380] The server then suggests recipes that can be made using only the ingredients in the refrigerator based on the recognized ingredients. To do this, the server refers to the ingredient database and searches for relevant recipes.

[1381] The server then suggests recipes that can be made by adding additional ingredients to the recognized ingredients, searches the database for the necessary additional ingredients, and generates a list of these ingredients, which is then displayed together with the recipe selected by the user.

[1382] The server also learns the user's past recipe selection history and suggests recipes based on the user's preferences. By using a machine learning model, the server can learn the user's preference patterns and provide recipes tailored to their preferences.

[1383] The user interface displays these recipes on the smartphone screen. The user interacts with the server through the smartphone and selects from the suggested recipes. A list of ingredients missing from the selected recipe is generated, and this information can be used to order delivery via delivery service integration software.

[1384] For example, a user can take a photo of the ingredients in their refrigerator with their smartphone and send the image to a server via the app. The server analyzes the image and recognizes the ingredients. The server then suggests recipes that can be made with the recognized ingredients, as well as dishes that require additional ingredients, and displays these to the user via a user interface. When the user selects a recipe, a list of missing ingredients is displayed, allowing them to order delivery.

[1385] Specifically, this involves suggesting new dishes (such as Western food if the user mostly chooses Japanese food) based on the user's past recipe preferences, and ordering missing ingredients through a delivery service.

[1386] An example of a specific prompt using a generative AI model is, "Based on the results of image analysis of the inside of the refrigerator, please suggest recipes that can be made with the ingredients I currently have. Also, please generate a list of ingredients that I am running low on."

[1387] This allows users to efficiently use ingredients in their refrigerator, expand their recipe options, and easily procure ingredients that they are running low on.

[1388] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1389] Step 1:

[1390] The user takes a picture of the inside of the refrigerator with the smartphone camera and sends it to the server via the app. The input is image data of the inside of the refrigerator, and the output is the image data sent to the server. The user records the state of the inside of the refrigerator using the smartphone camera and uploads the image to the server using the send button on the app.

[1391] Step 2:

[1392] The server preprocesses the received image data. The input is the transmitted image data, and the output is the preprocessed image data. Specific operations include noise reduction and resolution adjustment. For example, it applies a filter to reduce image noise and resizes the image to a resolution suitable for analysis.

[1393] Step 3:

[1394] The server analyzes the preprocessed image data using a deep learning model to recognize the ingredients in the refrigerator. The input is the preprocessed image data, and the output is a list of recognized ingredients. Specifically, it uses an object detection algorithm to identify each ingredient in the image and extract the name and quantity of each ingredient.

[1395] Step 4:

[1396] The server saves the recognized ingredient list in a database. The input is the recognized ingredient list, and the output is the latest ingredient database. Specifically, the operation involves connecting to the database and adding or updating new ingredient information.

[1397] Step 5:

[1398] The server references the ingredient database to search for recipes that can be made using only the ingredients in the refrigerator. The input is the ingredient database, and the output is a list of matching recipes. Specifically, it executes a database query to retrieve recipes based on the currently available ingredients.

[1399] Step 6:

[1400] The server searches for recipes that require additional ingredients and generates a list of missing ingredients. The input is an ingredient database and a list of recognized ingredients, and the output is a list of recipes with the additional ingredients and a list of missing ingredients. Specifically, the server performs a database query to identify ingredients that are missing from existing ingredients and then lists them.

[1401] Step 7:

[1402] The server learns the user's past recipe selection history and suggests recipes based on the user's preferences. The input is the user's selection history data, and the output is a personalized recipe list. Specifically, it uses a machine learning algorithm to analyze past data and identify the user's preference patterns.

[1403] Step 8:

[1404] The device displays the recipe list received from the server on the user interface. The input is the recipe list sent from the server, and the output is the recipe list displayed on the smartphone screen. The user can select the desired recipe from the displayed list.

[1405] Step 9:

[1406] The system checks the ingredients missing from the recipe selected by the user and displays a shopping list of the additional ingredients needed. The input is the selected recipe and the output is the shopping list. The user checks the displayed list and identifies the missing ingredients.

[1407] Step 10:

[1408] The terminal allows users to order missing ingredients for delivery based on the shopping list. The input is the shopping list, and the output is the execution of the delivery order. Specifically, the order details are sent via the delivery service's API, and the ingredients are delivered to the user.

[1409] Through the above processing steps, the user can efficiently utilize ingredients in the refrigerator, find suitable recipes, and easily procure ingredients that are in short supply.

[1410] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1411] This invention is a system that combines an image analysis system using a camera installed inside the refrigerator with an emotion engine that recognizes the user's emotions to make optimal use of ingredients in the refrigerator and suggest recipes based on the user's preferences and emotions. This system is composed of interactions between a server, terminals, and users, and realizes more personalized recipe suggestions.

[1412] Server Processing

[1413] 1. Acquiring camera images

[1414] The server periodically acquires images from a camera installed inside the refrigerator.

[1415] The camera takes pictures of each shelf and compartment inside the refrigerator and generates image data.

[1416] 2. Image data preprocessing

[1417] The server preprocesses the acquired image data, removing noise and adjusting the resolution.

[1418] The preprocessed image data is used for analysis.

[1419] 3. Image analysis and food ingredient recognition

[1420] The server analyzes the image data using a deep learning model to identify the type and quantity of ingredients.

[1421] The recognition results are stored in a food ingredient database, and freshness and expiration date are also recorded.

[1422] 4. Referencing the recipe database

[1423] The server refers to the ingredient database and searches for recipes that can be made using only the ingredients in the refrigerator.

[1424] It also searches for recipes that require additional ingredients and generates a list of the ingredients needed.

[1425] 5. Learning User Preference Data

[1426] The server collects past recipe selection history and rating data to train a preference prediction model.

[1427] Used to suggest recipes based on the user's preferences.

[1428] 6. Use of Emotion Engine

[1429] The server uses an emotion engine to recognize the user's emotions in real time.

[1430] The recognized emotion data is reflected in recipe suggestions.

[1431] 7. Generating Recipe Suggestions

[1432] The server generates the following recipe list:

[1433] Recipes that can be made using only ingredients in your refrigerator

[1434] Recipes and lists that require additional ingredients

[1435] Recipes based on user preferences

[1436] New recipes

[1437] Recipes based on user sentiment

[1438] 8. Data transmission

[1439] The server sends the generated recipe list to the terminal.

[1440] Terminal handling

[1441] 1. Display the recipe list

[1442] The terminal displays the recipe list received from the server on the user interface.

[1443] It also displays an overview of the recipe, its difficulty level, and the time required.

[1444] 2. View detailed recipe information

[1445] When a user selects a particular recipe, the device displays detailed information, including ingredients, steps, and any additional ingredients needed.

[1446] 3. Generate a shopping list

[1447] The device displays a shopping list of additional ingredients based on the selected recipe.

[1448] The user checks the list and prepares to purchase the necessary ingredients.

[1449] User operations

[1450] 1. Recipe selection

[1451] The user selects the recipe they want to make from the recipe list displayed on the terminal.

[1452] 2. Check and purchase ingredients

[1453] Users check the ingredients needed based on the recipe and purchase any missing ingredients.

[1454] 3. Providing Emotion Data

[1455] The system recognizes emotions from the user's facial expressions and voice in real time and transmits them to the server.

[1456] 4. Enter your feedback

[1457] After executing the selected recipe, the user inputs the results and impressions as feedback into the terminal.

[1458] Specific examples

[1459] Example 1: Recipe suggestions based on ingredients in the refrigerator and emotions

[1460] situation

[1461] The refrigerator contains "eggs, milk, butter, flour, and bacon," and the user is relaxed.

[1462] Server Processing

[1463] Ingredients are recognized through image analysis and the ingredient database is updated.

[1464] The emotion engine recognizes the user's relaxed state.

[1465] It searches the database for "hotcakes" and "bacon and eggs" and recommends "hotcakes" that best suits the user's condition.

[1466] Terminal display

[1467] "Hotcakes" is displayed to the user, along with detailed recipe information.

[1468] Example 2: User preferences and new recipe suggestions

[1469] situation

[1470] The user has often selected "Japanese food" in the past and is now interested in new cuisine.

[1471] Server Processing

[1472] It learns user preference data and suggests Japanese dishes such as "meat and potato stew" and "miso soup."

[1473] Add "lasagna" as a new suggestion.

[1474] Terminal display

[1475] It displays dishes such as "nikujaga" (beef and potato stew), "miso soup" and "lasagna" to encourage people to try new dishes.

[1476] This system allows users to make the most of the ingredients in their refrigerator and easily find dishes that suit their tastes and emotions, significantly reducing the time it takes to decide what to cook, minimizing food waste, and increasing the variety of their daily meals.

[1477] The processing flow will be explained below.

[1478] Step 1:

[1479] Camera image acquisition

[1480] The server periodically acquires images from a camera installed inside the refrigerator.

[1481] The camera takes pictures of each shelf and compartment inside the refrigerator and generates image data.

[1482] Step 2:

[1483] Image data preprocessing

[1484] The server preprocesses the acquired image data, removing noise and adjusting the resolution.

[1485] Save the preprocessed image data.

[1486] Step 3:

[1487] Image analysis and food ingredient recognition

[1488] The server analyzes the preprocessed image data using a deep learning model to identify the type and quantity of ingredients.

[1489] The results of ingredient recognition are stored in an ingredient database, and freshness and expiration date are also recorded.

[1490] Step 4:

[1491] Acquiring emotion data

[1492] The device captures the user's facial expressions and voice in real time to recognize their emotions.

[1493] An emotion engine analyzes the captured data to determine the user's current emotional state.

[1494] Step 5:

[1495] Recipe database reference

[1496] The server refers to the ingredient database and searches for recipes that can be made using only the ingredients in the refrigerator.

[1497] It also searches for recipes that require additional ingredients and generates a list of the ingredients needed.

[1498] Step 6:

[1499] Learning user preference data

[1500] The server collects past recipe selection history and rating data to train the machine learning model.

[1501] Predict recipes based on user preference data.

[1502] Step 7:

[1503] Emotional Data Integration

[1504] The server collects the recognized emotion data and adjusts the recipe suggestion options based on it.

[1505] If the emotion indicates relaxation, customizations such as suggesting simple and relaxing dishes are performed.

[1506] Step 8:

[1507] Creating a recipe list

[1508] The server generates the following recipe list:

[1509] Recipes that can be made using only ingredients in your refrigerator

[1510] Recipes that require additional ingredients and a list of the additional ingredients required

[1511] Recipes based on user preferences

[1512] New recipes

[1513] Recipes based on user sentiment

[1514] Step 9:

[1515] Sending data

[1516] The server sends the generated recipe list to the terminal.

[1517] Step 10:

[1518] Displaying the recipe list

[1519] The terminal displays the recipe list received from the server on the user interface.

[1520] It also displays an overview of the recipe, its difficulty level, and the time required.

[1521] Step 11:

[1522] Viewing detailed recipe information

[1523] When a user selects a particular recipe, the device displays detailed information, including ingredients, steps, and any additional ingredients needed.

[1524] Step 12:

[1525] Generate a shopping list

[1526] The device displays a shopping list of additional ingredients based on the selected recipe.

[1527] The user checks the list and prepares to purchase the necessary ingredients.

[1528] Step 13:

[1529] Cooking and getting emotional feedback

[1530] The user executes the selected recipe and inputs emotional feedback into the device during the process.

[1531] The emotion engine collects the user's emotion data during and after cooking.

[1532] Step 14:

[1533] Sending feedback data

[1534] The terminal transmits feedback data from the user to the server.

[1535] Step 15:

[1536] Database Update

[1537] The server updates the user's preference database and emotion database based on the received feedback data.

[1538] Use the updated data to improve your next recipe suggestion.

[1539] This system allows users to make the most of the ingredients in their refrigerator and easily find dishes that suit their tastes and emotions. It also utilizes real-time emotional data to provide more personalized recipe suggestions.

[1540] Example 2

[1541] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1542] Conventional refrigerator ingredient management and recipe suggestion systems are limited to suggesting recipes that can be made using only the ingredients in the refrigerator or recipes that require additional ingredients. This does not allow for personalized suggestions based on the user's preferences and emotions, leading to users always making the same dishes and wasting ingredients. Furthermore, the lack of increased cooking options tends to decrease users' motivation to cook. Therefore, the present invention aims to provide more personalized cooking suggestions by taking the user's preferences and emotions into account, thereby improving the enjoyment and efficiency of cooking.

[1543] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1544] In this invention, the server includes means for analyzing images captured by a camera installed inside the refrigerator and recognizing ingredients in the refrigerator, means for suggesting recipes that can be made using only the ingredients in the refrigerator based on the recognized ingredients, means for suggesting recipes that can be made by adding additional ingredients to the recognized ingredients, means for learning the user's past recipe selection history and suggesting recipes based on the user's preferences, means for recognizing the user's emotional state and suggesting recipes based on the emotions, and means for displaying on a user interface recipes based on the ingredients in the refrigerator, recipes that require additional ingredients, recipes based on the user's preferences, and recipes based on the user's emotions. This allows the user to effectively use the ingredients in the refrigerator and select recipes that suit their preferences and emotions.

[1545] A "camera installed inside a refrigerator" is a device installed inside the refrigerator for taking images of food items inside the refrigerator.

[1546] "Means for analyzing images and recognizing ingredients in the refrigerator" refers to technology and algorithms that use images acquired from a camera to identify the type and quantity of ingredients present in the refrigerator.

[1547] The "means for suggesting recipes that can be made using only ingredients in the refrigerator" is a function that searches for recipes that can be made using only the ingredients in the recognized refrigerator and suggests them to the user.

[1548] "Means for suggesting recipes that can be made by adding additional ingredients" is a function that provides recipes for dishes that can be made by adding other ingredients in addition to the ingredients in the refrigerator.

[1549] "Means for learning a user's past recipe selection history and suggesting recipes based on the user's preferences" refers to technology that collects and analyzes recipes previously selected by the user and their evaluation data, and then suggests recipes that suit the user's preferences based on that data.

[1550] "Means for recognizing the user's emotional state and suggesting recipes based on those emotions" is a function that analyzes the user's emotions in real time from their facial expressions, voice, etc., and suggests recipes that are appropriate for those emotions.

[1551] "Means for displaying on the user interface" refers to the technology or mechanism for displaying suggested recipes and related information on the screen of the device used by the user.

[1552] This invention is a system that combines an image analysis system using a camera installed inside the refrigerator with an emotion engine that recognizes the user's emotions to make optimal use of ingredients in the refrigerator and suggest recipes based on the user's preferences and emotions. This system is composed of interactions between a server, terminals, and users, and realizes more personalized recipe suggestions.

[1553] The server periodically acquires images from a camera installed inside the refrigerator and analyzes the image data to recognize ingredients. This analysis uses a deep learning model. The acquired image data is preprocessed to remove noise and adjust the resolution. This improves the accuracy of the image data, enabling accurate recognition of ingredients.

[1554] Once the ingredients are recognized, the server updates the ingredient database, recording the type, quantity, freshness, and expiration date of each ingredient. Based on this database, the server searches for recipes that can be made using only the ingredients in the refrigerator and suggests them to the user. It also searches for recipes that can be made by adding additional ingredients to the ingredients in the refrigerator and generates a list of the necessary ingredients.

[1555] Furthermore, the server collects the user's past recipe selection history and rating data to train a preference prediction model. This model allows it to suggest recipes based on the user's preferences. Meanwhile, the server uses an emotion engine to recognize the user's emotions in real time and reflects this emotion data in the recipe suggestions. This allows it to suggest simple dishes when the user is relaxed and quick dishes when the user is busy.

[1556] As a concrete example, suppose the refrigerator contains "eggs, milk, butter, flour, and bacon," and the user is in a relaxed state. When the server recognizes these ingredients and the emotion engine understands the user's relaxed state, it searches for recipes for "hotcakes" and "bacon and eggs." Among them, it suggests "hotcakes," which matches the user's relaxed state. The server sends this suggested recipe list to the terminal, which then displays it on the user interface.

[1557] The device accepts input from the user, and when a specific recipe is selected, it displays detailed information about it, allowing the user to check the ingredients needed and cooking steps. If additional ingredients are needed, they are displayed as a shopping list, which the user can use to purchase them. For example, the tomatoes needed to make a "tomato omelette" are added to the shopping list.

[1558] The user selects the recipe they want to make from a list of recipes displayed on the device, checks the ingredients needed, and purchases them. Emotional data from the user's facial expressions and voice is also collected in real time and sent to the server. After cooking, the user enters the results and their impressions as feedback into the device, and this data is used to suggest future recipes.

[1559] As an example of a prompt sentence, if you enter "I'm wondering what to have for dinner tonight," the system will suggest the best recipe based on the ingredients in the refrigerator and the user's preferences and emotions.

[1560] As described above, the system of the present invention makes effective use of ingredients in the refrigerator and proposes recipes that match the user's tastes and emotions, thereby reducing the time it takes to decide on a dish and minimizing the waste of ingredients, making everyday meals more varied and enjoyable.

[1561] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1562] Server Processing Steps

[1563] Step 1: Acquire camera images

[1564] The server periodically acquires images from a camera installed inside the refrigerator. The input is image data captured by the camera, which is sent to the server. The output is the acquired image data, which is stored in the server. Subsequent image processing is performed based on this image data.

[1565] Step 2: Preprocessing the image data

[1566] The server performs pre-processing on the acquired image data. Specific operations include noise removal, resolution adjustment, and image cropping. The input is raw image data acquired from the camera, and the output is clear, pre-processed image data. This processing improves the accuracy of image analysis.

[1567] Step 3: Image analysis and ingredient recognition

[1568] The server uses a deep learning model to recognize ingredients in the refrigerator from preprocessed image data. The input is the preprocessed image data, and the output is data recording the type and quantity of recognized ingredients. This data is stored in an ingredient database. Specifically, it identifies ingredients such as eggs, milk, and butter.

[1569] Step 4: Referencing the recipe database

[1570] The server refers to the ingredient database and searches for recipes that can be made using only the ingredients in the refrigerator. At the same time, it searches for recipes that require additional ingredients and generates a list of the necessary ingredients. The input is the recognition data from the ingredient database, and the output is a list of recommended recipes and a list of additional ingredients. Specifically, it includes recipes that can be made with "eggs and milk," and recipes that can be made by adding "tomatoes."

[1571] Step 5: Learning user preference data

[1572] The server collects the user's past recipe selection history and rating data and trains a preference prediction model. The input is the past recipe selection history and rating data, and the output is a model that learns the user's preference patterns. This model makes it possible to suggest recipes that suit the user's preferences.

[1573] Step 6: Use the Emotion Engine

[1574] The server uses an emotion engine to recognize the user's emotions in real time. The input is emotional data obtained from the user's facial expressions and voice, and the output is the recognized emotional state. Based on this, a recipe appropriate to the user's emotions is selected. For example, if the user is relaxed, a recipe that suits that state is suggested.

[1575] Step 7: Generate recipe suggestions

[1576] The server generates the following recipe list based on the analysis data so far:

[1577] Recipes that can be made using only ingredients in your refrigerator

[1578] Recipes and lists that require additional ingredients

[1579] Recipes based on user preferences

[1580] Recipes based on user sentiment

[1581] The input is food recognition data, preference data, and emotion data, and the output is a list of recommended recipes.

[1582] Step 8: Sending Data

[1583] The server sends the generated recipe list to the terminal. The input is the recommended recipe list, and the output is the list sent to the terminal. This list includes a summary of the recipe, the ingredients required, and cooking instructions.

[1584] Terminal processing steps

[1585] Step 1: View the recipe list

[1586] The terminal displays the recipe list received from the server on the user interface. The input is the recipe list from the server, and the output is the recipe information displayed on the user interface, including the recipe summary, difficulty, and time.

[1587] Step 2: Viewing Recipe Details

[1588] When the user selects a particular recipe, the device displays its details. The input is the user's selected recipe, and the output is details such as ingredients, steps, and any additional ingredients needed.

[1589] Step 3: Generate a shopping list

[1590] The terminal creates and displays a shopping list based on the selected recipe. The input is the selected recipe data, and the output is a shopping list listing the additional ingredients needed. The user can check this list and prepare to purchase the necessary ingredients.

[1591] User operation steps

[1592] Step 1: Recipe Selection

[1593] The user selects the recipe they want to make from the recipe list displayed on the terminal. The input is the recipe list displayed on the terminal, and the output is the selected recipe.

[1594] Step 2: Check and purchase ingredients

[1595] The user checks the ingredients needed based on the recipe and purchases any missing ingredients. The input is the list of ingredients for the recipe, and the output is the purchased ingredients.

[1596] Step 3: Provide emotion data

[1597] The system recognizes emotions from the user's facial expressions and voice in real time and transmits them to the server. The input is the user's facial expressions and voice, and the output is emotional data sent to the server.

[1598] Step 4: Provide feedback

[1599] After executing the selected recipe, the user inputs the results and their impressions as feedback into the terminal. The input is the recipe execution results and impressions, and the output is feedback data sent to the server.

[1600] (Application example 2)

[1601] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1602] In recent years, there has been a demand for making effective use of ingredients in the refrigerator and reducing waste, but it has been difficult for users to find recipes that suit their preferences and emotions. Furthermore, when using delivery services, food options are not optimized because suggestions are not made that take into account the ingredients in the refrigerator or the user's emotions. This has led to problems such as wasted ingredients in the refrigerator and inappropriate food delivery suggestions. The present invention aims to solve these problems by efficiently utilizing ingredients in the refrigerator and realizing appropriate recipe and delivery suggestions that match the user's emotions and preferences.

[1603] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1604] In this invention, the server includes means for analyzing images captured by a camera installed inside the refrigerator and recognizing ingredients in the refrigerator, means for suggesting recipes that can be made using only the ingredients in the refrigerator based on the recognized ingredients, means for suggesting recipes that can be made by adding additional ingredients to the recognized ingredients, means for learning the user's past recipe selection history and suggesting recipes based on the user's preferences, means for analyzing the user's emotions and suggesting recipes based on the analyzed emotions, and means for displaying on a user interface recipes based on the ingredients in the refrigerator, recipes that require additional ingredients, recipes based on the user's preferences, and recipes based on the user's emotions.This makes it possible to efficiently use ingredients in the refrigerator without waste and to suggest personalized recipes and delivery options that suit the user's emotions and preferences.

[1605] A "camera installed inside a refrigerator" is a device that takes pictures of the inside of a refrigerator and obtains information about the internal condition and ingredients.

[1606] "Means for analyzing images and recognizing ingredients in the refrigerator" refers to algorithms or software that process image data captured by a camera and identify the types and quantities of ingredients contained within.

[1607] "Means to suggest recipes that can be made using only ingredients in the refrigerator" is a function that presents users with recipes for dishes that can be made without using any additional ingredients, based on the information of recognized ingredients.

[1608] "Means for suggesting recipes that can be made by adding additional ingredients" is a function that presents users with recipes for dishes that can be made by taking into account the ingredients in the refrigerator and other ingredients that can be added from outside.

[1609] "Means for learning the user's past recipe selection history and suggesting recipes based on the user's preferences" is a function that analyzes the history of recipes that the user has selected in the past and suggests recipes that suit the user's preferences based on the results.

[1610] "Means for analyzing the user's emotions and suggesting recipes based on the analyzed emotions" is a function that evaluates the user's emotions from their facial expressions, voice, etc., and suggests the optimal recipe based on their emotional state.

[1611] "Means for displaying information in a user interface" refers to a display or screen display device that provides information in a form that can be seen by the user.

[1612] The "means for generating a list of additional ingredients" is a function that lists ingredients required for the recipe selected by the user that are in short supply in the refrigerator and presents them as a shopping list.

[1613] "Means to suggest new recipes" is a function that suggests recipes for new genres or types of food that the user has not tried before, based on the user's past recipe selection history.

[1614] "Means for suggesting ingredients based on the user's purchase history" is a function that analyzes the history of ingredients that the user has purchased and suggests new ingredients that should be purchased based on that history.

[1615] The system for realizing this invention consists of an algorithm that analyzes images from a camera installed inside the refrigerator and recognizes ingredients, an engine that analyzes the user's preferences and emotions, and a server and terminal that suggests recipes and delivery options based on this data.

[1616] Server Processing

[1617] The server first periodically acquires image data from a camera installed inside the refrigerator and preprocesses the images. The hardware used includes a camera and server, and software such as OpenCV and TensorFlow. OpenCV is used to remove noise and adjust the resolution of the images, and TensorFlow is used to analyze the images using a deep learning model and recognize ingredients.

[1618] The system then stores the recognized ingredient data in an ingredient database, and uses that data to reference a recipe database to search for recipes that can be made using only the ingredients in the refrigerator, as well as recipes that require additional ingredients. It also learns the user's preferences using the user's past recipe selection history and rating data. Furthermore, it uses an emotion engine to analyze the user's emotional state in real time and collects emotion data.

[1619] We combine these data to generate the following recipe list:

[1620] Recipes that can be made using only ingredients in your refrigerator

[1621] Recipes and lists that require additional ingredients

[1622] Recipes based on user preferences

[1623] New recipes

[1624] Recipes based on user sentiment

[1625] Terminal handling

[1626] The device displays the recipe list sent from the server on a user interface. A smartphone or tablet is used for the display, and detailed information such as a recipe summary, difficulty level, and required time is also provided. When the user selects a specific recipe, a list of additional ingredients is generated and presented to the user.

[1627] User operations

[1628] The user selects the recipe they want to make from a list of recipes displayed on their device and purchases additional ingredients as needed. The system also recognizes the user's emotions in real time from their facial expressions and voice, and sends this data to the server. After executing the recipe, the system provides feedback on the results and impressions, which are used to learn the user's preference data.

[1629] Specific examples

[1630] As a concrete example, consider a situation where the user has "eggs, milk, and cheese" in the refrigerator and is feeling "stressed." Based on this, the server can suggest recipes and delivery options such as a cheese omelet or stress-relieving herbal tea. Also, based on the user's past preference for "Japanese food," the server can suggest a new type of "Western food" recipe this time.

[1631] Prompt Sentence Examples

[1632] "Please recognize ingredients from camera images inside the refrigerator and analyze emotions from the user's facial expressions and voice. Based on the results, please generate a program that will suggest the best recipes and food delivery options for the user."

[1633] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1634] Processing Steps

[1635] Step 1:

[1636] The server periodically acquires image data from a camera installed inside the refrigerator. The image data captured by the camera is provided as input and is used for image pre-processing in the next step.

[1637] Step 2:

[1638] The server preprocesses the acquired image data. Here, OpenCV is used to perform basic image processing such as removing noise from the image and adjusting the resolution. The input is raw image data from the camera, and the output is image data that has been denoised and adjusted for resolution.

[1639] Step 3:

[1640] The server analyzes the preprocessed image data to recognize ingredients. A deep learning model using TensorFlow identifies the type and quantity of ingredients and stores the results in an ingredient database. The input is the preprocessed image data, and the output is the type and quantity data of ingredients.

[1641] Step 4:

[1642] Based on the information in the ingredient database, the server searches for recipes that can be made using only the ingredients in the refrigerator and recipes that require additional ingredients. It also references the recipe database and generates a list of ingredients required for each recipe. The input is the type and quantity data of ingredients, and the output is a list of recipes and a list of required ingredients.

[1643] Step 5:

[1644] The server learns the user's past recipe selection history and suggests recipes based on their preferences. Using the past selection data as learning data, it trains a preference prediction model and suggests recipes that match the user's preferences. The input is the past recipe selection history, and the output is a recipe based on the user's preferences.

[1645] Step 6:

[1646] The server uses an emotion engine to analyze the user's emotions. It uses facial expression images and voice data obtained by a camera and microphone as input to identify the user's emotional state. The input is the user's facial expression images and voice data, and the output is emotion data.

[1647] Step 7:

[1648] The server generates a recipe list by integrating recipes based on ingredients in the refrigerator, recipes that require additional ingredients, recipes based on preferences, and recipes based on emotions. This creates personalized recipe suggestions that take various data into consideration. The input is ingredient data, preference data, and emotion data, and the output is a recipe list.

[1649] Step 8:

[1650] The server sends the generated recipe list to the terminal, along with detailed recipe information and a list of required ingredients. The input is the recipe list, and the output is the recipe data sent to the terminal.

[1651] Step 9:

[1652] The terminal displays the recipe list received from the server, and when the user selects a specific recipe, displays its detailed information. If necessary, it also generates and displays additional ingredient lists. The input is the recipe list received from the server, and the output is the display on the user interface.

[1653] Step 10:

[1654] The user selects the recipe they want to make from the recipe list displayed on the device, and checks and purchases any additional ingredients they need. The user provides emotional data from their facial expressions and voice, and also inputs feedback. The input is the displayed recipe list and a list of additional ingredients, and the output is the selected recipe and feedback data.

[1655] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1656] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1657] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1658] [Fourth embodiment]

[1659] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1660] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1661] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1662] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1663] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1664] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1665] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1666] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1667] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1668] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1669] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1670] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1671] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1672] This invention relates to a system that uses an image analysis system with a camera installed inside a refrigerator to recognize ingredients in the refrigerator and then suggests optimal recipes to the user based on that information. This system is composed of a server, a terminal, and user interaction.

[1673] Server Processing

[1674] 1. Acquiring camera images

[1675] A camera installed inside the refrigerator periodically takes pictures of the inside of the refrigerator, and the server captures and receives these pictures.

[1676] 2. Image analysis and food ingredient recognition

[1677] The server preprocesses the received image data to remove noise and adjust the resolution, then applies image analysis algorithms (e.g., deep learning models for object recognition) to identify the ingredients in the refrigerator.

[1678] The type and quantity of recognized ingredients are stored in a database, and the freshness and expiration date of the ingredients are also recorded.

[1679] 3. Referencing the recipe database

[1680] The server references the ingredient database to search for recipes that can be made using only the ingredients in the refrigerator. It also searches for recipes that require additional ingredients and generates a list of the additional ingredients required.

[1681] 4. Learning user preference data

[1682] The server collects the user's past recipe selection history and rating data, and trains a machine learning model to predict the user's preferences. Based on this data, it suggests recipes that suit the user's tastes and new recipes that the user has not tried yet.

[1683] 5. Generating Recipe Suggestions

[1684] The server creates the following recipe list:

[1685] Recipes that can be made using only ingredients in your refrigerator

[1686] Recipes that require additional ingredients and a list of the additional ingredients required

[1687] Recipes based on user preferences

[1688] New recipes

[1689] 6. Data transmission

[1690] The server sends the generated recipe list to the terminal.

[1691] Terminal handling

[1692] 1. Display the recipe list

[1693] The terminal displays the recipe list received from the server on the user interface.

[1694] 2. View detailed recipe information

[1695] When a user selects a recipe, the device displays detailed information about the recipe, including ingredients, steps, and any additional ingredients needed.

[1696] 3. Generate a shopping list

[1697] The terminal displays a shopping list of additional ingredients based on the recipe selected by the user.

[1698] User operations

[1699] 1. Recipe selection

[1700] The user selects the recipe they want to make from the recipe list displayed on the terminal.

[1701] 2. Check and purchase ingredients

[1702] Users check the ingredients needed based on the recipe and purchase any missing ingredients.

[1703] 3. Enter your feedback

[1704] After the user executes the selected recipe, they input their feedback into the device, which then sends this data to the server and updates the user preference database.

[1705] Specific examples

[1706] Example 1: Recipes that can be made using only ingredients in your refrigerator

[1707] situation

[1708] The refrigerator contains "eggs, milk, butter, flour, and bacon."

[1709] Server Processing

[1710] These ingredients are recognized using image analysis and the ingredient database is updated.

[1711] Search the database for "hotcake recipes" and "bacon and egg recipes" and add them to the list.

[1712] Terminal display

[1713] "Hotcakes" and "Bacon and eggs" are displayed to the user as options.

[1714] Example 2: Recipe that requires additional ingredients

[1715] situation

[1716] The refrigerator contains "tomatoes, lettuce, cheese, and bread."

[1717] Server Processing

[1718] These ingredients are recognized using image analysis and the ingredient database is updated.

[1719] Search the database for "sandwich recipes" and see that they call for ham.

[1720] Terminal display

[1721] Display "Sandwich (additional ham required)" as a suggestion to the user.

[1722] Example 3: Recipes based on user preferences

[1723] situation

[1724] The user has often selected "Japanese food" in the past.

[1725] Server Processing

[1726] It learns user preference data and searches for Japanese recipes (e.g., "meat and potato stew" and "miso soup").

[1727] As a new suggestion, add "lasagna" to the list of Western food options.

[1728] Terminal display

[1729] The user is prompted to select from "nikujaga," "miso soup," and "lasagna."

[1730] This system allows users to maximize the use of ingredients in their refrigerators while also allowing them to try new dishes as needed, reducing the time it takes to decide what to cook, reducing food waste, and enabling a wider variety of meals.

[1731] The processing flow will be explained below.

[1732] Step 1:

[1733] Camera image acquisition

[1734] The server periodically acquires images from a camera installed inside the refrigerator.

[1735] The camera takes pictures of each shelf and compartment in the refrigerator and generates image data.

[1736] Step 2:

[1737] Image data preprocessing

[1738] The server performs noise removal and resolution adjustment on the acquired image data.

[1739] The preprocessed image data is saved in preparation for analysis.

[1740] Step 3:

[1741] Image analysis and food ingredient recognition

[1742] The server uses deep learning models for object recognition to analyze the pre-processed image data.

[1743] The server uses the model to recognize ingredients in the image and identify the type and quantity of each ingredient.

[1744] Step 4:

[1745] Ingredient database update

[1746] The server stores the recognized ingredient information in an ingredient database.

[1747] The freshness and expiration date of ingredients are also recorded at the same time, keeping the database up to date.

[1748] Step 5:

[1749] Recipe database reference

[1750] The server refers to the ingredient database and searches for recipes that can be made using only the ingredients in the refrigerator.

[1751] It also searches for recipes that require additional ingredients and generates a list of the required additional ingredients.

[1752] Step 6:

[1753] Learning user preference data

[1754] The server collects the user's past recipe selection history and evaluation data.

[1755] The server uses this data to train machine learning models to predict user preferences.

[1756] Step 7:

[1757] Creating a recipe list

[1758] The server creates the following recipe list:

[1759] Recipes that can be made using only ingredients in your refrigerator

[1760] Recipes that require additional ingredients and a list of the additional ingredients required

[1761] Recipes based on user preferences

[1762] New recipes

[1763] Step 8:

[1764] Sending data

[1765] The server sends the generated recipe list to the terminal.

[1766] Step 9:

[1767] Displaying the recipe list

[1768] The terminal displays the recipe list received from the server on the user interface.

[1769] To make it easier for users to select a recipe, the app also displays an overview of each recipe, its difficulty level, and the time required.

[1770] Step 10:

[1771] Viewing detailed recipe information

[1772] When a user selects a particular recipe, the device displays detailed information about that recipe, including ingredients, steps, and any additional ingredients needed.

[1773] Step 11:

[1774] Generate a shopping list

[1775] The terminal displays a shopping list of ingredients that are missing based on the recipe selected by the user.

[1776] The user checks the shopping list and prepares to purchase the necessary ingredients.

[1777] Step 12:

[1778] Enter your feedback

[1779] After the user executes the selected recipe, the results are input as feedback to the terminal.

[1780] Feedback includes the quality of the dish, its difficulty, and individual ratings.

[1781] Step 13:

[1782] Sending feedback data

[1783] The terminal transmits feedback data from the user to the server.

[1784] Step 14:

[1785] Database Update

[1786] The server updates the user's preference database based on the received feedback data.

[1787] Use the updated data to improve your next recipe suggestion.

[1788] Through this series of processing steps, the system will suggest optimal recipes to the user, enabling them to make efficient use of the ingredients in their refrigerator.

[1789] Example 1

[1790] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1791] In modern life, people need to effectively manage the ingredients in their refrigerators and use them up without wasting any. However, visually checking the contents of the refrigerator and planning recipes based on that information is time-consuming and labor-intensive. It is also difficult to suggest optimal recipes based on the user's preferences and past cooking history. Furthermore, it is time-consuming to have to check and purchase ingredients that are missing. To solve these issues, a system is needed that can efficiently recognize ingredients in the refrigerator and suggest optimal recipes to users.

[1792] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1793] In this invention, the server includes means for analyzing images captured by an image capture device installed in the refrigerator and recognizing food in the refrigerator, means for suggesting dishes that can be made using only the food in the refrigerator based on the recognized food, means for suggesting dishes that can be made by adding additional ingredients to the recognized food, means for learning the user's past dish selection history and suggesting dishes based on the user's preferences, means for predicting the user's preferences using a machine learning model and suggesting dishes based on the predicted preferences, and means for displaying on a display device dishes based on the food in the refrigerator, dishes requiring additional ingredients, and dishes based on the user's preferences. This allows the user to efficiently manage ingredients in the refrigerator and receive optimal recipe suggestions, thereby reducing the effort required for cooking and reducing food waste.

[1794] An "image capture device" is a device that is installed inside a refrigerator and is used to capture images.

[1795] "Analysis" refers to the act of processing acquired image data and extracting necessary information.

[1796] "Food" refers to all food ingredients and beverages stored in the refrigerator.

[1797] The "means of suggestion" is a means of selecting appropriate recipes and information based on predetermined conditions and providing them to the user.

[1798] "Cuisine" refers to food prepared using specific ingredients.

[1799] "Additional ingredients" are new ingredients or seasonings that are needed in addition to the ingredients already in the refrigerator.

[1800] "User preferences" refer to preferences and favorite foods that are inferred based on the user's past selections and ratings of dishes.

[1801] A "machine learning model" refers to an algorithm or mathematical model that analyzes large amounts of data and finds patterns.

[1802] "Prediction methods" are methods for predicting future trends and results based on past data and analytical results.

[1803] A "display device" is a device that functions as a user interface and visually presents various pieces of information.

[1804] The present invention relates to a system that efficiently manages food in a refrigerator and suggests optimal recipes to users. The system is composed of a server, a terminal, and user interaction. The following describes in detail an embodiment of the present invention.

[1805] Server configuration and operation

[1806] The server receives and analyzes images captured by the image capture device installed inside the refrigerator. Specifically, it uses the following hardware and software:

[1807] Hardware:

[1808] Camera installed inside the refrigerator

[1809] Server itself (equipped with a high-performance CPU, sufficient memory, and network interface)

[1810] software:

[1811] Image processing library (e.g. OpenCV)

[1812] Object recognition libraries (e.g. TensorFlow, YOLO)

[1813] Database management system (e.g. MySQL)

[1814] Machine learning libraries (e.g., scikit-learn)

[1815] The server performs the following steps to recognize the food in the refrigerator:

[1816] 1. Image Acquisition: A camera installed inside the refrigerator periodically takes images and sends them to a server, which receives the image data via Wi-Fi or a wired connection.

[1817] 2. Image analysis: The received image data is first pre-processed to remove noise and adjust the resolution, then an object recognition algorithm is used to identify food in the image.

[1818] 3. Storage in database: Information such as the type, quantity, freshness, and expiration date of the recognized food is stored in a database.

[1819] Recipe suggestion function

[1820] The server suggests suitable recipes to the user by:

[1821] 1. Recipes that can be made using only food in the refrigerator: Search for relevant recipes based on food information in the database.

[1822] 2. Recipes that require additional ingredients: Search for recipes that can be made by combining ingredients in the refrigerator with additional ingredients, and generate a list of the additional ingredients required.

[1823] 3. Recipes based on user preferences: We collect users' past recipe selection history and rating data, and use machine learning models to predict their preferences. Based on this data, we then suggest optimal recipes.

[1824] Terminal configuration and operation

[1825] The terminal has the following user interface features:

[1826] 1. Displaying the recipe list: Display the recipe list received from the server in the user interface. The recipe list is dynamically generated and displayed using HTML and JavaScript.

[1827] 2. Displaying detailed recipe information: When a user selects a recipe, detailed information (ingredients, steps, required additional ingredients, etc.) is displayed. This is done by retrieving data from the server using Ajax.

[1828] 3. Generate shopping list: Based on the recipe selected by the user, additional ingredients are displayed as a shopping list.

[1829] User operations

[1830] Users use this system through the following operations:

[1831] 1. Recipe selection: Select the recipe you want to make from the recipe list displayed on your device.

[1832] 2. Check and purchase ingredients: Check the ingredients needed based on the recipe and purchase any that are missing.

[1833] 3. Entering feedback: After executing the selected recipe, the user enters the feedback into the device, which then sends this data to the server and updates the user's preference database.

[1834] Specific examples

[1835] Here are some concrete usage examples:

[1836] Example of a recipe that can be made using only ingredients in the refrigerator: If the refrigerator contains "eggs, milk, butter, flour, and bacon," the server will recognize these ingredients and suggest recipes for pancakes or bacon and eggs.

[1837] Example of a recipe that requires additional ingredients: If the refrigerator contains "tomatoes, lettuce, cheese, and bread," the server suggests a sandwich and informs the user that additional ham is needed.

[1838] Example of recipes based on user preferences: If the user has frequently selected "Japanese food" in the past, the server will suggest recipes for Nikujaga (meat and potato stew) and Miso soup. It will also add Lasagna to the list as a new suggestion.

[1839] Examples of prompt statements

[1840] "Please tell me a recipe I can make using the eggs, milk, butter, flour, and bacon I have in my fridge."

[1841] This invention allows users to efficiently manage food in their refrigerators and reduce the effort required for cooking. In addition, the system continuously learns through feedback, enabling it to suggest more accurate recipes.

[1842] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1843] Server Processing

[1844] Step 1: Acquire an image

[1845] Operation: A camera placed inside the refrigerator periodically takes images of the inside of the refrigerator.

[1846] Input: Image data captured by the camera.

[1847] How it works: The camera takes pictures of the inside of the refrigerator and sends them over WiFi or a wired connection to a server, which then downloads the image data using HTTP requests or the FTP protocol.

[1848] Output: Image data received on the server side.

[1849] Step 2: Image analysis

[1850] Operation: The server preprocesses the received image data and recognizes the food.

[1851] Input: The image data received in step 1.

[1852] How it works: The server uses image processing libraries such as OpenCV to remove noise and adjust the resolution, then uses TensorFlow to apply an object recognition model to identify and classify food items in the image.

[1853] Output: Recognized food types and quantities.

[1854] Step 3: Saving to the database

[1855] Operation: The server stores the recognized food data in a database.

[1856] Input: Food type and quantity data obtained in step 2.

[1857] How it works: The server connects to a database management system such as MySQL and uses the "INSERT" statement to record food names, quantities, freshness information, etc. into the database.

[1858] Output: Food data stored in a database.

[1859] Step 4: Recipe suggestions

[1860] Operation: The server refers to the food database to search and suggest recipes suitable for the user.

[1861] Input: Food data saved in step 3.

[1862] How it works: The server uses an SQL query to search the recipe table in the database to find recipes that match the food list in the refrigerator. It also searches for recipes that require additional ingredients and generates a list of the additional ingredients in JSON format.

[1863] Output: A list of recipes that can be made using only the food in the refrigerator, and a list of recipes that require additional ingredients.

[1864] Step 5: Learning user preferences

[1865] Operation: The server trains a machine learning model based on the user's past recipe selection history.

[1866] Input: User's past recipe selection history data.

[1867] How it works: The server uses pandas to read historical data and scikit-learn to train a machine learning model that predicts user preferences and generates recipes based on them.

[1868] Output: A list of recipes based on the user's preferences.

[1869] Step 6: Sending data

[1870] Operation: The server sends a recipe list to the device.

[1871] Input: The recipe list generated in step 4 and step 5.

[1872] How it works: The server sends an HTTP POST request to the device via a RESTful API, sending a list of recipes in JSON format.

[1873] Output: Recipe list sent to the device.

[1874] Terminal handling

[1875] Step 1: View the recipe list

[1876] Operation: The terminal displays the recipe list received from the server on the user interface.

[1877] Input: Recipe list sent from the server.

[1878] How it works: Using HTML and JavaScript, the received recipe list is dynamically generated and displayed on a web page.

[1879] Output: The recipe list displayed in the user interface.

[1880] Step 2: Viewing Recipe Details

[1881] Action: The user selects a recipe.

[1882] Input: User's choice.

[1883] How it works: The device uses Ajax to retrieve detailed information about the selected recipe from the server and displays it, including ingredients, steps, and any additional ingredients needed.

[1884] Output: Detailed information about the recipe.

[1885] Step 3: Generate a shopping list

[1886] Operation: Generate a shopping list based on the user's selected recipes.

[1887] Input: The recipe data selected by the user.

[1888] What it does: Dynamically generates a list of additional ingredients using JavaScript and displays it as a shopping list.

[1889] Output: Purchase list.

[1890] User operations

[1891] Step 1: Recipe Selection

[1892] Operation: The user selects the recipe they want to make from the recipe list displayed on the device.

[1893] Input: A list of recipes displayed on the terminal.

[1894] Action: The user selects a recipe via a select box or click event.

[1895] Output: The selected recipe.

[1896] Step 2: Check and purchase ingredients

[1897] Operation: The user checks the ingredients needed based on the recipe and purchases any missing ingredients.

[1898] Input: Recipe details.

[1899] What it does: The user looks at the shopping list and notes or buys the ingredients they need.

[1900] Output: A list of ingredients confirmed.

[1901] Step 3: Provide feedback

[1902] Operation: The user executes the selected recipe and then enters their feedback into the terminal.

[1903] Input: User feedback data.

[1904] Behavior: A feedback input form is provided, and after the user enters their rating and comments, it is sent to the server.

[1905] Output: Feedback data sent to the server.

[1906] This allows users to efficiently manage the ingredients in their refrigerators and receive appropriate recipe suggestions, reducing the effort required for cooking and reducing food waste. In addition, the system continues to learn through feedback, allowing it to make even more accurate suggestions.

[1907] (Application example 1)

[1908] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1909] Conventional refrigerator food management systems only allowed users to track the ingredients in their refrigerators and suggested recipes. Furthermore, users had to go shopping separately to get ingredients they were running low on, which was inconvenient. Furthermore, the system lacked the ability to suggest new recipes based on the user's preferences. This limited the opportunities for users to efficiently use the ingredients in their refrigerators and try new recipes.

[1910] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1911] In this invention, the server includes means for analyzing images captured by a camera installed inside the refrigerator and recognizing ingredients in the refrigerator, means for suggesting recipes that can be made using only the ingredients in the refrigerator based on the recognized ingredients, means for suggesting recipes that can be made by adding additional ingredients to the recognized ingredients, means for learning the user's past recipe selection history and suggesting recipes based on the user's preferences, means for displaying on a user interface recipes based on the ingredients in the refrigerator, recipes that require additional ingredients, and recipes based on the user's preferences, means for recognizing ingredients that are missing from a recipe selected by the user and generating a list of those ingredients, and means for enabling the user to order missing ingredients for delivery based on the generated list of missing ingredients. This allows the user to use the ingredients in the refrigerator efficiently, expand recipe options, and easily procure missing ingredients.

[1912] A "camera installed inside a refrigerator" is a camera device that is attached inside a refrigerator and captures video or images of the interior.

[1913] "Analyzing images" is the process of recognizing and identifying specific objects or elements based on the acquired image data.

[1914] "Means for recognizing ingredients" refers to a method or device for identifying specific foods or ingredients in a refrigerator through image analysis.

[1915] The "means for suggesting recipes that can be made using only ingredients in the refrigerator" is a method or device that presents to the user recipes for dishes that do not require additional ingredients, based on recognized ingredients.

[1916] The "means for suggesting recipes that can be made by adding additional ingredients" is a method or device that presents to the user how to make dishes that can be made by adding missing ingredients to the recognized ingredients.

[1917] "Learning a user's past recipe selection history" is the process of collecting data about recipes previously selected by a user and analyzing that data.

[1918] The "means for proposing recipes based on user preferences" is a method or device for presenting recipes that match the collected and learned preferences of the user.

[1919] The "means for displaying on a user interface" refers to a method or apparatus for displaying suggested recipes and information on the screen of a device used by a user.

[1920] The "means for recognizing missing ingredients and generating a list thereof" is a method or device that identifies ingredients that are required for a recipe selected by the user but are not present in the refrigerator and lists them.

[1921] A "means for enabling delivery ordering of missing ingredients" is a method or device that enables identified missing ingredients to be purchased directly via an online ordering system.

[1922] An embodiment of the present invention uses hardware such as a server, a smartphone, and a camera inside a refrigerator, and software such as an image analysis algorithm, a recipe database, a user interface, and delivery service collaboration software.

[1923] The server analyzes images captured by a camera installed inside the refrigerator and recognizes the ingredients inside. A deep learning model is used as the image analysis algorithm for recognition. For example, frameworks such as TensorFlow and PyTorch can be used. Preprocessing involves noise removal and resolution adjustment, followed by object recognition to identify the ingredients. Information about the recognized ingredients is stored in a database.

[1924] The server then suggests recipes that can be made using only the ingredients in the refrigerator based on the recognized ingredients. To do this, the server refers to the ingredient database and searches for relevant recipes.

[1925] The server then suggests recipes that can be made by adding additional ingredients to the recognized ingredients, searches the database for the necessary additional ingredients, and generates a list of these ingredients, which is then displayed together with the recipe selected by the user.

[1926] The server also learns the user's past recipe selection history and suggests recipes based on the user's preferences. By using a machine learning model, the server can learn the user's preference patterns and provide recipes tailored to their preferences.

[1927] The user interface displays these recipes on the smartphone screen. The user interacts with the server through the smartphone and selects from the suggested recipes. A list of ingredients missing from the selected recipe is generated, and this information can be used to order delivery via delivery service integration software.

[1928] For example, a user can take a photo of the ingredients in their refrigerator with their smartphone and send the image to a server via the app. The server analyzes the image and recognizes the ingredients. The server then suggests recipes that can be made with the recognized ingredients, as well as dishes that require additional ingredients, and displays these to the user via a user interface. When the user selects a recipe, a list of missing ingredients is displayed, allowing them to order delivery.

[1929] Specifically, this involves suggesting new dishes (such as Western food if the user mostly chooses Japanese food) based on the user's past recipe preferences, and ordering missing ingredients through a delivery service.

[1930] An example of a specific prompt using a generative AI model is, "Based on the results of image analysis of the inside of the refrigerator, please suggest recipes that can be made with the ingredients I currently have. Also, please generate a list of ingredients that I am running low on."

[1931] This allows users to efficiently use ingredients in their refrigerator, expand their recipe options, and easily procure ingredients that they are running low on.

[1932] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1933] Step 1:

[1934] The user takes a picture of the inside of the refrigerator with the smartphone camera and sends it to the server via the app. The input is image data of the inside of the refrigerator, and the output is the image data sent to the server. The user records the state of the inside of the refrigerator using the smartphone camera and uploads the image to the server using the send button on the app.

[1935] Step 2:

[1936] The server preprocesses the received image data. The input is the transmitted image data, and the output is the preprocessed image data. Specific operations include noise reduction and resolution adjustment. For example, it applies a filter to reduce image noise and resizes the image to a resolution suitable for analysis.

[1937] Step 3:

[1938] The server analyzes the preprocessed image data using a deep learning model to recognize the ingredients in the refrigerator. The input is the preprocessed image data, and the output is a list of recognized ingredients. Specifically, it uses an object detection algorithm to identify each ingredient in the image and extract the name and quantity of each ingredient.

[1939] Step 4:

[1940] The server saves the recognized ingredient list in a database. The input is the recognized ingredient list, and the output is the latest ingredient database. Specifically, the operation involves connecting to the database and adding or updating new ingredient information.

[1941] Step 5:

[1942] The server references the ingredient database to search for recipes that can be made using only the ingredients in the refrigerator. The input is the ingredient database, and the output is a list of matching recipes. Specifically, it executes a database query to retrieve recipes based on the currently available ingredients.

[1943] Step 6:

[1944] The server searches for recipes that require additional ingredients and generates a list of missing ingredients. The input is an ingredient database and a list of recognized ingredients, and the output is a list of recipes with the additional ingredients and a list of missing ingredients. Specifically, the server performs a database query to identify ingredients that are missing from existing ingredients and then lists them.

[1945] Step 7:

[1946] The server learns the user's past recipe selection history and suggests recipes based on the user's preferences. The input is the user's selection history data, and the output is a personalized recipe list. Specifically, it uses a machine learning algorithm to analyze past data and identify the user's preference patterns.

[1947] Step 8:

[1948] The device displays the recipe list received from the server on the user interface. The input is the recipe list sent from the server, and the output is the recipe list displayed on the smartphone screen. The user can select the desired recipe from the displayed list.

[1949] Step 9:

[1950] The system checks the ingredients missing from the recipe selected by the user and displays a shopping list of the additional ingredients needed. The input is the selected recipe and the output is the shopping list. The user checks the displayed list and identifies the missing ingredients.

[1951] Step 10:

[1952] The terminal allows users to order missing ingredients for delivery based on the shopping list. The input is the shopping list, and the output is the execution of the delivery order. Specifically, the order details are sent via the delivery service's API, and the ingredients are delivered to the user.

[1953] Through the above processing steps, the user can efficiently utilize ingredients in the refrigerator, find suitable recipes, and easily procure ingredients that are in short supply.

[1954] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1955] This invention is a system that combines an image analysis system using a camera installed inside the refrigerator with an emotion engine that recognizes the user's emotions to make optimal use of ingredients in the refrigerator and suggest recipes based on the user's preferences and emotions. This system is composed of interactions between a server, terminals, and users, and realizes more personalized recipe suggestions.

[1956] Server Processing

[1957] 1. Acquiring camera images

[1958] The server periodically acquires images from a camera installed inside the refrigerator.

[1959] The camera takes pictures of each shelf and compartment inside the refrigerator and generates image data.

[1960] 2. Image data preprocessing

[1961] The server preprocesses the acquired image data, removing noise and adjusting the resolution.

[1962] The preprocessed image data is used for analysis.

[1963] 3. Image analysis and food ingredient recognition

[1964] The server analyzes the image data using a deep learning model to identify the type and quantity of ingredients.

[1965] The recognition results are stored in a food ingredient database, and freshness and expiration date are also recorded.

[1966] 4. Referencing the recipe database

[1967] The server refers to the ingredient database and searches for recipes that can be made using only the ingredients in the refrigerator.

[1968] It also searches for recipes that require additional ingredients and generates a list of the ingredients needed.

[1969] 5. Learning User Preference Data

[1970] The server collects past recipe selection history and rating data to train a preference prediction model.

[1971] Used to suggest recipes based on the user's preferences.

[1972] 6. Use of Emotion Engine

[1973] The server uses an emotion engine to recognize the user's emotions in real time.

[1974] The recognized emotion data is reflected in recipe suggestions.

[1975] 7. Generating Recipe Suggestions

[1976] The server generates the following recipe list:

[1977] Recipes that can be made using only ingredients in your refrigerator

[1978] Recipes and lists that require additional ingredients

[1979] Recipes based on user preferences

[1980] New recipes

[1981] Recipes based on user sentiment

[1982] 8. Data transmission

[1983] The server sends the generated recipe list to the terminal.

[1984] Terminal handling

[1985] 1. Display the recipe list

[1986] The terminal displays the recipe list received from the server on the user interface.

[1987] It also displays an overview of the recipe, its difficulty level, and the time required.

[1988] 2. View detailed recipe information

[1989] When a user selects a particular recipe, the device displays detailed information, including ingredients, steps, and any additional ingredients needed.

[1990] 3. Generate a shopping list

[1991] The device displays a shopping list of additional ingredients based on the selected recipe.

[1992] The user checks the list and prepares to purchase the necessary ingredients.

[1993] User operations

[1994] 1. Recipe selection

[1995] The user selects the recipe they want to make from the recipe list displayed on the terminal.

[1996] 2. Check and purchase ingredients

[1997] Users check the ingredients needed based on the recipe and purchase any missing ingredients.

[1998] 3. Providing Emotion Data

[1999] The system recognizes emotions from the user's facial expressions and voice in real time and transmits them to the server.

[2000] 4. Enter your feedback

[2001] After executing the selected recipe, the user inputs the results and impressions as feedback into the terminal.

[2002] Specific examples

[2003] Example 1: Recipe suggestions based on ingredients in the refrigerator and emotions

[2004] situation

[2005] The refrigerator contains "eggs, milk, butter, flour, and bacon," and the user is relaxed.

[2006] Server Processing

[2007] Ingredients are recognized through image analysis and the ingredient database is updated.

[2008] The emotion engine recognizes the user's relaxed state.

[2009] It searches the database for "hotcakes" and "bacon and eggs" and recommends "hotcakes" that best suits the user's condition.

[2010] Terminal display

[2011] "Hotcakes" is displayed to the user, along with detailed recipe information.

[2012] Example 2: User preferences and new recipe suggestions

[2013] situation

[2014] The user has often selected "Japanese food" in the past and is now interested in new cuisine.

[2015] Server Processing

[2016] It learns user preference data and suggests Japanese dishes such as "meat and potato stew" and "miso soup."

[2017] Add "lasagna" as a new suggestion.

[2018] Terminal display

[2019] It displays dishes such as "nikujaga" (beef and potato stew), "miso soup" and "lasagna" to encourage people to try new dishes.

[2020] This system allows users to make the most of the ingredients in their refrigerator and easily find dishes that suit their tastes and emotions, significantly reducing the time it takes to decide what to cook, minimizing food waste, and increasing the variety of their daily meals.

[2021] The processing flow will be explained below.

[2022] Step 1:

[2023] Camera image acquisition

[2024] The server periodically acquires images from a camera installed inside the refrigerator.

[2025] The camera takes pictures of each shelf and compartment inside the refrigerator and generates image data.

[2026] Step 2:

[2027] Image data preprocessing

[2028] The server preprocesses the acquired image data, removing noise and adjusting the resolution.

[2029] Save the preprocessed image data.

[2030] Step 3:

[2031] Image analysis and food ingredient recognition

[2032] The server analyzes the preprocessed image data using a deep learning model to identify the type and quantity of ingredients.

[2033] The results of ingredient recognition are stored in an ingredient database, and freshness and expiration date are also recorded.

[2034] Step 4:

[2035] Acquiring emotion data

[2036] The device captures the user's facial expressions and voice in real time to recognize their emotions.

[2037] An emotion engine analyzes the captured data to determine the user's current emotional state.

[2038] Step 5:

[2039] Recipe database reference

[2040] The server refers to the ingredient database and searches for recipes that can be made using only the ingredients in the refrigerator.

[2041] It also searches for recipes that require additional ingredients and generates a list of the ingredients needed.

[2042] Step 6:

[2043] Learning user preference data

[2044] The server collects past recipe selection history and rating data to train the machine learning model.

[2045] Predict recipes based on user preference data.

[2046] Step 7:

[2047] Emotional Data Integration

[2048] The server collects the recognized emotion data and adjusts the recipe suggestion options based on it.

[2049] If the emotion indicates relaxation, customizations such as suggesting simple and relaxing dishes are performed.

[2050] Step 8:

[2051] Creating a recipe list

[2052] The server generates the following recipe list:

[2053] Recipes that can be made using only ingredients in your refrigerator

[2054] Recipes that require additional ingredients and a list of the additional ingredients required

[2055] Recipes based on user preferences

[2056] New recipes

[2057] Recipes based on user sentiment

[2058] Step 9:

[2059] Sending data

[2060] The server sends the generated recipe list to the terminal.

[2061] Step 10:

[2062] Displaying the recipe list

[2063] The terminal displays the recipe list received from the server on the user interface.

[2064] It also displays an overview of the recipe, its difficulty level, and the time required.

[2065] Step 11:

[2066] Viewing detailed recipe information

[2067] When a user selects a particular recipe, the device displays detailed information, including ingredients, steps, and any additional ingredients needed.

[2068] Step 12:

[2069] Generate a shopping list

[2070] The device displays a shopping list of additional ingredients based on the selected recipe.

[2071] The user checks the list and prepares to purchase the necessary ingredients.

[2072] Step 13:

[2073] Cooking and getting emotional feedback

[2074] The user executes the selected recipe and inputs emotional feedback into the device during the process.

[2075] The emotion engine collects the user's emotion data during and after cooking.

[2076] Step 14:

[2077] Sending feedback data

[2078] The terminal transmits feedback data from the user to the server.

[2079] Step 15:

[2080] Database Update

[2081] The server updates the user's preference database and emotion database based on the received feedback data.

[2082] Use the updated data to improve your next recipe suggestion.

[2083] This system allows users to make the most of the ingredients in their refrigerator and easily find dishes that suit their tastes and emotions. It also utilizes real-time emotional data to provide more personalized recipe suggestions.

[2084] Example 2

[2085] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2086] Conventional refrigerator ingredient management and recipe suggestion systems are limited to suggesting recipes that can be made using only the ingredients in the refrigerator or recipes that require additional ingredients. This does not allow for personalized suggestions based on the user's preferences and emotions, leading to users always making the same dishes and wasting ingredients. Furthermore, the lack of increased cooking options tends to decrease users' motivation to cook. Therefore, the present invention aims to provide more personalized cooking suggestions by taking the user's preferences and emotions into account, thereby improving the enjoyment and efficiency of cooking.

[2087] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[2088] In this invention, the server includes means for analyzing images captured by a camera installed inside the refrigerator and recognizing ingredients in the refrigerator, means for suggesting recipes that can be made using only the ingredients in the refrigerator based on the recognized ingredients, means for suggesting recipes that can be made by adding additional ingredients to the recognized ingredients, means for learning the user's past recipe selection history and suggesting recipes based on the user's preferences, means for recognizing the user's emotional state and suggesting recipes based on the emotions, and means for displaying on a user interface recipes based on the ingredients in the refrigerator, recipes that require additional ingredients, recipes based on the user's preferences, and recipes based on the user's emotions. This allows the user to effectively use the ingredients in the refrigerator and select recipes that suit their preferences and emotions.

[2089] A "camera installed inside a refrigerator" is a device installed inside the refrigerator for taking images of food items inside the refrigerator.

[2090] "Means for analyzing images and recognizing ingredients in the refrigerator" refers to technology and algorithms that use images acquired from a camera to identify the type and quantity of ingredients present in the refrigerator.

[2091] The "means for suggesting recipes that can be made using only ingredients in the refrigerator" is a function that searches for recipes that can be made using only the ingredients in the recognized refrigerator and suggests them to the user.

[2092] "Means for suggesting recipes that can be made by adding additional ingredients" is a function that provides recipes for dishes that can be made by adding other ingredients in addition to the ingredients in the refrigerator.

[2093] "Means for learning a user's past recipe selection history and suggesting recipes based on the user's preferences" refers to technology that collects and analyzes recipes previously selected by the user and their evaluation data, and then suggests recipes that suit the user's preferences based on that data.

[2094] "Means for recognizing the user's emotional state and suggesting recipes based on those emotions" is a function that analyzes the user's emotions in real time from their facial expressions, voice, etc., and suggests recipes that are appropriate for those emotions.

[2095] "Means for displaying on the user interface" refers to the technology or mechanism for displaying suggested recipes and related information on the screen of the device used by the user.

[2096] This invention is a system that combines an image analysis system using a camera installed inside the refrigerator with an emotion engine that recognizes the user's emotions to make optimal use of ingredients in the refrigerator and suggest recipes based on the user's preferences and emotions. This system is composed of interactions between a server, terminals, and users, and realizes more personalized recipe suggestions.

[2097] The server periodically acquires images from a camera installed inside the refrigerator and analyzes the image data to recognize ingredients. This analysis uses a deep learning model. The acquired image data is preprocessed to remove noise and adjust the resolution. This improves the accuracy of the image data, enabling accurate recognition of ingredients.

[2098] Once the ingredients are recognized, the server updates the ingredient database, recording the type, quantity, freshness, and expiration date of each ingredient. Based on this database, the server searches for recipes that can be made using only the ingredients in the refrigerator and suggests them to the user. It also searches for recipes that can be made by adding additional ingredients to the ingredients in the refrigerator and generates a list of the necessary ingredients.

[2099] Furthermore, the server collects the user's past recipe selection history and rating data to train a preference prediction model. This model allows it to suggest recipes based on the user's preferences. Meanwhile, the server uses an emotion engine to recognize the user's emotions in real time and reflects this emotion data in the recipe suggestions. This allows it to suggest simple dishes when the user is relaxed and quick dishes when the user is busy.

[2100] As a concrete example, suppose the refrigerator contains "eggs, milk, butter, flour, and bacon," and the user is in a relaxed state. When the server recognizes these ingredients and the emotion engine understands the user's relaxed state, it searches for recipes for "hotcakes" and "bacon and eggs." Among them, it suggests "hotcakes," which matches the user's relaxed state. The server sends this suggested recipe list to the terminal, which then displays it on the user interface.

[2101] The device accepts input from the user, and when a specific recipe is selected, it displays detailed information about it, allowing the user to check the ingredients needed and cooking steps. If additional ingredients are needed, they are displayed as a shopping list, which the user can use to purchase them. For example, the tomatoes needed to make a "tomato omelette" are added to the shopping list.

[2102] The user selects the recipe they want to make from a list of recipes displayed on the device, checks the ingredients needed, and purchases them. Emotional data from the user's facial expressions and voice is also collected in real time and sent to the server. After cooking, the user enters the results and their impressions as feedback into the device, and this data is used to suggest future recipes.

[2103] As an example of a prompt sentence, if you enter "I'm wondering what to have for dinner tonight," the system will suggest the best recipe based on the ingredients in the refrigerator and the user's preferences and emotions.

[2104] As described above, the system of the present invention makes effective use of ingredients in the refrigerator and proposes recipes that match the user's tastes and emotions, thereby reducing the time it takes to decide on a dish and minimizing the waste of ingredients, making everyday meals more varied and enjoyable.

[2105] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2106] Server Processing Steps

[2107] Step 1: Acquire camera images

[2108] The server periodically acquires images from a camera installed inside the refrigerator. The input is image data captured by the camera, which is sent to the server. The output is the acquired image data, which is stored in the server. Subsequent image processing is performed based on this image data.

[2109] Step 2: Preprocessing the image data

[2110] The server performs pre-processing on the acquired image data. Specific operations include noise removal, resolution adjustment, and image cropping. The input is raw image data acquired from the camera, and the output is clear, pre-processed image data. This processing improves the accuracy of image analysis.

[2111] Step 3: Image analysis and ingredient recognition

[2112] The server uses a deep learning model to recognize ingredients in the refrigerator from preprocessed image data. The input is the preprocessed image data, and the output is data recording the type and quantity of recognized ingredients. This data is stored in an ingredient database. Specifically, it identifies ingredients such as eggs, milk, and butter.

[2113] Step 4: Referencing the recipe database

[2114] The server refers to the ingredient database and searches for recipes that can be made using only the ingredients in the refrigerator. At the same time, it searches for recipes that require additional ingredients and generates a list of the necessary ingredients. The input is the recognition data from the ingredient database, and the output is a list of recommended recipes and a list of additional ingredients. Specifically, it includes recipes that can be made with "eggs and milk," and recipes that can be made by adding "tomatoes."

[2115] Step 5: Learning user preference data

[2116] The server collects the user's past recipe selection history and rating data and trains a preference prediction model. The input is the past recipe selection history and rating data, and the output is a model that learns the user's preference patterns. This model makes it possible to suggest recipes that suit the user's preferences.

[2117] Step 6: Use the Emotion Engine

[2118] The server uses an emotion engine to recognize the user's emotions in real time. The input is emotional data obtained from the user's facial expressions and voice, and the output is the recognized emotional state. Based on this, a recipe appropriate to the user's emotions is selected. For example, if the user is relaxed, a recipe that suits that state is suggested.

[2119] Step 7: Generate recipe suggestions

[2120] The server generates the following recipe list based on the analysis data so far:

[2121] Recipes that can be made using only ingredients in your refrigerator

[2122] Recipes and lists that require additional ingredients

[2123] Recipes based on user preferences

[2124] Recipes based on user sentiment

[2125] The input is food recognition data, preference data, and emotion data, and the output is a list of recommended recipes.

[2126] Step 8: Sending Data

[2127] The server sends the generated recipe list to the terminal. The input is the recommended recipe list, and the output is the list sent to the terminal. This list includes a summary of the recipe, the ingredients required, and cooking instructions.

[2128] Terminal processing steps

[2129] Step 1: View the recipe list

[2130] The terminal displays the recipe list received from the server on the user interface. The input is the recipe list from the server, and the output is the recipe information displayed on the user interface, including the recipe summary, difficulty, and time.

[2131] Step 2: Viewing Recipe Details

[2132] When the user selects a particular recipe, the device displays its details. The input is the user's selected recipe, and the output is details such as ingredients, steps, and any additional ingredients needed.

[2133] Step 3: Generate a shopping list

[2134] The terminal creates and displays a shopping list based on the selected recipe. The input is the selected recipe data, and the output is a shopping list listing the additional ingredients needed. The user can check this list and prepare to purchase the necessary ingredients.

[2135] User operation steps

[2136] Step 1: Recipe Selection

[2137] The user selects the recipe they want to make from the recipe list displayed on the terminal. The input is the recipe list displayed on the terminal, and the output is the selected recipe.

[2138] Step 2: Check and purchase ingredients

[2139] The user checks the ingredients needed based on the recipe and purchases any missing ingredients. The input is the list of ingredients for the recipe, and the output is the purchased ingredients.

[2140] Step 3: Provide emotion data

[2141] The system recognizes emotions from the user's facial expressions and voice in real time and transmits them to the server. The input is the user's facial expressions and voice, and the output is emotional data sent to the server.

[2142] Step 4: Provide feedback

[2143] After executing the selected recipe, the user inputs the results and their impressions as feedback into the terminal. The input is the recipe execution results and impressions, and the output is feedback data sent to the server.

[2144] (Application example 2)

[2145] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2146] In recent years, there has been a demand for making effective use of ingredients in the refrigerator and reducing waste, but it has been difficult for users to find recipes that suit their preferences and emotions. Furthermore, when using delivery services, food options are not optimized because suggestions are not made that take into account the ingredients in the refrigerator or the user's emotions. This has led to problems such as wasted ingredients in the refrigerator and inappropriate food delivery suggestions. The present invention aims to solve these problems by efficiently utilizing ingredients in the refrigerator and realizing appropriate recipe and delivery suggestions that match the user's emotions and preferences.

[2147] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[2148] In this invention, the server includes means for analyzing images captured by a camera installed inside the refrigerator and recognizing ingredients in the refrigerator, means for suggesting recipes that can be made using only the ingredients in the refrigerator based on the recognized ingredients, means for suggesting recipes that can be made by adding additional ingredients to the recognized ingredients, means for learning the user's past recipe selection history and suggesting recipes based on the user's preferences, means for analyzing the user's emotions and suggesting recipes based on the analyzed emotions, and means for displaying on a user interface recipes based on the ingredients in the refrigerator, recipes that require additional ingredients, recipes based on the user's preferences, and recipes based on the user's emotions.This makes it possible to efficiently use ingredients in the refrigerator without waste and to suggest personalized recipes and delivery options that suit the user's emotions and preferences.

[2149] A "camera installed inside a refrigerator" is a device that takes pictures of the inside of a refrigerator and obtains information about the internal condition and ingredients.

[2150] "Means for analyzing images and recognizing ingredients in the refrigerator" refers to algorithms or software that process image data captured by a camera and identify the types and quantities of ingredients contained within.

[2151] "Means to suggest recipes that can be made using only ingredients in the refrigerator" is a function that presents users with recipes for dishes that can be made without using any additional ingredients, based on the information of recognized ingredients.

[2152] "Means for suggesting recipes that can be made by adding additional ingredients" is a function that presents users with recipes for dishes that can be made by taking into account the ingredients in the refrigerator and other ingredients that can be added f...

Claims

1. a means for analyzing images acquired by a camera installed inside the refrigerator and recognizing ingredients inside the refrigerator; A means for suggesting recipes that can be made using only ingredients stored in the refrigerator based on the recognized ingredients; A means for suggesting recipes that can be made by adding additional ingredients to the recognized ingredients; A means for learning a user's past recipe selection history and suggesting recipes based on the user's preferences; a means for displaying, on a user interface, recipes based on ingredients in the refrigerator, recipes requiring additional ingredients, and recipes based on the user's preferences; A system including:

2. The system of claim 1 , further comprising means for generating a list of additional ingredients based on a user-selected recipe.

3. The system of claim 1 , further comprising means for suggesting new family recipes based on the user's past recipe selection history.

Images